http geturl 'limiting'?

[DayCuts]

I am writing a tcl script to return certain information from a .csv on a website using the http package. Is there a way to limit the amount of bytes the script will read from the file. This file can become quite large, as such using normal http geturl can be slow to respond (due to reading the entire file). It would be much more efficient if i could just read the first XX bytes or X lines, since the most updated information that the script will be using is in the first few lines.

Any and all help much appreciated, thanks in advance.

[user]

-blocksize combined with -progress should do the trick.

[DayCuts]

Hmm, can you give me an example of the syntax for this? This is what i had before my original post, however it still transfers the entire page of data. This is my first script using the http package so i am sure i do not understand the proper usage and syntax of it all.

Code: Select all

set token [::http::geturl $itrurlg -blocksize 50]

I have read the -progress help, however i still don't quite understand proper use and syntax. So am a little lost on how to combine the two. The information after the first break is not at all needed, so stopping after the first block is what i am trying to do.

[user]

I took another look at that manual page and noticed another option that seems more suitable for your needs: -handler

Code: Select all

package require http
proc yourCallback {sock tok} {
	while {[gets $sock line]==0} continue;# read untill we get a non-empty line or eof
	puts $line;# you probably want to do something else here :)
	http::reset $tok;# i'm not sure this one's needed, but I included it anyway :P
	http::cleanup $tok;# this one's mandatory
}
http::geturl http://www.your.url/ -handler yourCallback -command nah -timeout 20000

the -command thing is just a hack to avoid blocking (and nasty error messages because of the cleanup performed before geturl returns )

[stdragon]

Another thing is to use the regular form and send the http byte range request header.

It's like:

Range: bytes=0-100

[user]

Another thing is to use the regular form and send the http byte range request header.

yeah..that's a good idea as you'll also avoid wasting lots of bandwidth transfering data you don't want/need.

[DayCuts]

yeah..that's a good idea as you'll also avoid wasting lots of bandwidth transfering data you don't want/need.

This is what i was wanting to do in the first place.. not transfer un-needed data.

I tried using -headers "Range: bytes=0-10" in the http geturl but it was unsuccesful so i assume the server doesn't allow byte rangers.

Also spent awhile playing with your method, user. It made almost no difference than my original code, accept that it organised the data a little different, still transfered the entire file. Which is what i am trying to avoid.

Code: Select all

  if {$text == ""} {
    set token [::http::geturl $itrurlg]
    set content [::http::data $token]
    ::http::cleanup $content
    regsub -all -- {,} $content { } cont2
    list $cont2
    # for debug purposes
    putlog "[lrange $cont2 0 end]"
    return 0
  } else {

Sample of content

Code: Select all

Date,Rating,Latency,Loss
13:25 6/15/2004,85,145,2
13:20 6/15/2004,85,144,2

the raw data is updated every 5 minutes, the only information i need is line no. 2

[strikelight]

I tried using -headers "Range: bytes=0-10" in the http geturl but it was unsuccesful so i assume the server doesn't allow byte rangers.

It's -headers "Range bytes=0-10"
The http package will properly format it with a : when it sends to server

[DayCuts]

It's -headers "Range bytes=0-10"
The http package will properly format it with a : when it sends to server

I tried this also. : )

As i mentioned in my first post, this is a .csv file i am accesing, not an average html page, this is probably why the html headers are ignored.

[stdragon]

I'm pretty sure Range will work with any file, but you have to be using http 1.1. Normally the http package uses http 1.0. So you have to modify it a bit. Find the 1.0 and change it to 1.1. You might have to add a bit to it too, like the Host header (I think) and Connection: close.

[DayCuts]

Ic, yeah looking at the http tcl it uses 1.0.. I changed 1 occurance of 1.0 to 1.1 in \lib\tcl8.4\http2.4\http.tcl

You might have to add a bit to it too, like the Host header (I think) and Connection: close.

Where would i change this in the http tcl?

What i did is this..
change: puts $s "$how $srvurl HTTP/1.0" to puts $s "$how $srvurl HTTP/1.1"
added line putlog "$key: $value" in the -headers
added line puts $s "Connection: close" immediately after the the foreach for the -headers, this succesfully send info to the consol, so i know its processing the range: bytes. However it still transfers the entire file....

Im totaly lost, is their another way? user mentioned using -bloacksize and -progress? (but as i said before i don't full understand how to implement -progress)

[stdragon]

I changed 1.0 to 1.1 just like you and added puts $s "Connection: close" right below it.

Then I did:

set token [http::geturl http://localhost/test.html -headers

• ]
  puts [http::data $token]
  
  It worked for me.