Non-blocking use of http::geturl returns results very slow ?

[madwoota]

Can anyone offer any suggestions or reasons why a non-blocking style http get (using the http package) ala demonds rssnews script would take such a long time in comparison to a regular get/wait retrieve ?

I've timed the non-blocking method at ~15-20 seconds whereas a regular (blocking) read/wait only takes ~2-3 seconds.

Not overly sure what causes the delay. Is this to be expected/normal ?

[incith]

It is related to this: http://www.eggheads.org/pipermail/bugs/ ... 00960.html

There is a patch which he attaches, located here: http://eggheads.org/pipermail/bugs/atta ... levent.bin (mirror)

Update: The patch above needs to be applied in order for things to work properly. After applying the above patch, non-blocking http gets work properly and normally.

[strikelight]

I'm pretty sure by undefining that, you are essentially getting rid of non-blocking Tcl operation from eggdrop. Hence, you may as well just use the blocking method of the http::geturl command.

[incith]

I would say that's a fair assumption. In my tests, the callback routine will get an incorrect token, which essentially breaks http again, despite improving/fixing the events as well in a way.

Basically, you will call ::http::geturl, the callback will get ::http::1 and then ::geturl returns ::http::2, which is really bizarre. After recompiling eggdrop back to normal this behavior disappeared. So applying the 'patch' above will both fix and break http callbacks. This lead to some strange things, such as the callback would be trying to work on the previous URL/token, because the ::http::geturl is returning a different token. I even tried to increment the token, but then the callback didn't have any data to parse because ::http::geturl hadn't actually returned the new token yet.

Well, hopefully this will be fixed one day as a whole and we can use fast non-blocking events in eggdrop, because the callback was pretty cool even while it was broken.

[strikelight]

I'd be interested in knowing how egghttp.tcl compares to the initial poster's script.

[incith]

I jumped the gun a bit, it appears he attached a patch file which does more than just undef USE_TCL_EVENTS. I will give it a go later on and see if it actually does make Tcl events work properly, and fast. Second post updated with links.

Update: The patch fixes everything properly.

[arfer]

Perhaps this is relevent here

I found the same difficulty with execution times and as a solution took to declaring the callback proc outside of the geturl statement to speed things up considerably, as per the following :-

Two styles of using callback

pFetch1 proc with call to pCallback outside of the geturl statement executes quite quickly

pFetch2 proc with the call to pCallback within the geturl statement is comparatively slow to execute

no idea why

proc pFetch1 {address} {
set agent [::http::config -useragent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"]
if {[catch {set token [::http::geturl $address -timeout 10000]}]} {
putlog "cant connect to $address"
} else {pCallback $token}
return 0
}

proc pFetch2 {address} {
set cmd pCallback
set agent [::http::config -useragent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"]
if {[catch {set token [::http::geturl $address -command $cmd -timeout 10000]}]} {
putlog "cant connect to $address"
}
return 0
}

proc pCallback {token} {
# callback code here
}

[nml375]

The difference in these two cases, are that in pFetch1, the call to ::http::geturl will block until the HTTP transaction is completed and token is ready, and not relying on Tcl_DoOneEvent() being called; whereas in pFetch2, the call to ::http::geturl will return immediately, and launch an event-driven job "in background".

What might be worth keeping in mind, is that with pFetch1 your eggdrop will become unresponsive until the HTTP transaction is completed. Be careful when selecting timeout values so your bot won't get disconnected (ping timeout)

[strikelight]

Not to mention if you have other scripts loaded that require accurate timing or fast execution as well, that they will be unable to do so when the bot is executing that blocked code.

[incith]

This is why it is a good thing to apply the patch above to your eggdrops source.

@arfer: Only one is using callback in your example, the one with -command.

I have begun adding 'variable callback' to my scripts for those of you that do want to enable callback procedures. I did notice slowdowns still, but nothing too painful. For others, it is very simple and basically the same as above to add optional callback, something like:

Code: Select all

set callback 0/1  # disable/enable
if {$callback >= 1} {
  set token [::http::geturl -command httpCallbackProcedure BlahBlahBlah]
} else {
  set token [::http::geturl BlahBlahBlah]
  httpCallbackProcedure $token
}

One problem I found was when errors occur in the callback, there is no output or logging. Even if you call non-existant variables, it just stops at the failing line of code, but doesn't let you know why it stopped.

[strikelight]

What is the cpu consumption of an eggdrop applied with that patch compared to an eggdrop without it applied?

[nml375]

@incith:
Errors generated in event-driven code is reported through the bgerror mechanism (since there is no "toplevel" caller which to return the error).

A very simple way of logging these errors would be something like this:

Code: Select all

proc bgerror {message} {
 putlog "bgerror: $message"
 putlog " - errorInfo: $::errorInfo"
 putlog " - errorCode: $::errorCode"
}

Also see http://www.tcl.tk/man/tcl8.0/TclCmd/bgerror.htm