[SOLVED] Back for More - Socket timeout help needed

speechles · Post by **speechles** » Sat Feb 28, 2009 7:43 pm

proc racInfo {} {

This is likely the problem, the curly braces are touching.

Code: Select all

proc racInfo { } {

Post by **nml375** » Sat Feb 28, 2009 7:47 pm

Speechles, have you been smoking weed again?

Whether the list of arguments is completely empty or contains spaces has no implication whatsoever...

There is, however one error in my post where I'm building the url for the request... basically, I'm missing one close quote ("). I'll fix this as soon as I'm done with this post...

Post by **nml375** » Sat Feb 28, 2009 8:09 pm

@DJ-zath:
With the exception of the quote I missed, I can run my code just perfectly out of the box. If you are still getting the "extra characters after close-quote" error, I'd guess you've overdone your "fixing".

Also, as I'm using namespace addressing with variable names, the global command is not needed here. Of course, there is nothing wrong with using it if you prefer that convenience .

If you would've lacked the required package (http), the very first line would've caused the script to stop loading.

dj-zath · Post by **dj-zath** » Sat Feb 28, 2009 8:29 pm

I got the example to work..

I added "putlog {rac connected and read}" in the "ok" subscript

but still the same..

1.. not getting any data from the connection
2.. bot locks up and then reports an error after 5 minutes (average) if the RAC machine is unplugged from the net

[18:11] Running racInfo
[18:11] Rac connected and read

[18:17] Tcl error in script for 'timer12':
[18:17] can't read "state(status)": no such variable

Post by **nml375** » Sat Feb 28, 2009 9:03 pm

You could try decreasing the timeout from 30 seconds to perhaps 5 or so.
It would seem the http-package does block until the connection is made, despite using the callback functions.
I'm not sure this is what the authors intended, but whenever you use -timeout, vwait is called which will block until the connection is established and request sent.

I do find it very odd that your eggdrop locks up for 5 minutes, as the timeout is set to 30 secs...

That said, you could still get a few good ideas as to implementing this by checking the source for the http package (http.tcl). Check http://moosaico.com/clients/http.tcl if you're having a hard time finding the source for it.

speechles · Post by **speechles** » Sat Feb 28, 2009 9:04 pm

@nml375, what ever gave you that idea

haw

Code: Select all

set ::RacInfo [::http::data $token]

This is what causes that problem, the error is occuring inside the http package so you either can catch this, or simply upvar the state array and check if infoexists within state(meta) if so good, if not bad...

Also, seems before it can even get to this point

Code: Select all

switch [::http::status $token] {

The switch used is actually the error your seeing presently. This entire segment would need to be caught, or the upvar/if infoexists used above put before the switch to check if that can even still be done.

Post by **nml375** » Sat Feb 28, 2009 9:13 pm

speechles,
please my dear friend, do a traceback when the error occurs, and you'll see that _that_ part never gets executed in the case of a timeout.
It is from the ::http::wait function, which didn't expect me to clean up the token in the gotInfo proc in the case of a timeout.

Now, look alittle closer, and you'll see that we first test the state using the ::http::status function (pretty much just gets us the status-element of the token, just a fancy way of doing it). If, and only if, this is "ok" (which means that the request worked just well, and there's data ready for us), we will move on and executing ::http::data, which just gets the body-element from the token. So, now, could you please explain how this is in error?

Oh, and by the way...
If the switch is the error, how come we get our custom log output saying the request failed due to a timeout? (atleast I do in my tests) :p

Edit:
Further... remove the ::http::cleanup and the error disappears.. leaves alot of garbage in memory though.

speechles · Post by **speechles** » Sat Feb 28, 2009 9:22 pm

nml375 wrote:Oh, and by the way...
If the switch is the error, how come we get our custom log output saying the request failed due to a timeout? (atleast I do in my tests) :p

This may depend entirely on the version of http package sourced into the script, or defaulted by install or what have you. This could also be related to the state after peculiarity, and not really be an error at all but a bug within http package itself.

Post by **nml375** » Sat Feb 28, 2009 9:35 pm

I believe the "state after" thread pretty much covered my first paragraph in my previous post... and as I also stated within that post, _not_ calling ::http::cleanup in the event of a timeout solves the issue.

I believe you'd find it hard to find systems with recent versions of tcl that wont load the 2.5 version of http-package by default. Of course, I could alter the package require line to demand 2.5 or more recent, should that make you more comfortable.

Still, you have not answered my question as to how the way I use ::http::state and ::http::data is in error...

dj-zath · Post by **dj-zath** » Sun Mar 01, 2009 3:23 am

guys guys!

first off, I really do appreciate the help and input!

second, I did find the "missing quote" error and got it to load without any errors.. (though it still hangs the bot)

I THINK I know where the 5 mins comes into play.. when a connection through the router is opened, but no data is sent.. it will DROP the connection but not tell the sender (or receiver).. data between the 2 just doesn't happen.. I think this confuses TCL in that perhaps it DOES receive somethg at connect which makes it believe its a good socket.. then after a while longer (than the initial timeout) it will eventually drop connection... I have had it happen from 2 mins to infinite... avg is 5 mins though..

in either case... running async, http, gets, puts, open, etc etc etc the following is true:

"blocking" socket will read ONLY on the initial connection... and that, that connection doesn't experience ANY connection issues.. or the bot hangs- sometimes indefinately!

non-blocking sockets won't read- at ALL... I can NOT get a asynched (non-blocking) socket to read by ANY means.. I tried fileevent, while, if's -you name it- it seems the RAC IS getting the puts.. but the bot isn't getting the reads/gets...

if it matters, I'm running FreeBSD-6x and the ISP runs litestream/cisco routers and Fedora FC9 throughout its network...

I CAN open the Rac to yous- and perhaps you can see this for yourselves..

or

most-likely, I'm a DUNTZ- and been at this particular issue for TOO long.. and most-likely missing a step somewhere.. in this case, I ask for a example of a simple non-blocking puts and gets that uses standard TCL commands... then I can cross-reference it to what I've done.. and see what the HELL is going on with this blasted thing!

I tell you! this ISP is WEIRD... TCL sees connections to machines that aren't even there.. of course, theres NO data to be sent.. and this seems INTERMITTANT even when the machine is there..and I have NO WAY of knowing if, indeed, my code is the fault or the ISP!

I've spent way too long on this MESS... God, I need a beer.. and I don't even drink!

-DjZ-

Post by **nml375** » Sun Mar 01, 2009 12:40 pm

With the -timeout option used with the http-package, regardless of the network state regarding the socket in use, this should return control to the script within the timeout period. Basically, the package uses the after command (think of it as tcl's own timer command, similar to eggdrop's timer/utimer, yet it uses tcl's event engine) to reset the token after the timeout period, thus aborting the http-transaction, and in the case of "connection pending", return control back to the script.
If the call to ::http::geturl blocks longer than that, there is something seriously wrong.

As for blocking/non-blocking:
Blocking commands will block until they complete their task. If they block, they will prevent your eggdrop from performing any other tasks until they complete.
Non-blocking commands will not block if they are unable to complete the task, instead they will either abandon the task or schedule it for event-based execution. This will not inhibit the operation of eggdrop.

So, the point here, we want non-blocking operations to make sure your eggdrop does not freeze. This means, that we must code in a new way to cope with the conditions that the command(s) did not complete their task.
Making asynchronous connections with the socket command may be considered a non-blocking task, yet the socket itself is not non-blocking. We can easily solve that using fconfigure.

Now.. How do we do this properly?
Well, first off, we create the socket and instructs it to try and connect somewhere:

Code: Select all

set socket [socket -async 127.0.0.1 80]
fconfigure $socket -blocking off

Next, we have to find out when the socket has become connected (writable). We'll use tcl's event-engine along with fileevent for this.

Code: Select all

fileevent $socket writable [list socketConnected $socket]

Here we call a proc named socketConnected when the socket becomes writable (connected), so we'll have to create this one...

Code: Select all

proc socketConnected {socket} {
 putlog "connected: $socket"
 fileevent $socket writable [list doRequest $socket]
}

Once we've connected, we'll redirect the fileevent to use a different proc (just to keep the steps separated, could probably stack it all into this one) named doRequest. This one should be responsible for making the request:

Code: Select all

proc doRequest {socket} {
 if {[llength $::theData] > 0} {
  puts $socket [lindex $::theData 0]
  set ::theData [lrange $::theData 1 end]
 } {
  fileevent $socket writable ""
  fileevent $socket readable [list getResponse $socket]
  flush $socket
 }
}

This one retrieves the request from a list named theData in globalspace. Each list item should be a line of text. So, lets create it..

Code: Select all

set theData [split "GET /x/playlist.cgi HTTP/1.0
Host: somehost:someport

" \n]

Also, once the request has been sent, it changes the event handles again, this time connecting getResponse to the readable event. Guess we better create that proc aswell...

Code: Select all

proc getResponse {socket} {
 gets $socket line
 if {[eof $socket]} {
  putlog "Connection closed."
  close $socket
 } elseif {![fblocked $socket]} {
  append ::RacInfo "$line\n"
 }
}

Tying it all together would yield something like this:

Code: Select all

#Start with the procs...
proc socketConnected {socket} {
 putlog "connected: $socket"
 fileevent $socket writable [list doRequest $socket]
}
proc doRequest {socket} {
 if {[llength $::theData] > 0} {
  puts $socket [lindex $::theData 0]
  set ::theData [lrange $::theData 1 end]
 } {
  fileevent $socket writable ""
  fileevent $socket readable [list getResponse $socket]
  flush $socket
 }
}
proc getResponse {socket} {
 gets $socket line
 if {[eof $socket]} {
  putlog "Connection closed."
  close $socket
 } elseif {![fblocked $socket]} {
  append ::RacInfo "$line\n"
 }
}

#Prepare our query and clear RacInfo:
set RacInfo ""
set theData [split "GET /x/playlist.cgi HTTP/1.0
Host: ${RacIP}:${RacPort}

" "\n"]

#Lets fire up the socket and start the request:
set socket [socket -async $RacIP $RacPort]
fconfigure $socket -blocking off
fileevent $socket writable [list socketConnected $socket]

This code should be fully able to retrieve the data from the remote host, should the connection succeed at some point in time.
If the remote host is not available, it will not hang your eggdrop. However, it will not abort the request either, so you might end up with a large number of opened socks.
For this we need a timeout feature, but we'll save that for laters. Start with getting this to work with your script...

Edit: Out of old habit I used the 1.1 version of the http protocol. But since we don't support chunked transfers, I updated the code to use the older 1.0 standard.

dj-zath · Post by **dj-zath** » Sun Mar 01, 2009 9:59 pm

hi there nml375 and speechless!

YOU DA MAN!

I think I got it figured out!

heres the working test code:

Code: Select all


proc    RacSkt {} {
        global RacIP RacPort
        if {[catch {set RacSock [socket -async $RacIP $RacPort]; fconfigure $RacSock -blocking 0;}]} {
                putlog {Socket NOT Detected};
                return 0;
        } else {
                putlog {Socket Detected};
                fileevent $RacSock writable [list RacNfo $RacSock];
        }
}

proc    RacNfo {RacSock} {
        global RacInfo RacIP RacPort

                flush $RacSock;
                puts $RacSock "GET /x/playing.cgi HTTP/1.0";
                puts $RacSock "User-Agent: Mozilla";
                puts $RacSock "HOST $RacIP:$RacPort";
                puts $RacSock "";
                flush $RacSock;
                putlog {Puts to Socket: COMPLETE!};
                set T 0;
                while {($T <= 1000)&&(![eof $RacSock])} {
                        incr T;
                        set RacInfo [gets $RacSock];
                }
                putlog {Gets From Socket: COMPLETE};
                flush $RacSock;
                close $RacSock;
                putlog {Connection Close and Complete}
}

Now, of course, this is just the test code- to test the behavior of the bot with the funky connections.. I have a lot to do to it still.. like interactive host-detect and "dead connection detection" but this is a REAL GOOD start! .. Then, once approved, I'll modify the master code that its derived from... I have to add error control and parsersand all that .. but I have it working WITH timeout and theres no more blocking or stalling.. and yes, it should clean up after itself too- I only open the socket once per cycle.. thing is I have to open three sockets from three different places and this socket twice to read more from them; thats gonna take some more work.. but this was the stickler! You and Speechless have been a GREAT help (and yeah, I spent all last night reading the man pages but your example above was the part that helped the most)

I owe you both a beer!
okay.. make it a six-pack!

If you're interested in seeing the finished product. let me know..

But, for now, I have a radio station to get working.. oh! and how about http auth? LOL I was reading about set $string[encrypt $string] a bit last night.. that will be my NEXT venture (icecast wants a secured login with auth.. oh boy!) I have ideas on how to do it however

again thanks for all your help!

-DjZ-

dj-zath · Post by **dj-zath** » Mon Mar 02, 2009 12:00 pm

At first, I thought I had iit..

it still needs work, though...

now I found that once I got the RAC working, then Icecast wouldn't work!

turns out I have to have Icecast as "blocking" with "read" and the RAC as non-blocking with "gets"..

I'm also experiencing a lot of mis-reads.. (causes the song titles to keep updating and resetting)

at this time, I'm not sure as to why just yet...

I'll let cha know what I find (been at this thing all night long again)

(I think I need to reboot my radio modem and have the ISP reboot their flaky tower again)

OOPS!

I think I see whats wrong! Eggdrop's EATING the CPU like CANDY!

hmmmmm...

Post by **nml375** » Mon Mar 02, 2009 12:47 pm

When using non-blocking sockets, it is strongly recommended that you do not use the read command, but use the gets command along with the readable fileevent - as shown in my example.

Edit: read may also be used, but with caution.

dj-zath · Post by **dj-zath** » Mon Mar 02, 2009 1:02 pm

yes, I understand.. problem is ICECAST isn't responding to the Gets command- probably cause it sends XML/XSL output! gets seems to return an empty string on this type of output...

I got a new problem as well ammost 100% CPU usage.. and i think this is causeing erratic output..

oh, I guess I shoud explain:

for Icecast, I have the "old" method with blocking and a simnple read of eof...

for the RAC I have your example (modified) I donno why I spent 8 plus hours trying to get your example to work on Icecast.. it just doens't wnat to work.. I can only giess its because icecast outputs depending on what browser and what template is loaded... it output a XSL style sheet from another site or somethig weird like that (you know one of those "compiles in the browser" type of deals.. a "read" just gives it to me RAW- and I have parsers pulling out the junk.. but I have that running in blocking mode- since thats the only way it seems to work...

obviously, I have a lot more work to do..