[SOLVED] Back for More - Socket timeout help needed

[dj-zath]

First off, I want to thank everyone who helped me resolve some odd things in the prev posts..

The overall problem then was FEDORA FC9 .. I since DUMPED it like a bad habit and went back to FreeBSD and all those goofy problems went away- well, except for one.. and its to do with FC9 as well; this one is caused because my ISP uses Fedora FC9 throughout its network (ughh!) and I still have to connect through their FC9 sockets..

Now, I have to contend with "black hole sockets" on the immediate network for incoming connections OUTSIDE of the box itself... this will take some explaining...

It seems that my ISP engages in the pratice of "accepting connections that don't go anywhere" and yes.. TCL has a COW over it! (its believed they do this to "break" torrents and port scanners- in which it is VERY effective at doing!)

The problem? INDEFINATE "hang" on to an established socket connection- even though the machine it connects to is NON-EXISTANT!

the Fix.. "if said socket doesn't complete/close within 'x' time, KILL it!"

I have tried -async and event timers, etc.. but to no avail.. the problem is that TCL sees a socket- but its a fake one generated by my isp.. and this fake socket intends to confuse and HANG the connection indefinately!

heres a piece of some "test" code I've been working on that addresses this problem specifically...

Code: Select all

if {![info exists loop]} {
        utimer 1 [list RunLoop1];
        set loop 1;
};

proc    RunLoop1 {} {
        [list RacInfo];
        utimer 1 [list RunLoop1];
        return 1;
};

proc    RacInfo {} {
        global RacIP RacPort
        if {[catch {set RacSock [socket $RacIP $RacPort];} sockerror]} {
                putlog {Socket BAD!};
                return 0;
        } else {
                putlog {Socket GOOD!};
                close $RacSock;
        };

}

In a normal situation, the code above works perfectly, however, TCL seems lacking in today's net-envirement where connections are less-than reliable and ISP's break protocol with intent to traffic-shape and torrent-block (not to mention ping-block and port-scan/intrudsion as well) TCL really needs to add a socket timeout function to its core and not some proprietary patch/verision you have to buy and/or re-learn all new sintax to use it (aka TCL-tk) (Okay, that is an argument for another day )

What happens here is kinda hard to explain.. so I will describe the "testing situations" and what they resault based on the above mentioned code..

At FIRST turn-on/loading of the bot, with local machine (to where it connects TO called "RAC") is OFF

-- Socket Bad -- reports immediately- (as it SHOULD)

then RAC is started....

--Socket GOOD -- reports immediately (again, as it should)

then I UNPLUG RAC from the net (to simulate a broken connection)

NOTHING reports and the bot hangs INFEFINATELY -OR- I'll rescieve a "Timer Spun" error followed by a sometimes CRASH of eggdrop.

if the bot doesn't crash after a "timer Spun" error, its completely NON-responsive and has to be restarted anyways!

what causes this is the ISP will send a "null" socket in the RAC's place- this socket appears "valid" at "set-up" but, once established, becomes a BLACK HOLE.. and TCL/eggdrop will either stall/then timer spun.. or HANG for EVER trying to (re)establish.. its NOT TCL or eggdrop's fault.. its just my ISP playing DIRTY!

the solution, I believe is to only allow the socket to remain open for 500 ms, then force it closed.. irreguardless if it got data or not.. (I have determined that, when working properly, 200 ms is MORE than enough time to open, gets, flush and close a properlly-working socket in this situation)

I tried many different things. but had no real success in finding somethig that works reliably and/or effciently.. (the code above is the BASE code, after I cleaned up all the "mess" I've made in it)

any help/suggestions would be appreciated!

-DjZ-

[nml375]

This... rather sounds like a firewall simply dropping SYN packets on any non-authorized ports. This is a classic setup to prevent any malware from calling home, and really has nothing to do with freebsd, linux, or FC. Sockets are sockets, and they don't come in various distribution brands or such.

Now, lets get to the tcp/ip interface within tcl:
The socket command will only block when doing a synchronous connection, until either an ICMP-error is returned or a SYN+ACK packet is recieved from the remote connection.

Your eggdrop does not make a connection to something non-existant, and there is no established connection.

So, obviously your ISP is dropping your SYN packets ("connection packet" if you so like).

When you worked with asynchronous sockets, did you remember to configure it as non-blocking? Otherwise, as you might have read from the socket manpage, any attempts to read on an yet unconnected socket will block until the connection is completed. And no, this is not a fake socket created by your ISP. It's just the way tcl handles unconnected blocking sockets (hence the name "blocking").

So, how do we sort this one out?
1st off, check that you don't have some firewall rules enabled on your local system, that prevents outbound connections. Very often, this is the main cause for issues like this, and way beyond the control of your ISP
2nd, are you behind a NAT? And are you trying to connect to a forwarded port on the NAT router/firewall from within the protected area? Some routers/firewalls explicitly drops these kind of connections for various reasons.
3rd, talk to your ISP about their policy on firewalls and outbound connections?

Of course, script should be able to handle such a case anyway. I'd suggest using something like this:

Code: Select all

if {[catch {socket -async $host $port} socket]]} {
 putlog "Connection failed: $socket"
} {
 putlog "Connection started: $socket"
 fconfigure $socket -blocking 0
}
...
gets $socket line
if {[eof $socket]} {
 putlog "Socket $socket closed by remote end"
} elseif {![fblocked $socket]} {
 putlog "Read $line from $socket"
}

This could then be extended with some manual timeout that will not block the operation of your eggdrop.

Unrelated:
Now, why do you insist on using the list command like this:

Code: Select all

[list RacInfo]

[dj-zath]

Wow! I was still posting (editing) when you responded.. AWESONE!
thank you much for your help!

Now, to answer your questions:

1.. yes, there are firewalls involved.. and I have them set to specific ports and IPs etc etc and both the remote machie (sitting in the ISPs racks as a CoLo) and the local "RAC" machine are set up to talk to each other- and only each other in this manner..

2.. there IS a NAT- but I don't have controil over it! The NAT is at the ISP..

My machies here AND the machine in the "racks" are on the SAME EXACT
SUBNET; in fact, the entire town is on the same LAN- the ISP assigns its subscribers their IPs.. and we all share the SAME set of gateways...

yes, its GOOFY, but thats how they do it here.. think if it as one giant "home based router" configuration.. there is NO WAN here!

and yes, you are correct in saying that a NAT may be the issue.. however, when the machine is present, even if the application itself isn't running, the returned condition will be "socket bad" (or not present) and the response will be immediate.. its when the (local) machine is unplugged from the bridge that causes the problems..

I have tried what you have suggested, however, you show the code laid out differently that what I had.. perhaps I had it WRONG (though there were no errors reported) I will take your code and try it out! again, thanks for that example! I'll let you know what comes of it

now about using "list"

more of a holdover from using Fedora, I suppose.. when I was trying to run TCL in FC9 , I HAD to use "list" to invoke a proc in a timer event.. I read that using "list" assures that operations in the list will run in their specific order (the orginal code this all rerives from is as follows:

Code: Select all

if {![info exists loop]} {
        set DetM "offair.gif";
        utimer 30 [list RunLoop1];
        utimer 32 [list RunLoop2];
        utimer 34 [list RunLoop3];
        utimer 36 [list RunLoop4];
        utimer 38 [list RunLoop5];
        utimer 40 [list RunLoop6];
        set loop 1;
}

its important that these subloops load in their EXACT respective order when called or I get lots of "missing varable" errors since each subloop runs a proc that generates a set of globals and some procs read other's globals transversly!

its CRITICALLY-IMPORTANT than things load "in order" and, once, thats done, then it doesn't matter.. I read that "list" helps achieve this- list as to "load one thing after another" and, as I said before, when in Fedora, I HAD to use list or I was completely UNABLE to call ANY thing in a timer event loop without a list command. in FreeBSD, list isn't neesed or nessessary.. iof you suggest I not use list, I can do so withhout a problem.

-DjZ-

[nml375]

Actually, what you are suffering from, is routing issues. For some reason, your ISP's router fails to detect the absence of the remote host (Rac), and hence keep trying to send packets to it.
The effect of this, is that the router thinks it did it's job well, and forgets about the whole matter, when it really should send an ICMP host-unreachable response back to the other end. As such, this could very well be thought of a "black hole" or "sink". However, since the connection never is established, no sockets (which is a local resource, not something spread over the network) are created at all.

Using the code I posted, along with fileevent, would allow you to wait for the connection to be established (or dropped), without blocking eggdrop. With some creative coding, you'll also be able to add some cleanup code to remove stale sockets.

[nml375]

Regarding list and timers:
You should always use list with timers. However, not for that reason, but a completely different (and way more important) reason.
If you were to include variables or similar, you would suffer from something known as "double evaluation". I believe this is best explained with an example:

Say we have some public command !fuu, which responds with "<nickname> bar" after 10 seconds. The poor code would look like this:

Code: Select all

bind pub - !fuu startFuu
proc startFuu {nick hand host chan text} {
 utimer 10 "puthelp \"PRIVMSG $chan :$nick bar\""
}

Looks neat and well, eh?
Well, lets say the nick i[die]ha triggers that command..
So we start a 10 sec timer with a command body as this:

Code: Select all

puthelp "PRIVMSG #somechan :i[die]ha bar"

After 10 secs, that code will be executed, as since it contains [], whatever is inside will be evaluated and substituted, effectively killing the eggdrop...

Should we do it properly, like this:

Code: Select all

bind pub - !fuu startFuu
proc startFuu {nick hand host chan text} {
 utimer 10 [list puthelp "PRIVMSG $chan :$nick bar"]
}

The command body for the timer instead becomes this:

Code: Select all

puthelp {PRIVMSG #somechan :i[die]ha bar}

Doesn't look that different, eh? Well, the curly braces instead of the quotes does all the magic, as it tells tcl to do no substitutions whatsoever (hence ignoring [die]). And our eggdrop survives!!!!

However, when you're not using timers, there'd be no point in doing this:

Code: Select all

[list somecommand]

As it simply executes somecommand anyway.

[dj-zath]

you are such a great and helpful resourse again, thanks so much for your prompt reply and help!

I can see exactly what you are saying about the SYN's and the router NOT detecting the missing connection.. I shoud add that I have determine this also occurs within 2 minutes after the rac is unplugged (or if I lost connection to the net) and, thus, that socket goes completely DEAD even AFTER resetting/reconnecting the RAC machine. or restarting eggdrop on the server...

example of another "related" issue is XBL will DISCONNECT after initial connection within 2 minutes IF theres no activity on the connection once established.. same also applies to any computer on this network as well.. (say, open IE, and go to some website such as eggheads.org) and, if no activity for 2 minutes, the socket will go DEAD.. yet will NOT drop.. and theres NOTHING I can do to any machine (even the xbox itself) except to completely REBOOT it! same applies to writing this message- I have to keep "saving/refreshing" or I LOSE the post page completely!

at least the server itself seems to handle this situation better than other network connections in town.. (yes this problem extends beyond my own computers- it happens to EVERY one in town) I can also understand what you are saying, which, is why I said about killing the dead socket in its tracks..

theres one thing I didn't quite uinderstand in your example of the socket code- mainly because I didn't include what the socket DOES once its opened.. (thats MY bad) I should post the ACTUAL proc section itself.. (minus the parsers whihin) the reason I didn't post it in the first place, is the entire code is 790 lines!

heres the orginal "socket" code (minus the parsers):

Code: Select all

proc    RacInfo {} {
        global RacIP RacPort Tmr DetO DetAP DetPY DetPL MetaPY MetaPL
        if {![info exists DetO]} {
                set DetO "offair.gif";
                set DetAP "offair.gif";
                set DetPY "offair.gif";
                set DetPL "offair.gif";
                set MetaPY "Not Available";
                set MetaPL "Not Available";
                return 0;
        };

        if {[catch {set RacSock [socket $RacIP $RacPort];} sockerror]} {
                set DetO "offair.gif";
                set DetPY "offair.gif";
                set DetPL "offair.gif";
                return 0;
        } else {
                set DetO "onair.gif";
                flush $RacSock;
                puts $RacSock "GET /x/playing.cgi HTTP/1.0\r\nUser-Agent: Mozilla\r\n\HOST $RacIP:$RacPort\r\n\r\n";
                flush $RacSock;
                while {![eof $RacSock]} {
                       set RacInfo "[read $RacSock]";
                };
                flush $RacSock;
                close $RacSock;
                if {[catch {set RacSock [socket $RacIP $RacPort] } sockerror]} {
                        return 0;
                } else {
                        flush $RacSock;
                        puts $RacSock "GET /x/playlist.cgi HTTP/1.0\r\nUser-Agent: Mozilla\r\n\HOST $RacIP:$RacPort\r\n\r\n";
                        flush $RacSock;
                        while {![eof $RacSock]} {
                                set PlayInfo "[read $RacSock]"
                        };
                        flush $RacSock;
                        close $RacSock;
                };

                if {$RacInfo == "" || $PlayInfo == ""} {
                return 0;
                };
}

[nml375]

Sounds like you've got some serious network issues there.
If I were you, I'd get on the phone asking that ISP what is wrong with the network.

[dj-zath]

been there, done it!

they tell me "theres nothing wrong with it"

they then blame my WiMax connection (and I won't even mention the "flakey relay tower" that needs rebooting once a night... (the ISP also reboots their system once a week- to close all those "broken and dead sockets" they usually have to reboot things come monday morning, every time!

thats what net in small-town America is all about!


okay, I have transcoded your example into the test proc.. and running the tests now.. will report my findings:

heres the code for your reference (in case you see I did somethig wrong)

Code: Select all

proc    RacInfo {} {
        global RacSock RacIP RacPort
        if {[catch {set RacSock [socket -async $RacIP $RacPort];}sockerror]} {
                putlog {Socket BAD!};
                return 0;
        } else {
                putlog {Socket GOOD!};
                fconfigure $RacSock -blocking 0
                flush $RacSock;
                puts $RacSock "GET /x/playing.cgi HTTP/1.0\r\nUser-Agent: Mozilla\r\n\HOST $RacIP:$RacPort\r\n\r\n";
                flush $RacSock;
                while {![eof $RacSock]} {
                        set RacInfo "[read $RacSock]";
                }
                if {[eof $RacSock]} {
                        return 0;
                } elseif {![fblocked $RacSock]} {
                        return 0;
                };
                flush $RacSock;
                Close $RacSock;
        };
}

-DjZ-

[dj-zath]

still breaking it...

• [22:05] Socket BAD!
  [22:05] Socket BAD!
  [22:05] Socket BAD!
  [22:05] Socket GOOD!
  [22:06] Tcl error in script for 'timer66':
  [22:06] error reading "sock5": connection timed out

see? the RAC hasn't even been connected yet! TCL sees a "good" socket from outta thin air! its BAD- no doubt about it...

eggdrop then reports an error and "sig-terms"...

it also still "hangs" even on the async too...

so this leads me to the NEXT (or orginal) question:

if I should time it out manually, how do I do THAT.. since this proc runs once a sec.. but the timeout must occur BEFORE the proc ends- or CONTIUES in realtime!

thats NOT as easy as it sounds!

in the past, I had to use PHP as the socket "front end" to get around this.. it was slow and constantly wrote to the drive! I woud like to AVOID this situation..

[dj-zath]

UPDATE:

its not going to work...

running the sockets in async and trying to set a time-limit just son't working.. I can't read or gets anything out of them unless I:

1 set blocking
2 not set -async
3 not assign a "timed element" in an event loop (while)
4 not read any error outputs

5. theres just something "not right" somewhere... I feel like a n00b all of a sudden.. I guess 8+ solid hours at this crap and not getting anywhere willl do that to one... -sigh-

-DjZ-

[nml375]

First off... Bad/Good socket simply means was there an error when the socket was created... since we're using asynchronous sockets, the mere existence of a socket tells us nothing about the underlying network layer.

You really, really need to understand this to move on with this kind of coding:
Sockets are merely an interface to make network communications simpler for the coder, and only exist on the local system. It's merely a tag or identifier used to access the networking layer within the kernel with standardized function calls. A socket may have many different states, including unconnected. A socket might not even represent a network connection, but point to any kind of FIFO (such as named pipes, unix sockets, etc).

In the case of asynchronous outbound connections, the only case there will be an error creating the sockets, is if any part of the address is invalid, or the network API is currently unavailable (unrelated to the physical status of the network). The whole idea of asynchronous mode is that right now, we don't care to wait for the connection to be established, we simply want the socket so we may test later on whether we're connected or not...

In your code, you try to use the socket as if it was created using a synchronous method, that is, as soon as the socket command returns, the socket is connected. We don't know that, so things break...
Also, if you check my code, I actually included the error reason when socket failed to connect (and the socket id when it succeeded). This kind of info is vital for debugging these kinds of issues.

Furthermore, when dealing with non-blocking sockets, you should really should consider using tcl's event-engine (fileevent is your friend here), as this allows you to trigger code when, and only when, there is data to be read from the socket.

Regarding your results:

[22:05] Socket BAD! - Unable to create socket for some reason, but since the actual error message wasn't included, we have no idea why.
[22:05] Socket BAD!
[22:05] Socket BAD!
[22:05] Socket GOOD! - Finally the socket was created. Yet we don't know if it's in a connected state or not...
[22:06] Tcl error in script for 'timer66':
[22:06] error reading "sock5": connection timed out - And here's our answer, the socket never managed to connect, and instead timed out. I guess this is what you were looking for in the first place, a way for sockets to timeout if they fail to connect?

Yes, tcl sees a socket. We've created a socket. That tells us nothing whether that socket is actually connected anywhere. It just tells us that we've created a socket that allows us to interact with the networking layer within the kernel. Once you understand this, you'll have a lot better success in your coding.

And, to be honest, that last script will not be able to hang your eggdrop regardless of the network status. The socket is created in asynchronous mode, and instantly configured to non-blocking mode. As such, no operations on that socket may block code execution. If your eggdrop hangs when running that script, something else is very wrong with your local system. Especially, the SIGTERM signal should rarely be sent by the kernel or system from a network issue. Also, the default behaviour on recieving a SIGTERM is to shutdown nicely, saving user and channels files. Optionally, you may configure your eggdrop to do a rehash and continue running after saving those files. In neither cases, however, should your eggdrop "hang".

All this said, had you mentioned you were doing http-transactions, I'd simply told you to use the http-package, as this already implements timeouts, callbacks, full support for both GET and POST requests, aswell as full customization of request headers.

[dj-zath]

hi there nml:

I probably sound like an idiot to you... (thats OKay, sometimes I think I am one myself)

I DO understand "sockets" and that they are merely "channels" or "markers" to (in this case) network connections and that they are generated locally.. I have been using this term incorrectly- though, convient.. nevertheless...

about that posted log entry:

"Socket Bad" in this case means: no socket/channel-id was able to be created because the server(RAC) was unavailable which is correct and that socket SHOULDN'T have been created since at the time it was one: powered-down and 2.. NOT CONNECTED AT ALL. Of all, that was the expected result!

Now, the interesting part was when it reported "Socket GOOD" but then timed out.. Sure it timed out- there was nothing there to connect to! this is where things don't behave properly.. and they screw and confuse things up.

I've spent a lot of time on this pig... and mostly not getting anywhere fast (I don't mean just with sockets) This is a case where I need something "custom" because I couldn't find it on the net.. so I had to write it myself.. I think I did pretty darn GOOD learning it on my own (and the man/docs for TCL really do SUCK. PHP man/docs were much-better laid out) see? I orginally did it in PHP- but then had to write to the drive- and send over parts for eggdrop and update the webite, display results in chat- but only cirtain ones, and yadda yadda yadda.. in the end, I was running PHP, python, C, and TCL for this single procedure on a virtual host! it was DIRRRRRRRRRRR-TY!

In short, theres a lot of .mrc scripts out there- that read song titles and display them in chat channels.. sure! but NOTHING that reads them through eggdrop itself, reads to a remote server ,runs from a remote server, and NOTHING that can read ICECAST... and I see the reasons WHY.. I achieved all of this- and a whole lot MORE! I figured it all out.. even a "hop detector" to keep the stats correct when mounts switch over.. I have a "auto switch" that reads the local playout system (not winamp) and interjects its titles in when needed and a control system WITH FEEDBACK (it doesn't flood the local network with repeated/ignored commands) so that eggdrop can start and stop the local playout system whenever a remote DJ logs into the system- or, if one of my system admins needs to start, stop and even SECLECT and play tunes REMOTELY...

Yes, I have come a long way.. but I try to be thorough- I don't NEED the socket timeout.. but I know if I don't, I'll run into a problem- if/when I lose net- and my stupid WiMax connection FAILS while eggdrop is "talking to" the playout system- the result is cathostrophic- eggy hangs, then falls out of chat.. once that happens, admins LOST control... yadda yadda yadda...

I'm sure you have worked on a project before where you've spent hours and hours and became very very fustrated.. What I did wrong here, was I waited untill I was EXTREMELY fustrated before I came asking for help.. I shoud have come eariler- when I may have been more "reasonable" in my terminiology and understanding/explanations... I DO appreciate your help! please don't get me wrong, I'm not scolding you- rather I'm scolding myself at this point.. this banter is here just to give you some understanding as to why I'm appearing like an idiot!

now, its a new day, and I got myself some sleep... I'm going to read up on "filevent" and see if I can understand the Docs.. hehe I do understand why I can't read the socket- and I might have to open the socket twice- once in async mode to check/test then in blocking to read it (if its present) thats a tad inefficient, however, so I'm gonna make the attempt to "do it right"... if you're interested in this project, I'd be happy to demonstrate it.. (assuming I can get it working properly in time heheh)

again, thanks for your help! I appreciate your support and trying to be a good student

-DjZ-

[nml375]

The point being here...
Sockets do not need to be connected at both ends to exist...

The interpretations of your result was based on the code snippet you posted as reference, which used asynchronous connections. In the case of -async, the socket command will only yield an error if the address/port is invalid or the network layer isn't available (not the same as unplugging the network cable). In the case where it is unable to establish the connection, or yet not succeeded in that, the socket is still created - hence a "good" result. That is, the socket was successfully created.

If you do not want your socket command to block, you'll need asynchronous sockets. In order to use them, you need to understand the nature of these.

[nml375]

Btw, have you put any more thought into using the http package?

A quick sample you should be able to build upon is something like this:

Code: Select all

package require http

proc racInfo {} {
 set url "http://${::RacIP}:${::RacPort}/x/playing.cgi"
 ::http::config -useragent "Mozilla"
 ::http::geturl $url -command gotInfo -timeout 5000
}

proc gotInfo {token} {
 switch [::http::status $token] {
  "ok" {
   set ::RacInfo [::http::data $token]
   set ::DetO "onair.gif"
   ::http::cleanup $token
  }
  "eof" {
   set ::DetO "onair.gif"
   set ::RacInfo ""
   putlog "Recieved no data from playing.cgi"
   ::http::cleanup $token
  }
  "error" {
   set ::DetO "offair.gif"
   putlog "Unable to retrieve playing.cgi: [::http::error $token]"
   ::http::cleanup $token
  }
  "timeout" {
   set ::DetO "offair.gif"
   putlog "Unable to retrieve playing.cgi: Request timed out"
  }
  "reset" {
   putlog "Unable to retrieve playing.cgi: Request aborted by user"
   ::http::cleanup $token
  }
 }
}

Edit: Fixed minor typo (missing ").
Added support for additional states "timeout" and "reset".
Fixed an issue where calling ::http::cleanup in an timeout condition would cause an error in ::http::wait. Reduced timeout to 5secs.

[dj-zath]

Now that looks more enticing!

I have thought of it, yes.. wasn't sure if I have it or where to get it though.. (the http package)

your example there is what I needed- something I can work with and it looks like you got things right too!

I could always open the RAC for you to try out if you were really daring!

of course, I then recommend direct comm through my chatroom.. (since you woud need me to unplug during those states/tests)

• serv: irc.warp-radio.com
  chan: #WARP-Radio

just let me know!

and again, thanks for your assistance! I appreciate it a lot!

-DjZ-


UPDATE:

I plugged in the example.. fixed:

"racInfo" to "RacInfo"
"DetO onair.gif" to "DetO offair.gif"
added "global RacIP RacPort"
I got:

• [17:42] Tcl error in script for 'timer1':
  [17:42] extra characters after close-quote

I since undid the changes- (reloaded orginal script)

I assume I don't have the required libs/package.. I'll study the thing more nevertheless...