This is just what my guts tell me, from this thread and earlier ones..
You are pretty much trying to do way too much at the same time. I know you have a lot of code that's triggered by 5sec utimers - and I could see the code you pasted taking several seconds to complete under certain conditions. If you're doing alot more like this, I could easily see how your scripts could keep your eggdrop so occupied starting new processes and stacking up more timers, that your eggdrop ends up "unresponsive" - or perhaps "occupied" would be a better word... Frankly, I'm surprised that your eggdrop barely manages to stay online for any period of time...
So, as for limits..
The main limit within eggdrop, is that while it is doing one thing, it will not be able to take any other actions. Timers and triggered bindings will instead be queued and handled as soon as the current task is completed. In general cases - that would mean whenever your proc has completed.
So, if you have a proc that takes 5s to complete, that means during those 5 seconds, your eggdrop will not be responsive. Should you, at the start of calling this proc, create a timer with 5s or shorter duration - that will call this proc again, the new timer would've expired when the first invocation of the proc has completed, and the new one would be instantly executed. In essence, leading up to an infinite loop where your eggdrop hardly does anything else except running this proc over and over...
There's no problems what so ever (perhaps except limitations enforced by the hardware or kernel) to load your eggdrop with script code exceeding 10-15,000 lines. There isn't even any problem having individual procs extending huge number of lines, or nesting deep recursive calls. What you have to be wary of, is runtime. That is, how long will it take for this "entity" of code to complete, and let the control be handed back to eggdrop (sort of).
If you've got a lot of code like that loop doing 100 exec's in your script, then I really suggest you rethink your whole approach, starting from scratch. If you are coding it that way, you'll constantly run into these kind of problems over and over...
I would also recommend that you utilize the tcl "time" command to profile the time usage of your various code snippets.
This is the new home of the egghelp.org community forum.
All data has been migrated (including user logins/passwords) to a new phpBB version.
For more information, see this announcement post. Click the X in the top right-corner of this box to dismiss this message.
All data has been migrated (including user logins/passwords) to a new phpBB version.
For more information, see this announcement post. Click the X in the top right-corner of this box to dismiss this message.
[SOLVED] Large TCL scripts, RAM and Eggdrop...
Hi nml!
thanks for your reply!
You are CORRECTin figuring out what the code does.. you should have also noticed that I don't expect any return of input from the external link.. also, the "catch" basically ignores any fault or timing issue- in turn, this part of the script simply sends out 250 calls to the external link- do or die.. if the external link is ready, all the better, but, if not, no harm done! (the pre-condition elements helps to determine that the external link is "good" before the subroutine is ran anyhow) THAT part of the script runs FLAWLESSLY! To note, this part of the script will only run if a specific set of circumstances exists.. these are:
1.. the studio encoder is online AND streaming. ($DetSA) AND...
2.. The Playout Box is online ($DetOT) AND the Playout Cache is empty($DetPL)
3.. The Mode is set to "Auto Play" ($DetAP)
when these conditions are met, then the script will send 250 (in the latest version) "add track $PlayRand to the Playout cache" commands to the remote link and then "start playing"....
The code is quite ingenious, actually.. its smart enough to accept feedback on some parts (when required) once the commands are sent, the code will now STOP sending the commands since $DetPL will now be "on" and the conditions are no longer met.. the code will be ignored untill the 250 entries are dispatched!
about the timing issues...
NOW you're understanding my delimnea!
For this, I call external commands such as "bot.fch" which is merely the "fetch" command localized just for eggdrop.. the command strings/switches in the call are set up to have very short timeouts and such.. that I have planned the process thread to take 500 ms to complete.. though I give it (up to) 5 seconds to complete.. (I have sucessfully tested it to run at 1 second intervals.. but then a different problem occurs where icecast itself won't close its connections! that problem still exists, however its a matter of 5 to 20 hung connections versus 60 to 500! why this is? I haven't really determined yet! perhaps something to do with my ISP, Fedora and the way their router handles the connections through its network.. I have the eggdrop using an external fetch to get the info I want then to "drop" the connection- if it closes or not.. I let the Fetch command deal with the connection timout/hung issues... I merely had to come up with a soolution! Remember when you had me trying to use the http package and stuff? I could NOT keep eggdrop from hanging on EVERY connection it opened... and some indefinately so, the solution was to let FETCH open the connections- and let eggdrop continue on its merry way.. and that works PERFECTLY! no more hung eggdrop! the Bot doesn't even fall out of the channel or fall off IRC.. EVER.. (though the connections still continue to open and stay open....)
As for "having the bot do too much".. hmmmmm "limit" comes to mind again. I have been running the eggdrop- all this time.. and perhaps I DO have it doing too much.. LOL! You're right that its doing a LOT.. from managing the chatroom to rebooting the server at 6 am. .. from switching in the DJ feeds to generating all the dynamic elements of the website.. to even allowing listeners to search the arichive and to place their requests in the playlist for play. Yeah, that eggie's doing a lot!
I have refined the code once again.. and I have the eggie working well with all the code except the search and request.. I took that out untill I can come up with a better way to retrieve results without a parser entry for every line of text.. its massively NASTY! and I think having 500+ lines of parsers might be too much! was that the culprit? not likely since it WAS working before.. with the code.. the problem came in once I added the image-handling stuff.. and then it broke.. now, the image handler is in place and all is working nicely and fast... I have taken a break from it all- and even took my website down for the time being.. (like you said eariler) for I have been REALLY fustrated- not just with the eggdrop, but a lot of stuff.. but thats going beyond the scope of this thread...
if you're really interested in seeing the whole code, I could pop it up somewhere for you.. though I don't want to release it to the public just yet I'm planning to make a "light" version for the playout system to put on its forums.. but this version now has some "trade secrets" in it.. I have put a lot of time and effort into it.. its not quite hap-hazzardly thrown together.
-DjZ-
thanks for your reply!
I used to get LOTS of timer drifts when trying to use an internal command set to call the external link.. however, I also use catch to either "get or no-get" and this seems to have helped emmensely..I do notice that you've got a loop in your code that iterates 100 times, each time launching an external application that fetches some remote webpage. Due to the single-threaded nature of eggdrop (regardless of whether tcl is threaded or not), your eggdrop will "freeze" until all nested calls (if any) has completed. Especially, launching an external application takes a lot of resources compared to the other code, and doing it this way could very wellcause your eggie to become very sluggish... Do you get any "Timer drift" log-entries?
You are CORRECTin figuring out what the code does.. you should have also noticed that I don't expect any return of input from the external link.. also, the "catch" basically ignores any fault or timing issue- in turn, this part of the script simply sends out 250 calls to the external link- do or die.. if the external link is ready, all the better, but, if not, no harm done! (the pre-condition elements helps to determine that the external link is "good" before the subroutine is ran anyhow) THAT part of the script runs FLAWLESSLY! To note, this part of the script will only run if a specific set of circumstances exists.. these are:
1.. the studio encoder is online AND streaming. ($DetSA) AND...
2.. The Playout Box is online ($DetOT) AND the Playout Cache is empty($DetPL)
3.. The Mode is set to "Auto Play" ($DetAP)
when these conditions are met, then the script will send 250 (in the latest version) "add track $PlayRand to the Playout cache" commands to the remote link and then "start playing"....
The code is quite ingenious, actually.. its smart enough to accept feedback on some parts (when required) once the commands are sent, the code will now STOP sending the commands since $DetPL will now be "on" and the conditions are no longer met.. the code will be ignored untill the 250 entries are dispatched!
about the timing issues...
NOW you're understanding my delimnea!
For this, I call external commands such as "bot.fch" which is merely the "fetch" command localized just for eggdrop.. the command strings/switches in the call are set up to have very short timeouts and such.. that I have planned the process thread to take 500 ms to complete.. though I give it (up to) 5 seconds to complete.. (I have sucessfully tested it to run at 1 second intervals.. but then a different problem occurs where icecast itself won't close its connections! that problem still exists, however its a matter of 5 to 20 hung connections versus 60 to 500! why this is? I haven't really determined yet! perhaps something to do with my ISP, Fedora and the way their router handles the connections through its network.. I have the eggdrop using an external fetch to get the info I want then to "drop" the connection- if it closes or not.. I let the Fetch command deal with the connection timout/hung issues... I merely had to come up with a soolution! Remember when you had me trying to use the http package and stuff? I could NOT keep eggdrop from hanging on EVERY connection it opened... and some indefinately so, the solution was to let FETCH open the connections- and let eggdrop continue on its merry way.. and that works PERFECTLY! no more hung eggdrop! the Bot doesn't even fall out of the channel or fall off IRC.. EVER.. (though the connections still continue to open and stay open....)
As for "having the bot do too much".. hmmmmm "limit" comes to mind again. I have been running the eggdrop- all this time.. and perhaps I DO have it doing too much.. LOL! You're right that its doing a LOT.. from managing the chatroom to rebooting the server at 6 am. .. from switching in the DJ feeds to generating all the dynamic elements of the website.. to even allowing listeners to search the arichive and to place their requests in the playlist for play. Yeah, that eggie's doing a lot!
I have refined the code once again.. and I have the eggie working well with all the code except the search and request.. I took that out untill I can come up with a better way to retrieve results without a parser entry for every line of text.. its massively NASTY! and I think having 500+ lines of parsers might be too much! was that the culprit? not likely since it WAS working before.. with the code.. the problem came in once I added the image-handling stuff.. and then it broke.. now, the image handler is in place and all is working nicely and fast... I have taken a break from it all- and even took my website down for the time being.. (like you said eariler) for I have been REALLY fustrated- not just with the eggdrop, but a lot of stuff.. but thats going beyond the scope of this thread...
if you're really interested in seeing the whole code, I could pop it up somewhere for you.. though I don't want to release it to the public just yet I'm planning to make a "light" version for the playout system to put on its forums.. but this version now has some "trade secrets" in it.. I have put a lot of time and effort into it.. its not quite hap-hazzardly thrown together.
-DjZ-
The use of catch makes no difference here, neither that you run the external program in background. The issue is the time it takes for tcl to spawn a new process environment. Using the exec command, irregardless of what you are actually executing, is very time- and resource-consuming. When you are doing this 100 times in a row, this could extend to several seconds...
Since you instruct exec to pipe all outputs to /dev/null and force the application into background, what parameters you send to fetch does not matter at all. The exec command would probably have returned before fetch even have had time enough to parse the URI, let alone try to access it.
I actually time'd a similar loop (using wget to replace fetch) which also launched the external application into background and immediately returning control to tcl.. 100 launches averaged at 3½ seconds.
The big problem is not that your eggdrop has a lot to do, but you are trying to make it do it all at once, and most likely way too often... Once again, I suggest you make use of the time command to profile the time usage of your different procs. Whenever there's a proc taking a second to complete, there's probably room for a lot of improvements, or you'd perhaps should consider taking these actions slower.
A remote example might be a script reading a file of several hundred lines, taking one action (of 500ms) for each line. If you were to do this in a loop (think along "while {![eof $fd]} {gets $fd data; parse $data}"), the application would freeze for minutes.. In this case, having the event engine fire up a proc that just reads one line and parses that line, and then return, would allow the event engine to do other tasks inbetween parsing lines. Doing the whole parsing of the file would probably take a little longer, but your application would still remain responsive during the parsing.
And yes, I do remember your issues with the http-package. That was well-diagnosed to the package using synchronous connections even in the "callback-mode", and would only occur if the connection-attempt is neither accepted or refused. This is something I'd call a bug in the http-package, as the behaviour does not comply with the documentations..
Btw, I'm not sure what you are talking 'bout with "catch" ignoring timing issues... Catch will only trap error return codes from executing the "script" parameter, nothing else.
Since you instruct exec to pipe all outputs to /dev/null and force the application into background, what parameters you send to fetch does not matter at all. The exec command would probably have returned before fetch even have had time enough to parse the URI, let alone try to access it.
I actually time'd a similar loop (using wget to replace fetch) which also launched the external application into background and immediately returning control to tcl.. 100 launches averaged at 3½ seconds.
The big problem is not that your eggdrop has a lot to do, but you are trying to make it do it all at once, and most likely way too often... Once again, I suggest you make use of the time command to profile the time usage of your different procs. Whenever there's a proc taking a second to complete, there's probably room for a lot of improvements, or you'd perhaps should consider taking these actions slower.
A remote example might be a script reading a file of several hundred lines, taking one action (of 500ms) for each line. If you were to do this in a loop (think along "while {![eof $fd]} {gets $fd data; parse $data}"), the application would freeze for minutes.. In this case, having the event engine fire up a proc that just reads one line and parses that line, and then return, would allow the event engine to do other tasks inbetween parsing lines. Doing the whole parsing of the file would probably take a little longer, but your application would still remain responsive during the parsing.
And yes, I do remember your issues with the http-package. That was well-diagnosed to the package using synchronous connections even in the "callback-mode", and would only occur if the connection-attempt is neither accepted or refused. This is something I'd call a bug in the http-package, as the behaviour does not comply with the documentations..
Btw, I'm not sure what you are talking 'bout with "catch" ignoring timing issues... Catch will only trap error return codes from executing the "script" parameter, nothing else.
NML_375
hi nml and thanks for your reply!
In reading your last post, I understand what you are saying.. let me see if I can explain why I'm doing all this "in a straight line" and not "multiple processing".
In fact, the things you mentioned above is exactly the reasons I have to run things the way I do...
Orginally, when I started writing this thing, one of the problems I ran into was a case where one proc was running while another proc which the first proc "relied" on was not yet finished.. this resulted in irratic if not errored output.
example:
the "logic" proc running before the "parsers" procs finishing.
This most ALWAYS ended up with either an "online" state reporting to be "offline" or the proc crashing- reporting a "missing or non-existant varable" Although, I have ended up having to run a proc called "SetVars" which sets all the globals to their "offair"/offline state and to have this proc run first thing at startup, to help guarentee that the varable will never be missing in any state, I could not prevent one proc from eventually "catching" another proc in the "constraint" mode between states.. this was the case especially if the connection was "less than perfect" (and we both KNOW how reliable the internet is these days!). In the end, I was ending up with the eggie "freaking out"; setting "broadcast onair" then "broadcast offair" then back "onair" and then "offair", etc,etc,etc... (yes DSL is a wonderful thing... NOT!)
Another problem I had was that the eggie was "slamming" the connection with more than one connection request all at the same time.. being that the "return" path is over DSL.. I found this was causing the "concurrent connection limit" (which is three) to be exceeded- resulting in a dropped connection- and thus a mis-state onair/offair/onair/offair loop.. again results.
To combat this, I decided to run the parsers and THEN run the logic; all in the same proc.. and thus, one-state-at-a-time which helped to reduce the logical "snarfu" of catching one proc in constraint.
another trick was to put in a "change detector" which consisted of a "state read/write and compare" operation:
This example will read $Show and compare it to $Cash01.. if it matches, the proc is then ended and the operation is ignored.. this helps in the case where a "missed" connection occurs by requiring TWO cycles to run before a change is detected (beit another one at the start of the proc and another from within the proc itself). The only disadvantage to this approach, however, is I have to generate a global just to "hold" the state of $Cash01 (in fact $Cash01 itself) now imagine a bunch of these amongst the ranks of the procs.. yes- they pile up quickly! (one of the reasons I have so many globals.. because I have a lot of these!)
Now, I DO have multiple procs, of course, but the "main" proc contains the "get info", then the parsers and then the logic. Then, theres the output control which is another proc that contains the topic/song and website updators and server title stream setters.. Then comes the procs for the key binds and control commands and all that... not to mention the login proc and the "setVars" proc.. and finally the search and request procs and other misc procs for greets, stats and other things... and not to forget the run loops... its quite a busy little thing! LOL
-DjZ
In reading your last post, I understand what you are saying.. let me see if I can explain why I'm doing all this "in a straight line" and not "multiple processing".
In fact, the things you mentioned above is exactly the reasons I have to run things the way I do...
Orginally, when I started writing this thing, one of the problems I ran into was a case where one proc was running while another proc which the first proc "relied" on was not yet finished.. this resulted in irratic if not errored output.
example:
the "logic" proc running before the "parsers" procs finishing.
This most ALWAYS ended up with either an "online" state reporting to be "offline" or the proc crashing- reporting a "missing or non-existant varable" Although, I have ended up having to run a proc called "SetVars" which sets all the globals to their "offair"/offline state and to have this proc run first thing at startup, to help guarentee that the varable will never be missing in any state, I could not prevent one proc from eventually "catching" another proc in the "constraint" mode between states.. this was the case especially if the connection was "less than perfect" (and we both KNOW how reliable the internet is these days!). In the end, I was ending up with the eggie "freaking out"; setting "broadcast onair" then "broadcast offair" then back "onair" and then "offair", etc,etc,etc... (yes DSL is a wonderful thing... NOT!)
Another problem I had was that the eggie was "slamming" the connection with more than one connection request all at the same time.. being that the "return" path is over DSL.. I found this was causing the "concurrent connection limit" (which is three) to be exceeded- resulting in a dropped connection- and thus a mis-state onair/offair/onair/offair loop.. again results.
To combat this, I decided to run the parsers and THEN run the logic; all in the same proc.. and thus, one-state-at-a-time which helped to reduce the logical "snarfu" of catching one proc in constraint.
another trick was to put in a "change detector" which consisted of a "state read/write and compare" operation:
Code: Select all
proc update topic {} {
global Cash01 Show MyChan;
if {($Cash01 != $Show)} {
Set Cash01 $Show;
putserv "TOPIC $MyChan :Now Online with $Show";
};
return 0;
};
Now, I DO have multiple procs, of course, but the "main" proc contains the "get info", then the parsers and then the logic. Then, theres the output control which is another proc that contains the topic/song and website updators and server title stream setters.. Then comes the procs for the key binds and control commands and all that... not to mention the login proc and the "setVars" proc.. and finally the search and request procs and other misc procs for greets, stats and other things... and not to forget the run loops... its quite a busy little thing! LOL
-DjZ
UPDATE:
I have rewritten the TCL code... and the bot seems to be as fast as ever.. I have refined some things.. and eliminated about 600 lines of code.. (though I still haven't put the help or AI system back in yet) the Search and request System, along with the icecast/RAC parsers and switching logic has been revamped and some parts were completely rewritten.
Theres still some "minor bugs" in how the switching logic functions handle the graphics aspects.. when a graphic isn't available, it will display a "null" graphic instead of the current DJ's graphic.. this isn't a problem, though, just a different way of doing something... so I'm not worried about it.. (you can see this first-hand on my myspace page http://www.myspace.com/warpradio )
Thanks everyone for all the suggestions and help!
for those who wish to see this thing in action, come see me at irc.warp-radio.com:6667 channel #WARP-Radio and I'll be happy to demonstrate it for you!
oh, heres one for you..
how about a "remote refresh" for pages without using an interval or iFrame? preferbly PHP...
-DjZ-
I have rewritten the TCL code... and the bot seems to be as fast as ever.. I have refined some things.. and eliminated about 600 lines of code.. (though I still haven't put the help or AI system back in yet) the Search and request System, along with the icecast/RAC parsers and switching logic has been revamped and some parts were completely rewritten.
Theres still some "minor bugs" in how the switching logic functions handle the graphics aspects.. when a graphic isn't available, it will display a "null" graphic instead of the current DJ's graphic.. this isn't a problem, though, just a different way of doing something... so I'm not worried about it.. (you can see this first-hand on my myspace page http://www.myspace.com/warpradio )
Thanks everyone for all the suggestions and help!
for those who wish to see this thing in action, come see me at irc.warp-radio.com:6667 channel #WARP-Radio and I'll be happy to demonstrate it for you!
oh, heres one for you..
how about a "remote refresh" for pages without using an interval or iFrame? preferbly PHP...
-DjZ-
Nice to hear you got it working again.
One suggestion for improving your 100-250 url fetches.. Let your tcl-script write a single bash-script to do the large number of fetches, then launch this script into background (using exec). This way, your eggdrop ends up doing a single call to exec, and once the child process is created, it should hand over the control back to the script, leaving the task of spawning 100-250 child processes to the child...
I'm not sure what you are referring to with "remote refresh"..
If you are talking about forcing the client browser to reload the page, there's the meta http-refresh tag: <meta http-equiv="refresh" content="10" />
Another option would be to use some JavaScript with timers. Plenty of examples on how to do this spread all over the web.
You could use PHP to generate the needed tags/javascripts, though in most cases this could just be static text. Keep in mind, that once the page has been generated and the http-transaction has been completed, the PHP-script will terminate, so the PHP-script can't "tell" the browser 1 minute later that it should do this or that..
One suggestion for improving your 100-250 url fetches.. Let your tcl-script write a single bash-script to do the large number of fetches, then launch this script into background (using exec). This way, your eggdrop ends up doing a single call to exec, and once the child process is created, it should hand over the control back to the script, leaving the task of spawning 100-250 child processes to the child...
I'm not sure what you are referring to with "remote refresh"..
If you are talking about forcing the client browser to reload the page, there's the meta http-refresh tag: <meta http-equiv="refresh" content="10" />
Another option would be to use some JavaScript with timers. Plenty of examples on how to do this spread all over the web.
You could use PHP to generate the needed tags/javascripts, though in most cases this could just be static text. Keep in mind, that once the page has been generated and the http-transaction has been completed, the PHP-script will terminate, so the PHP-script can't "tell" the browser 1 minute later that it should do this or that..
NML_375
hi nml!
thanks again for the sound advice...
just to iterate a little...
the "call to the 250 connections" only happens on a specific set of circumstances and I have been watching it "communicate" between the playout box and the eggie and have found this operation completes in under 2 seconds (at most) even if I unplug the box from the network.. the script won't hang.. thats good! but not where the problem was coming from!
I THINK the problem was coming from the search and request system- which did a simular thing- but this time it had to GET the results, and then PARSE the results and then display them in an IM to the party who init'ed it! The problem is that only 10 results would display per call (yes that had to all be parsed and displayed) and then another set of 10, and so on and so on... now here comes johnny-user.. who decides he wants to look up "tainted love" while suzie-user is currently searching for "right here waiting"... now johnny looks up "the" and receives a listing for EVERY song or artist with "the" in the string! being theres 3800+ tracks... OUCH!
Now, of course, I have put in a mean limit of 50 tracks max PER search query AND a time limit to wait between the searches.. well there was still 1000 lines just in the PERSERS alone!
I have completely redid the entire S&L system and its considerably more lean and mean now.. the bot isn't hanging up on all the parsing, well, actually, it does still have to parse (and reparse) up to the max limit of 50 times based on what ppl search for... but its now only 160 lines or so instead of over 1000 ...
but, now I got a new one... I'll start a new thread for this one...
-DjZ-
thanks again for the sound advice...
just to iterate a little...
the "call to the 250 connections" only happens on a specific set of circumstances and I have been watching it "communicate" between the playout box and the eggie and have found this operation completes in under 2 seconds (at most) even if I unplug the box from the network.. the script won't hang.. thats good! but not where the problem was coming from!
I THINK the problem was coming from the search and request system- which did a simular thing- but this time it had to GET the results, and then PARSE the results and then display them in an IM to the party who init'ed it! The problem is that only 10 results would display per call (yes that had to all be parsed and displayed) and then another set of 10, and so on and so on... now here comes johnny-user.. who decides he wants to look up "tainted love" while suzie-user is currently searching for "right here waiting"... now johnny looks up "the" and receives a listing for EVERY song or artist with "the" in the string! being theres 3800+ tracks... OUCH!
Now, of course, I have put in a mean limit of 50 tracks max PER search query AND a time limit to wait between the searches.. well there was still 1000 lines just in the PERSERS alone!
I have completely redid the entire S&L system and its considerably more lean and mean now.. the bot isn't hanging up on all the parsing, well, actually, it does still have to parse (and reparse) up to the max limit of 50 times based on what ppl search for... but its now only 160 lines or so instead of over 1000 ...
but, now I got a new one... I'll start a new thread for this one...
-DjZ-