[SOLVED] Large TCL scripts, RAM and Eggdrop...

[dj-zath]

Hi again everybody!

If anyone ever wondered, yes, I DID get that search and request thing going!

but I recently ran into an interesting problem...

I tried to add a new piece of code and eggdrop "freaked out".. code elements started "dissappearing" and, of course, the bot didn't know what to do!

the total code base at this point is over 3000 lines.. and when I tried to add 20 more lines, it all just came to a screetching HALT!

the system has 2 Gigs RAM and 2 Dual-Core Xeon CPUs (4 CPUs) running BSD 6x

my question is :

"Is there a defined LIMIT to how much code can be implemented or some hard limit to the amount of processes that eggdrop can handle?"

being, that TCL is single-ended, I didn't think this would be a problem so soon (I can see 20,000 lines of code.. but then the thing would be SENTIENT at that point!).

the first signs of "trouble" appear to be TCL/eggdrop not catching some errors.. such as missing globals or an "if" statement with an open ended equation.. that happens at about 1300 lines or so...

I have tried (re)writing the code to be as efficient as I can possablly understand.. ("elseif"s and arrays can be a real bitch!)but even after spending days and days checking every single line, one-at-a-time.. and finding nothing wrong... when its all put together, the weirdness begins!

I also learned that nested "if"s are a BAD idea.. and to avoid them as much as possable.. but theres a place in my code that MUST have 'em... and you can only go 3 levels down.. any more and you start running into some strange behaviors! but, perhaps its all related.. I also wonder if theres a limit of the number of globals a single proc can have (one proc has about 60... and they are in one long straight line!)

I have placed remarks all through my code and kept a VERY close eye on the number of varables I have used and even recycled them wherever possable.. (yes, I have checked, checked, and rechecked to make SURE there were no varable conflicts)

everything all worked.. untill I added ONE more "if" statement set and it all just grinded and crashed!


any ideas?

for now, I'm going to try a different approach... though I kinda recall not being able to break the code up into seperate files due to the globals that share from proc to proc...

this bot of mine does everything from managing the DJ logins, switching mount title assignments, updates for all the title servers... to generating the website!

-- saved 3261 lines --

-DjZ-

[raider2k]

nothing I heard of or seen so far,
anything in logs? debug/core file being created?
eggdrop properly compiled?
another - but far assumption - Id say could be the harddisc

and btw: seen lots of different code over all the years, code that features 10.000-30.000 lines, code that contains a few hundred vars as well as global vars and udef flags, etc .. dont think theres some kind of code which would cause the eggdrop to run wild (as long as the code is working properly of course)

[dj-zath]

thanks Raider! thats what I thought.. I didn't think it would break THIS early on...

I wasn't sure if it was an eggdrop issue- though I didn't think it was, rather more of an issue with the TCL shell handler itself.. I don't see any errors in the logs.. just a sluggish-responsive "debug" mode (to say running the bot with -n shows SOME signs that its struggling with the code somewhere)

I have begun trying to split up the code into seperate files.. and already seeing problems with varables from one proc in one file not being seen in another proc in another file (I had this problem before very early on using source... learned it works only when its all contained in the same file)

Hard drive issues? hmmmmm....

They are new drives.. new machine actually (but its proven to be a lemon in some respects..)

eggdrop not compiled properly?? .. thats possable, since I had a HELL of a time getting eggdrop to compile at all on BSD 6 in the first place.. I did manage to do it, however, I don't remember quite exactly what I did to get it to work.. I think I had to install it from the ports, then reinstall it "locally" without uninstalling it from the ports.. then back up what was there, then delete the local installation AND the ports installation then put the backup on and run that.. something like that... in the end, I ended up with a 1441994 (1.5 meg) eggdrop exec file that loads without any noticable errors and SEEMS to be okay Previous attempts have resulted in in LOTS of compiling errors and corrupted bot compiles!

I have ran this bot and it seems to be okay not showing any errors... anyways.

thanks for the input though! thats what I thought and needed to know.. I'll assume, then, that I made a mistake somewhere.. I'm not THAT great with TCL.. learing as I go along.. and reading posts from this forum helped a lot.. not to mention nml and TK and many others...

at least its not Fedora! OMG Fedora was a NIGHTMARE!

-Djz-

[raider2k]

hmm ..
honestly I have to admit that im not familiar with bsd in any way .. im running debian lenny, latest release, 64bit, on a 2gigs ecc, dual opteron machine - fine, very fine.

but .. Ive had a different system in the past which apt to get stuck once in a while (actually too much in my opinion) with no errors at all in syslog, kern.log or messages, went through noapic nolapic acpi=off and nosmp as well, played with bios settings, used 3 different harddiscs, one of them a brandnew, deepscanned harddiscs for errors, set up debian about 3 or 4 times a week ago, put in a new psu, new vga card, no error, no rational solution .. the not-so-proper-solution: I changed entire system and it works again like its supposed to work.

another but ... even if you run an eggdrop which has been compiled on a different system it should work .. should .. or might .. most probably it will work - but like I said, Im NOT 100% sure about bsd or any non-debian, non-gentoo distribution.

downloading the tar.gz file from egghelp.org and do it as shown in "setting up your eggdrop" area didnt work out?

[nml375]

Posting on a mobile device at the moment, so I'll be brief..

Limits in general:
Unless you've compiled your eggdrop with memory debugging enabled, there'd be no hard limits - only performance degradation as additional hooks into the various bind tables, functions, namespaces, and variable objects are registered within the tcl environment..
As I recall, tcl stores the above mentioned objects in rather optimized storages, so the performance degradation even with a huge number of objects should be limited.

Namspaces and files:
Neither eggdrop nor tcl makes any difference what-so-ever from which file (or any other script-code source for that matter) a variable is defined in globalspace. If it's created in globalspace, then it is in fact global to the interpreter environment, just like any globally defined proc...

Nested if's:
The limit for nested calls should in no way restrict three-four nested ifs... I've had recursive calls go beyond 1000 levels... Biggest risk with nested ifs is that you (as a coder) risk loosing track of which level you're currently at.

My experience with amd64 is rather limited, but I've ben given the impression that it can be rather cumbersome getting eggies compiled and running on some of these systems..

Fedora:
I've had eggies compiled and running just fine under various FC dists/systems. Many dists (not only fedora) comes with tcl8.5 precompiled with threads enabled. This will cause a well-known issue where the library falsely is detected as non-threaded. Fix is posted all over this forum...

All in all:
Double-check your scripts, as the source for most of your problems is most likely errors in your syntax or logic.
Feel free to post a link to the sources of your scripts if you ned help debugging.

[speechles]

the total code base at this point is over 3000 lines.. and when I tried to add 20 more lines, it all just came to a screetching HALT!

"Is there a defined LIMIT to how much code can be implemented or some hard limit to the amount of processes that eggdrop can handle?"

I know of a certain script I've adapted to well over 7000+ lines with more globals than you can shake a stick at encountering no problems what so ever. It's a behemoth weighing in at more than 340KB. This script can be found here. It works on a variety of platforms utilizing either windrop or eggdrop and never once have I heard of any complaints about it crashing a users system, yes even BSD flavors play nice. This is why it sounds like you have more problems than poorly nested if's. It sounds like you may have missing double-quotes in places which cause the interpreter to fault over things. The main problem with nesting if's isn't how many levels deep you've went. It's the logic behind why these if's even need to be nested that deeply that becomes problematic. If you don't need them nested, consider using the [switch] command. The script above also has several spots with nested if/elseif/elseif/etc's in it (early amateur code left for legacy purposes, aka, it works and I'm too lazy to correct it.. haw), and again because the logic there is solid they work exactly as intended. What you may need to do is use a site which can parse your tcl and give you clues about syntactical errors, such as missing double-quotes and missing braces in pairs, missing both a } and a { in different spots. These issues don't crash your bot, these cause it to behave erratically.

What I suggest is you use http://paste.tclhelp.net/ and paste each procedure from your script here. Hit the preview button after your done and look for resulting errors and warnings the site generates. It will give you clues about what is wrong without having to disclose any code to others. This is how I solve these types of problems with complicated scripts. Break them down procedure by procedure and check them with that site until the offending code is discovered and can be corrected.

If after doing this, and everything appears fine than the issue is not one of grammar or syntax. The issue is clearly caused by logical evaluation errors. These can only be discovered by others (which requires you posting code as nml375 suggested) or by adding debug statements in complicated areas so these conditions can be checked and the logic verified. Most people do this with simple [putlog]'s and idling on their bot's partyline. If your not up to the arduous task of adding all this extraneous debug code the only option is to follow nml375's advice. We can only help your further at this point if you allow us to see the script in it's entirety.

[dj-zath]

hi everyone and thanks for the replies!

Many many good suggestions.. and I, in fact, have "taken it apart" more than once and have checkred the code down to the operators.. I use putlogs and putserv's (putlogs don't seen to parse the varables, but rather place them at their face value such as $DetHA instead of "onair.gif") for this I just "putserv" the varables to a test channel and check the logic "in place".

about the TCL library being "single threaded" YES I recall seeing this error somewhere.. but I think I corrected it when I reinstalled TCL... I can't recall now, though, sorry... (migraine starting up.. ughh) as I mentioned above, I recal having a bunch of problems with installing the eggie.. but, I can't recall exactly what those issues are at this point...

a brief history:

Before I ran my own server, I used Verio Virtual server shells.. they never had a problem with eggdrop.. but then Verio changed their policies and I had to drop them for having a bad backbone. I sinced tried running a Fedora Core 9 box from my ISP.. Fedora FC9 acted VERY strangly with TCL.. mainly, some of the operators weren't recongnized while others needed to be used this was the MAIN problem I had with trying to get the start loops running (previous postings on here) I didn't install Fedora, the ISP did... and it has some strange issues- not related to eggdrop, but effects eggdrop in ways- such as any sockets that were opened, STAYED open- even after the connection on the other side closed.. the "dead" sockets would pile up and eventually CRASH the box!.. after that I went back to BSD but this time 6x. where all the problems went away... (expect the ISP still uses Fedora on its side and connections through its proxy contimue to have the "sockets stuck open" issue.. thus requires me to reset the box once every 24 hours... (now, one of the functions the eggie does!)).

I have gone through the code over and over and over... one of the ways I am doing this is to create a new file and copy-paste over each line one-at-a-time this way I could double check to make sure brackets were in place and quotes and all that.. I do admit that the parsers for icecast and the RAC system are BRUTAL.. but I tried to make them as efficient as possable.. I spent a lot of time on this thing and it DOES run pretty smoothly... I have since rewritten the code once again in places.. as of the time of this writing, the bot is operational- all the code is in place including the new "20 lines of code" that crashed it before.. the exception is I omitted the search and request code though that didn't cause the problems.. in the end, it seems that if I added 20 lines of code, I have to take out some code elsewhere.. beit the image parsers, or the search and request logic or the hop detectors.. etc etc all three in place and the code is sluggish and unreliable.. and they have nothing to do with each other! this is why I asked about a limt- in case there was a limit- and I would be rewriting myself into oblivian!

about the "nested IFs" you're right, of course, about losing one's place and/or getting the logic out of order... has happened! the part in my code that uses it....

Code: Select all

if {($DetIA == "onair.gif")} {
        if {($DetPY == "onair.gif")} {
                catch {exec ./Bot.fch -q -A -m -T 1 -o /dev/null "http://$RacIP:$RacPort/x/playing.cgi?c=stop" >& /dev/null &};
        };
        set Image $MetaG;
        set Show $MetaT;
        set DJ $MetaDJ;
        set Song $MetaS;

} elseif {($DetHA == "onair.gif")} {
        if {([file exists "/usr/Rec/show.shw"])} {
                set DetRC "onair.gif";
        } else {
                set DetRC "offair.gif";
        };

        if {($DetAP == "onair.gif")} {
                if {($DetPL != "onair.gif")} {
                        set Song "Generating Playlist..";
                        set T 0;
                        while {($T <= 200)} {
                                set C [rand $PlayCount];
                                catch {exec ./Bot.fch -q -A -m -T 1 -o /dev/null "http://$RacIP:$RacPort/x/playlist.cgi?id=$C" >& /dev/null &};
                                incr T;
                        };
                };
                if {($DetPY == "onair.gif")} {
                        set Image $MetaPI;
                        set Song $MetaPY;
                } else {
                        set Image "/DJs/$botnick.jpg";
                        set Song "Starting Playout...";
                        catch {exec ./Bot.fch -q -A -m -T 1 -o /dev/null "http://$RacIP:$RacPort/x/playing.cgi?c=play" >& /dev/null &};
                };
                set Show "Automated DJ Playout";
                set DJ "$botnick";
        } elseif {($DetED == "onair.gif")} {
                if {($DetPY == "onair.gif")} {
                        set Image $MetaPI;
                        set Song $MetaPY;
                } else {
                        set Image "/DJs/DJ-Mental.jpg";
                        set Song "Live Insane Rantings";
                };
                set Show "Extentia Dementia";
                set DJ "DJ Mental";
        } else {
                if {($DetPY == "onair.gif")} {
                        set Image $MetaPI;
                        set Song $MetaPY;
                } else {
                        set Image "/DJs/DJ-Zath.jpg";
                        set Song "Live DJ/Audio";
                };
                set Show "Master Control";
                set DJ "DJ Zath";
        };
} else {
        set DetRC "offair.gif";
        set Image "--Not Available-- Master";
        set Show "Not Available - Master";
        set DJ "Not Available - Master";
        set Song "Not Available - Master";
}; 

I have tried many different ways of laying out this logic sequence.. what you see here isn't the most-efficient way of doing it, but its the ONLY way it seems to work with the least amount of "side effects" The side effect here is that $Image won't display the Bot's image when the bot is playing a track that doesn''t contain album art
1.. I CAN fix this by having a "null" image displayed
2.. I can live with it for now

the "logic snarfu" here is that the same states are repeated- but with slightly different results.. example: the Bots image displays for "automatic playout" but not when "master control" is on...

I didn't display this for debugging purposes. I was just giving you an example of what I have done..

part of the "20 lines" of code are contained in this piece- they are the lines that deal with the image handling.. if those lines are in there AND the "search and request system" and "image parsers" (rest of the 20 lines of code) the bot begins to act oddly and unreliably.. but take either one or the others(s) out and it works FINE.. it just seemed to not like having that many lines of code..

one of the next tricks will be for me to add 20 lines of "some other code" to see if its an issue...

one other thing, I said eariler about using 60 varables... I meant 60 globals all in the same proc!

and I exaggerated a little- perhaps not 60, but theres a LOT of them.. basically the "SetVars" proc that sets all the "$Det??"''s varables.. and EVERYthing on the server, website, RAC and cast system has at least one or more for some function and/or state!


-DjZ-

[nml375]

putlog vs. putserv and variables:
Since the variable substitution is done before either of these commands are invoked at all, there should be no difference as you describe it. The only thing that would prevent variable substitutions would be either escaping the dollar sign (\$varname), or enclosing the parameter with {} as opposite to using "".

I've never talked about "single threaded" tcl-libraries. The tcl-libraries are either compiled with threads enabled, or threads not enabled. The proper code for handling threaded tcl is implemented in eggdrop, there is only the issue of detecting threaded tcl-libraries with the v8.5 libtcl - which is easily handled once detected.

Your earlier posts:
I've yet to find any examples in your posts where you were forced to use certain commands under Fedora, but not under other systems; and having other commands being "not reccognized". The main issue in your thread "Strange Issues in TCL" was that you did not provide the proper arguments to your function calls in one place or the other. This would make no change whether you are running *BSD, Linux, Windows, or any other OS or kernels.
I did point out in that thread that some bindings have changed the arguments used when calling the associated function since the v1.3.18 eggdrop.

Stale sockets can be a resource hogger, regardless of the underlying OS/dist. This would however, usually be an issue within the client and/or server software, rather than an issue within the kernel. There could also be some improperly configured network hardware dropping the packets needed to notify the other end of the closed connection.

Regarding your script snippet; the logics are rather hard to follow, as I have no clues what so ever what any of those variables are supposed to indicate, or their values given any state. Thus it is rather hard to either comment your "side effects" or suggest improvements.
Only recommendation I could give would be to take a piece of paper and write a logics-table, then use this as a blueprint for your code...

20 lines of code:
What if you just add 20 lines doing nothing? Such as:
set i $i
Does your script still suddenly break for no good reason?

Globals:
I just created a proc that linked 1000 local variables to their corresponding globalspace variables. I had no problems getting it working either by using one global command for each variable, or one global command with a long list of variables...

[dj-zath]

Hi there nml and Speechless:

first off, thanks for the replies!

I guess I'm speaking in "broken idea" here- which would make things hard to follow from the outside world.. I apologize for that.. I didn't mean to take you on a roller-coaster only to end up either stuck in a tunnel- or to exit the ride dissatisfied.. that wasn't at all my intent.

I guess I was trying to say that I have been experiencing some strange behaviors with eggdrop, my server, my ISP, their routers, TCL, the compile and finally.... my code!

although, things seem to be functioning OKay, on their own, or as individual components.. but, as a set in this particular situation, they aren't operating as I recall they did when I ran a simular configuration on an entirely-different shell/OS.

of all of this, I didn't consider anything else BUT my code being the culprit- however, from the replies I received from this thread, perhaps the problem may be more "involved" that I initially anticipated.

You all probably think I'm some weird or mental dude whos just, plain, out of his mind.. and you probably aren't far from the truth there, but..I have to admit that when I don't fully-understand something, I can go into a sorta "panic" and I become extremely-flustered- expecially when I'm trying to understand the problem and explain it to everyone here.. and I just "freak out" and say things all out of context or exaggerated to a point where its just mangled.. and, reading back, I have done that here a few times allready!

with this all being said, perhaps I need to start over....

First off, I asked a question about there being a "hard limit" of some sort in TCL or the TCL handler (the thing that eggdrop uses to access the TCL libs) and I was told "no, there is no limit, at least within the scope of your problem".

now I'm going to carefully describe what I have experienced..and why I came in here thinking there was a limit..

if I load eggdrop into "debug mode", using -n, with just eggdrop's base configuration code plus some minor functions.. and I purposely put something in that should make eggdrop "bawk" (like using a quote without an escape) I would expect the bot to error- and the config file not to load, instead, return an error and return to prompt.. which it does.. as expected.

my code, on the other hand, won't do that- instead it will continue to load and never "catch" that error.. even if I deliberately put the error in! of course, the code won't work right.. if even at all.. but the bot will load nevertheless.. I understand the idea that perhaps theres another link of code that "cancels out" the inital errors.. but, though, likely for one, not for many in a row.. I have taken my code apart and carefully, line-by-line have repasted it into a new file and looked carefully at each line, making sure its clean and proper.. I'm not saying I'm an expert with TCL, no, not a chance.. and perhaps I could hire someone from here to look it over and give suggestions to what it needs in order to work right... but, before I could do that, I'll need to make sure the eggdrop and TCL compiles are right...

For now, I'll assume the code is at fault.. and, again, take it apart....

As for putlog{}, if I put (example):

Code: Select all

set $color "orange";
if {($color != "green")} {
     set $color "not green";
};

putlog {what is color? $color}

what I see in the log is:

"what is color? $color"

and not:

"what is color? not green"

for this, I have to "putserv" it to the channel in order to read the varable as it should be set/shown... why is this? I assume theres something wrong with the compile.. based on what I have read here in this thread. (I DO recall it working properly on the Verio box)

did that help explain things or have I hopelessly started another roller-coaster ride?!

-DjZ-

[arfer]

raider2k wrote

another but ... even if you run an eggdrop which has been compiled on a different system it should work .. should .. or might .. most probably it will work - but like I said, Im NOT 100% sure about bsd or any non-debian, non-gentoo distribution.

I would have thought at one time that the above statement was reasonable. I now know that you cannot be sure of this. My shell provider recently changed linux distro from Fedora to CentOS due to support issues. One of my scripts that previously ran flawlessly for many months simply ceased to function, whilst there was seemingly no effect on other scripts.

The script in question was http code and I suspected that it failed on a Tcl 'catch' command.

the first signs of "trouble" appear to be TCL/eggdrop not catching some errors

I can only say suspect because I spent relatively little time trying to prove it. I don't believe in coincidences and I therefore pretty quickly focused on the linux changes and completely recompiled/reinstalled the bot. The script functioned as expected again without any code changes.

[dj-zath]

Hi arfer!

I recompiled eggdrop fresh on Fedora and BSD the one that ran on the Verio box would not even load at all! error log returned "missing libs" errors..

-DjZ-

[raider2k]

compile fine, running eggdrop failed?
or when does missing libs errormessage appear?

[dj-zath]

hi Arfer:

Sorry for the long delay...

The problem isn't in the compile, I have a working eggdrop which was compiled on the box and OS in which it runs..

The problem is when I exec a seemingly large piece of code, eggdrop gets sluggish/laggy. The more lines I add to the code, the more laggy the eggie gets... untill the other day, at around 3280 lines or so it just stopped alltogether.. once I trimmed off a 100 lines or so it starts up again, but its still very sluggish..

-DjZ-

[nml375]

zath, in those situations, just sit back, put your mind on other things, and just try to calm down. We all get very frustrated every now and then with things "just not working for no god reason" - usually that means we're overlooking something to trivial to make notice of...

Yuour putlogs, I explicitly said in my last post, that if you use curly brackets instead of quotes, nor variable or command subsitutions will occur. You'd get the very same result if you use them (the beackets) on putserv..
Ever tried something like this?

Code: Select all

set color "blue"
putlog "The value of color is $color"

I do notice that you've got a loop in your code that iterates 100 times, each time launching an external application that fetches some remote webpage. Due to the single-threaded nature of eggdrop (regardless of whether tcl is threaded or not), your eggdrop will "freeze" until all nested calls (if any) has completed. Especially, launching an external application takes a lot of resources compared to the other code, and doing it this way could very wellcause your eggie to become very sluggish... Do you get any "Timer drift" log-entries?

[raider2k]

mh, external calls .. always a nice thing ^^
the only thing popping into my mind right now is:

try to run your script on any other box, see if its the script or the configuration .. cant get any easier than that in my opinion, at least its an easy-to-do option.