Log Monitor

[kenh83]

I would love to have a script that I could use to parse log files "live".


I'm not even sure where to really start with how to read a log file basically every second and have it return the new lines of text that are put into the log file.

As far as parsing the text, I can handle that much.. I just dont know how to get the first part.


Thanks so very much in advance!

[nml375]

With the 1.6.20 version, the LOG binding was introduced. This binding can be set up to trigger on any log-message matching the given "channel text" pattern:

Code: Select all

    (47) LOG (stackable)
         bind log <flags> <mask> <proc>
         proc-name <level> <channel> <message>

         Description: triggered whenever a message is sent to a log. The mask is
           matched against "channel text".
           The level argument to the proc will contain the level(s) the message
           is sent to, or '*' if the message is sent to all log levels at once.
           If the message wasn't sent to a specific channel, channel will be set
           to '*'.
         Module: core

A very simple example would be as follows:

Code: Select all

proc parseLog {level channel message} {
  puthelp "PRIVMSG #thechannel : Received log message \"$message\" from \"$channel\" ($level)"
}
bind log - "*" parseLog

If you'd rather observe a given file on the filesystem (not necessarily generated by your eggdrop), you'd have to resort to asynchronous file IO using the fileevent mechanism in tcl.

[kenh83]

nml,

Yeah.. i'm looking for example on how to do it for a system log file.

[nml375]

Roger that,
Then you'll need to use asynchronous IO:

Code: Select all

proc openFile {file} {
  set fd [open $file "RDONLY"]
  fconfigure $fd -blocking 0
  fileevent $fd readable [list readFile $fd]
}

proc readFile {file} {
  if {[eof $file]} {
    close $file
  } else {
    gets $file line
    //Do something with $line
    puthelp "PRIVMSG #somechannel :Read \"$line\" from log"
  }
}

openFile is used to open the file and set up the file-events. readFile will then be called by the event whenever there is a new line to be read in the (opened) file.

[kenh83]

How can I safely start this and allow it to keep running so it actively checks the files.. without it interferring or 'locking' up the bot?

[kenh83]

I guess what I'm getting at here, is I want to actively monitor a log file so that each time something new is wrote the tcl script catches it.. but doesn't care about previous entries..

and.. the log file must remain intact like a normal log file.



Hopefully I'm not confusing you or asking too much, as I really do appreciate your offerings here. You're very kind!

[nml375]

Well, as I said, this is asynchronous IO. Which means the code is event-driven and non-blocking by nature. You simply call the openFile proc once to open the file and set up the fileevent, and the event engine will call the readFile proc with appropriate arguments whenever there is something readable within the file.

That said, I do now notice that I wrote the code for socket communications, where eof would indicate that the socket had been closed. For a "tail-like" behavior, the readFile proc would be re-written like below:

Code: Select all

proc readFile {file} {
  if {[gets $file line] >= 0} {
    #gets returned a complete line from the file, do some further processing
    puthelp "PRIVMSG #somechannel :Read \"$line\" from logfile"
  }
}

Also, I should'nt use // for comments in tcl...

[nml375]

Or actually, scrap that..
The file will always be considered readable once it reaches the end of file, resulting in the event being triggered over and over (blocking eggdrop).

There are a few recipes for doing this using timers though; the following code tries to read one line every second (with some modifications, it could be made to try and read as many lines as possible once every second ).

Code: Select all

proc readFile {fileId} {
  #(Try to) read one line
  set bytes [gets $fileId line]
  if {$bytes >= 0} {
    #We've got some data, append it to the buffer.
    append ::tailBuffer($fileId) $line
    if {![eof $fileId]} {
      #Verify that we got a line with EOL; do something intelligent with the line of text
      puts stdout "Read line: $::tailBuffer($fileId)"

      #Clear the buffer..
      set ::tailBuffer($fileId) ""
    }
  }
  #Start another 1000 ms timer to call readFile again
  after 1000 [list readFile $fileId]
}

proc openFile {file} {
  #open file for read-only access
  set fileId [open $file RDONLY]

  #Move to the end of the file after opening, optional:
  seek $fileId 0 end

  #Configure the file for non-blocking access
  fconfigure $fileId -blocking 0

  #Prepare a buffer for lines missing EOL
  set ::tailBuffer($fileId) ""

  #Start a 1000 ms timer to call readFile
  after 1000 [list readFile $fileId]
}

[kenh83]

nml, whats the proper syntax to get this started.. and it stays running right?

[nml375]

You call "openFile" to open and start monitoring a file. Yes, it will continue to check the file roughly once every second for new content - assuming there's no error condition within "readFile" (in it's current state, there isn't much that could cause an error to occur).

Obviously, you'll have to modify "readFile" to implement whatever parsing you've set your mind to (I simply had the script write the read line to stdout for demonstration purposes).

[kenh83]

nml,

I changed the following line:

Code: Select all

puts stdout "Read line: $::tailBuffer($fileId)"

to:

Code: Select all

putlog "Read line: $::tailBuffer($fileId)"

from partyline on the bot i execute the following command:

Code: Select all

.tcl [openFile /path/to/filename.log]

The actual log file i'm using is 100% real the user account the bot is running under has full read access to the log file and works fine from ssh console when i do something like: tail -f /path/to/filename.log

When I execute the above command on the partyline of the bot the output is the following:

Code: Select all

Tcl error: invalid command name "after#0"

nml .. I know I've said it before, but thank you very much for all the time you've taken to address this script.

EDIT: It does start reading the log file.. but how can I get rid of that initial error.

Lastly whats the best way to start this openFile proc when the bot is first started?
- KenH83

[nml375]

When using the .tcl-command, don't use those brackets. "openFile" does not return anything that's supposed to be executed as a command.

Just use

Code: Select all

.tcl openFile thefile

[kenh83]

Ok Nml. Thank you.

Whats the best way to make sure that this "log monitoring" starts as soon as the bot is started?


KenH83.

[kenh83]

Never mind. I just load the script and have it do:

Code: Select all

proc evnt:init_server {type} {
        logMonitorOpenFile $::logMonitorFilename
}

Thanks again for all your help NML!