Increase dequeuing speed of Server module

KevKev · Post by **KevKev** » Fri Sep 03, 2004 8:17 pm

I've got a bot that is dumping several lines of text to users at a rather speedy clip. Due to the nature of the reuqests coming in (they tend to be in bursts), people are complaining of significant lag in response times.

I'd like to increase the speed at which the queues are flushed to the server. Flooding the server isn't a major issue. I'm aware that this is controled by the server module and isn;t somethign that is terribly straightforward to scale up but i'd like to make whatever changes i can to try to increase the dequeueing speed or increase the allowed size of bursts.

I've seen some other threads regarding similar issues but most resulted in completely skipping the eggdrop queues. With the rest of my project i'd much rather avoid re-inventing the wheel and simply step up the dequeue rate of eggdrop's server module.

Any suggestions?

KrzychuG · Post by **KrzychuG** » Sat Sep 04, 2004 5:56 am

Code: Select all

proc putnow { a } { 
  append a "\n" 
  putdccraw 0 [string length $a] $a 
}

You can find a discution about that procedure on: http://forum.egghelp.org/search.php?mode=results ;)

KevKev · Post by **KevKev** » Sat Sep 04, 2004 11:06 am

Sure i can use putdccraw to dump it straight out at whatever rate the users make requests. But that's a bit extreme.

I'd like ot use SOME queueing. At this poitn it's feasable for one of my users to request several hundred lines of response from the bot. Add to that there are many people using the bot at the same time and you're in for a very high number of lines per second.

While I said excess flood wasn't a major issue. It IS an issue at some level so i'd like to use the standard queuing mechanism whilst still increasing it's speed or it's burst rate.

KrzychuG · Post by **KrzychuG** » Sat Sep 04, 2004 12:08 pm

You can modify msgrate value in src/mod/server.mod/server.c (line 123 in egg1.6.16):

Code: Select all

/* Number of seconds to wait between transmitting queued lines to the server
 * lower this value at your own risk.  ircd is known to start flood control
 * at 512 bytes/2 seconds.
 */
#define msgrate 2

KevKev · Post by **KevKev** » Sat Sep 04, 2004 12:40 pm

results in it dequeuing a little faster but messages are still in the queue for a significant period of time.

is that value required to be an int or can it be a decimal? (i set it to 1 so it shoudl be at double the rate if i'm reading it right)

KevKev · Post by **KevKev** » Sat Sep 04, 2004 2:29 pm

OK here's what i'm thinking. I'd like to make the server queue burst along with the mode queue (with the same burst counter of course)

Near as i can tell from the code ( I DO NOT KNOW C so PLEASE correct me if i'm being stupid) this is the block that pops out the messages from the server queue.

Code: Select all

  if (mq.head) {
    burst++;

    if (deq_kick(DP_SERVER))
      return;

    if (fast_deq(DP_SERVER))
      return;

    write_to_server(mq.head->msg, mq.head->len);
    mq.tot--;
    last_time += calc_penalty(mq.head->msg);
    if (raw_log)
      putlog(LOG_SRVOUT, "*", "[s->] %s", mq.head->msg);
    q = mq.head->next;
    nfree(mq.head->msg);
    nfree(mq.head);
    mq.head = q;
    if (!mq.head)
      mq.last = NULL;
    return;
  }

i THINK this should work to make it burst with the mode queue

Code: Select all

if (mq.head) {
	  while (modeq.head && (burst < 11) && ((last_time - now) < MAXPENALTY)) {
		  burst++;
		  if (deq_kick(DP_SERVER))
			  return;
		  if (fast_deq(DP_SERVER))
			  return;

		  write_to_server(mq.head->msg, mq.head->len);
		  mq.tot--;
		  last_time += calc_penalty(mq.head->msg);
		  if (raw_log)
			  putlog(LOG_SRVOUT, "*", "[s->] %s", mq.head->msg);
		  q = mq.head->next;
		  nfree(mq.head->msg);
		  nfree(mq.head);
		  mq.head = q;
		  if (!mq.head)
			  mq.last = NULL;
		  return;
	  }
  }

have i done anything incredibly stupid in this change / does it make sense?

EDIT: Ok.. update.. the above doesn't work. it simply sent nothing to the server

any pointers to where i might have gone wong?

awyeah · Post by **awyeah** » Mon Sep 06, 2004 1:17 am

Heck, I don't know C. But you were saying you wanted to use some queues and not directly pass it to the sever.

Well you can do a thing:
[1] putquick is the fastest in all the queues.

Use putquick with -next that is very fast, almost the same as putdccraw I have seen. It wouldnt flood off the bot from the client server too. Else use putquick without the -next option, if you don't want an extremely fast output and want it to be queued.

I remember BarkerJr released a code in TCL espescially for handling queues. It is a tcl script code to be loaded within your bot's config file. So you can define, make and use your own queue.

I have subscribed to the eggheads mailing archive list so I get all their threads. If you goto their website and check on the past or the recent archives of this month or the previous one you can find it there. I may have the file but I am not sure.

awyeah · Post by **awyeah** » Mon Sep 06, 2004 3:34 am

Hehe, your in good luck. Here I found it... go and use it.

Code: Select all

# Copyright (C) 2003 BarkerJr
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

namespace eval put {
  # Name the queues here.  The further left, the higher priority queues.
  set queues [list akill kill echo ctcp2 ctcp msg warn]

  # Script begins here (nothing to do below here).
  foreach q $queues {
    if {![info exists queue($q)]} { set queue($q) {} }
  }
  unset q

  proc serv {q text} {
    variable queue
    variable errorInfo
    if {![info exists queue($q)]} {
      set errorInfo "No such queue named: $q"
      return 1
    }
    if {[lsearch -exact $queue($q) $text] != -1} {
      set errorInfo "Already exists in queue ($q): $text"
      return 2
    }
    lappend queue($q) $text
    return 0
  }

  foreach timer [utimers] {
    if {[lindex $timer 1] == {put::bisecondly}} { killutimer [lindex $timer 2] }
  }
  if {[info exists timer]} { unset timer }
  utimer 2 put::bisecondly
  proc bisecondly {} {
    variable queues
    variable queue
    foreach q $queues {
      if {[llength $queue($q)]} {
        putserv [lindex $queue($q) 0]
        set queue($q) [lreplace $queue($q) 0 0]
        utimer 2 put::bisecondly
        return
      }
    }
    utimer 2 put::bisecondly
  }

  bind dcc - qstat put::queuestat
  proc queuestat {hand idx text} {
    variable queues
    variable queue
    putdcc $idx {Queue      Lines}
    putdcc $idx {=================}
    foreach q $queues {
      putdcc $idx "[format %-10s $q] [format %5i [llength $queue($q)]]"
    }
    putdcc $idx {=================}
  }

  proc uninstall {} {
    unbind dcc - qstat put::queuestat
    foreach timer [utimers] {
      if {[lindex $timer 1] == {put::bisecondly}} { killutimer [lindex $timer 2] }
    }
    namespace delete [namespace current]
  }
}

KevKev · Post by **KevKev** » Wed Sep 08, 2004 12:51 pm

Using putquick seems to be the solution. This seems totally counterintuitive to me. To me it would seem that all queues should dequeue at the same rate but in a specific order of priority.

nacho · Post by **nacho** » Mon Jun 19, 2006 6:50 pm

how exactly do you use that put script posted above? Loading it into the eggdrop.conf didn't seem to have any effect.

Thanks!

NACHO

*EDIT* Also, I've tried using the `putquick $msg -next` but I was unable to get any kind of noticeable improvement. I've tried this both on my own private ircd-hybrid as well as efnet/linknet.
Thanks again.

De Kus · Post by **De Kus** » Tue Jun 20, 2006 3:53 am

I use '#define msgrate 1' though I am bound to excess flood. Since if 512bytes/2sec is true, you won't have problems unless you make lines longer than 256bytes (and puthelp will still be slower than 1line/sec)

. I believe I read that '#define msgrate 0' also bursts, but isnt really crazy (does it send only 1 line per main loop cyle?!) and should keep them in order (first putquick, putkick and mode, then server, then help).

nacho · Post by **nacho** » Tue Jun 20, 2006 3:27 pm

well im looking for more of a bursty way of doing things here. I want the bot to be able to spam 15 lines quickly, but then maybe nothing really happens for 5 or 30 minutes. Also, it IS important that the bot doesn't get disconnected for flooding..

De Kus · Post by **De Kus** » Tue Jun 20, 2006 6:34 pm

nacho wrote:Also, it IS important that the bot doesn't get disconnected for flooding..

In that case, setting msgrate to 1 is the fastest I can advise for you... which will just dump 1 line every 1sec... which in normal cases will give the impression of "fluently" output (I doubt a normal person reads faster than 1 line per second

)