Normally when u use something like dynamic channel settings
and set need-unban this happens.
putquick "PRIVMSG
chanserv@services.dal.net :UNBAN $chan $botnick"
The auto rejoin timer built within self to the bot and you would have to modify the source then recompile it for this. I tried but I couldn't find where these timers are located in the .c files. Also modifying them can cause problems so I suggest you leave them alone.
You can try like adding a command like putserv "JOIN $chan" after
the need ban command followed by a semi-colen (;), that might work
if not. Or maybe you can use tcl scripts.. but that wouldnt really affect this as much, would almost be the same.