Your bot is removing the invites and exempts probably because you have +dynamicexempts and +dynamicinvites .. which means they aren't really removed from the bot, but just from the channel's invite/exempt list.
As for the DCC problem .. check if everyone has a matching host and needed flags (depends on the settings).
bot has taken off exempts (e) and invites (I),
if you do
set invite-time 0
set exempt-time 0
it won't remove e/I modes
if bot refuses your DCC chat telnet to it, or /msg botnick ident pass
Once you are connected to the bot via a telnet application, you should do a ".match +p" and make sure the people that need access are set +p.
If not, add them.
Next, do a ".ignores" and find out if he is ignoring any1.
Lastly, do a ".channel #channame" replacing #channame with a channel your users are in, and check it is matching the users correctly (they should have there ahdnles next to there nicknames). If not, you need to look through each user in the userfile, as there is most likely a conflicting hostmask.