If you're talking about channel bans, no there's no relation between channel bans and Eggdrop. If there's any slowdown then the problem is with the ircd or the server your Eggdrop's connected through. If you're talking about Eggdrop's internal banlist, then probably if the number of internal bans is too high it would take more processing for an Eggdrop to actually find the ban in the list; but it's not too slow to be actually called a "slowdown."
That would depend on the performance of the host running the eggdrop, the number of channel-members, and so forth. Much simplified, whenever your eggdrop rechecks the channel for new members to kick, it will iterate through each ban, testing it against all channel members while also testing for any present exempts or privileged user. Also keep in mind that there are different tests for "ircbans" and bans in eggdrop's internal banlist.
But, a very rough example; a channel with 600 members and 1 bans would probably cause roughly the same load as a channel with 30 members and 20 bans.