I can't find the reference but I remember seeing something saying that the problem with zodb-cache-size-bytes had been fixed. Some googling hasn't turned up anything. We don't use it in production at the moment but we were considering it as a way to ensure our memory usage was more controlled.
Best I could find is
-
in 2012 (more discussion on if it works and details) https://sourceforge.net/p/plone/mailman/message/29172258/
-
someone using it in production last year - Slow restart of zeo/four instance big site. Repeated restarts are fast. Restart after 24hours plus is slow. Reason? but they use both limits so perhaps they never reach the bytes limit like you did
-
a bug fix in 2010 - https://bugs.launchpad.net/zodb/+bug/533015 but I'm not sure that explains what you saw.
Did you raise a bug about it? Its possible the code that works out what to evict is/was slow when the cache is large?