"ZODB: The <class 'BTrees.IOBTree.IOBucket'> object you're saving is large" on cut/paste operation

Env:

  • Plone 5.2.1
  • plone.restapi 6.3.0

I tried to cut/paste the contents of a folder, roughly 50 PDF files (default file content type) to another folder. I tried this in Volto and vanilla Plone and it did not work and I did not get any meaningful error message. So I went to the ZMI and cut/paste the folder contents, then I got the following error trace:

3|XXX-internet-live-api-instance3  | /home/kitconcept/.buildout/eggs/ZODB-5.5.1-py3.7.egg/ZODB/Connection.py:580: UserWarning: The <class 'BTrees.IOBTree.IOBucket'>
3|XXX-internet-live-api-instance3  | object you're saving is large. (18769557 bytes.)
3|XXX-internet-live-api-instance3  | Perhaps you're storing media which should be stored in blobs.
3|XXX-internet-live-api-instance3  | Perhaps you're using a non-scalable data structure, such as a
3|XXX-internet-live-api-instance3  | PersistentMapping or PersistentList.
3|XXX-internet-live-api-instance3  | Perhaps you're storing data in objects that aren't persistent at
3|XXX-internet-live-api-instance3  | all. In cases like that, the data is stored in the record of the
3|XXX-internet-live-api-instance3  | containing persistent object.
3|XXX-internet-live-api-instance3  | In any case, storing records this big is probably a bad idea.
3|XXX-internet-live-api-instance3  | If you insist and want to get rid of this warning, use the
3|XXX-internet-live-api-instance3  | large_record_size option of the ZODB.DB constructor (or the
3|XXX-internet-live-api-instance3  | large-record-size option in a configuration file) to specify a larger
3|XXX-internet-live-api-instance3  | size.
3|XXX-internet-live-api-instance3  |   warnings.warn(large_object_message % (obj.__class__, len(p)))
3|XXX-internet-live-api-instance3  | 2020-03-05 22:37:06,131 ERROR   [Zope.SiteErrorLog:17][waitress] ConflictError: https://XXX.XXX.com/api/XXX/archiv/manage_pasteObjects
3|XXX-internet-live-api-instance3  | Traceback (innermost last):
3|XXX-internet-live-api-instance3  |   Module ZPublisher.WSGIPublisher, line 162, in transaction_pubevents
3|XXX-internet-live-api-instance3  |   Module transaction._manager, line 252, in commit
3|XXX-internet-live-api-instance3  |   Module transaction._manager, line 131, in commit
3|XXX-internet-live-api-instance3  |   Module transaction._transaction, line 311, in commit
3|XXX-internet-live-api-instance3  |   Module transaction._compat, line 50, in reraise
3|XXX-internet-live-api-instance3  |   Module transaction._transaction, line 302, in commit
3|XXX-internet-live-api-instance3  |   Module transaction._transaction, line 447, in _commitResources
3|XXX-internet-live-api-instance3  |   Module transaction._compat, line 50, in reraise
3|XXX-internet-live-api-instance3  |   Module transaction._transaction, line 424, in _commitResources
3|XXX-internet-live-api-instance3  |   Module ZODB.Connection, line 692, in tpc_vote
3|XXX-internet-live-api-instance3  |   Module ZEO.ClientStorage, line 752, in tpc_vote
3|XXX-internet-live-api-instance3  |   Module ZEO.asyncio.client, line 764, in call
3|XXX-internet-live-api-instance3  |   Module ZEO.asyncio.client, line 743, in call
3|XXX-internet-live-api-instance3  |   Module ZEO.asyncio.client, line 756, in wait_for_result
3|XXX-internet-live-api-instance3  |   Module concurrent.futures._base, line 435, in result
3|XXX-internet-live-api-instance3  |   Module concurrent.futures._base, line 384, in __get_result
3|XXX-internet-live-api-instance3  | ZODB.POSException.ConflictError: database conflict error (oid 0x0de664, class plone.app.contenttypes.content.File, serial this txn started with 0x03d626f056564b66 2020-02-28 20:00:20.235255, serial currently committed 0x03d6544f30358066 2020-03-05 21:35:11.298982)
3|XXX-internet-live-api-instance3  | 2020-03-05 22:37:09,417 ERROR   [Zope.SiteErrorLog:17][waitress] CopyError: https://XXX.XXX.com/api/de/XXX/archiv/manage_pasteObjects
3|XXX-internet-live-api-instance3  | Traceback (innermost last):
3|XXX-internet-live-api-instance3  |   Module ZPublisher.WSGIPublisher, line 156, in transaction_pubevents
3|XXX-internet-live-api-instance3  |   Module ZPublisher.WSGIPublisher, line 338, in publish_module
3|XXX-internet-live-api-instance3  |   Module ZPublisher.WSGIPublisher, line 256, in publish
3|XXX-internet-live-api-instance3  |   Module ZPublisher.mapply, line 85, in mapply
3|XXX-internet-live-api-instance3  |   Module ZPublisher.WSGIPublisher, line 62, in call_object
3|XXX-internet-live-api-instance3  |   Module OFS.CopySupport, line 320, in manage_pasteObjects
3|XXX-internet-live-api-instance3  |   Module OFS.CopySupport, line 204, in _pasteObjects
3|XXX-internet-live-api-instance3  | OFS.CopySupport.CopyError: Item Not Found

It seems the cut/paste worked though. All files are there, the old folder is empty. Though, I would like to understand what goes on here.

The PDF files are not very large, a few MB at most. It seems Andreas had a similar problem during a migration:

Because of another ZODB problem that I fixed before I ran into this issue, I am 100% sure we have a clean index (did clear-and-rebuild-catalog without any issues) and a clean Solr index (did full reindex without any issues).

Any idea what could cause this?

I'm guessing based on that, that you're indexing all text from the PDFs and store that in the catalog somehow. An 18MB IOBucket seems a bit much :slight_smile:

I'm unsure about the ZODB.POSException.ConflictError. Might waitress somehow retries the request by itself, as an conflict on a file being pasted makes not that much sense to me.

1 Like

About the ZODB.POSException.ConflictError and waitress:

1 Like

Thank you @jaroel and @jugmac00 for the pointers! Will look into it and report back here.

It might be helpful to log the oid of the object, so we can load the object and see where it comes from.
Open issue for that is at https://github.com/zopefoundation/ZODB/issues/197