Zodbpack UnicodeDecodeError on migrated zodb

I have several dbs migrated from python 2 to python 3, Plone 5.2. One of them is giving me an error when packing the database

Traceback (most recent call last):
  File "/plone/plone5.2/buildout/bin/zeopack", line 42, in <module>
    sys.exit(plone.recipe.zeoserver.pack.main(host, port, unix, days, username, password, realm, blb_dir, storage))
  File "/plone/eggs/plone.recipe.zeoserver-2.0.1-py3.6.egg/plon/recipe/zeoserver/pack.py", line 65, in main
    _main(*args, **kw)
  File "/plone/eggs/plone.recipe.zeoserver-2.0.1-py3.6.egg/plon/recipe/zeoserver/pack.py", line 46, in _main
    cs.pack(wait=True, days=int(days))
  File "/plone/eggs/ZEO-5.2.1-py3.6.egg/ZEO/ClientStorage.py", ine 562, in pack
    return self._call('pack', t, wait)
  File "/plone/eggs/ZEO-5.2.1-py3.6.egg/ZEO/asyncio/client.py",line 764, in call
    return self.__call(self.call_threadsafe, method, args, **kw)
  File "/plone/eggs/ZEO-5.2.1-py3.6.egg/ZEO/asyncio/client.py",line 743, in call
    return self.wait_for_result(result, timeout)
  File "/plone/eggs/ZEO-5.2.1-py3.6.egg/ZEO/asyncio/client.py",line 756, in wait_for_result
    return future.result(timeout)
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
ZEO.Exceptions.ServerException: ('builtins.UnicodeDecodeError', ('ascii', b'\x00\x00\x00\x00\x00\x0\xeb$', 6, 7, 'ordinal not in range(128)'))

Everything should have been converted with zodbupdate with command ./bin/zodbupdate --convert-py3 --file=$path_to_db --encoding-fallback latin1. I'm not sure if the update had an error that went unnoticed when I migrated a couple months ago or if this is new. The site appears to functional normally and I can create backups, I just can't pack.

I tried copying this to a dev environment and rerunning the conversion in python 3 as described by documention for zodbupdate. It fails at the packing stage.

Packing storage ...
Traceback (most recent call last):
  File "./bin/zodbupdate", line 322, in <module>
    sys.exit(zodbupdate.main.main())
  File "/plone/eggs/zodbupdate-1.4-py3.6.egg/zodbupdate/main.py", line 239, in main
    storage.pack(time.time(), ZODB.serialize.referencesf)
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/FileStorage.py", line 1231, in pack
    pack_result = self.packer(self, referencesf, stop, gc)
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/FileStorage.py", line 1184, in packer
    opos = p.pack()
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/fspack.py", line 405, in pack
    self.gc.findReachable()
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/fspack.py", line 191, in findReachable
    self.findReachableAtPacktime([z64])
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/fspack.py", line 276, in findReachableAtPacktime
    for oid in self.findrefs(pos):
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/FileStorage/fspack.py", line 329, in findrefs
    return self.referencesf(self._file.read(dh.plen))
  File "/plone/eggs/ZODB-5.5.1-py3.6.egg/ZODB/serialize.py", line 650, in referencesf
    u.noload()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 6: ordinal not in range(128)

What's my recourse here? Is there a way to find these objects and remove them?

I think this is the same issue that @pbauer mentions here: https://github.com/zopefoundation/ZODB/issues/285

I got the oid from running zodbupdate -v and simply tried to load it in debug mode:

>>> oid=0x02eb03
>>> from ZODB.utils import p64
>>> app._p_jar.get(p64(oid))
ERROR:ZODB.Connection:Couldn't load state for BTrees.OIBTree.OISet 0x02eb03
Traceback (most recent call last):
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/Connection.py", line 795, in setstate
    self._reader.setGhostState(obj, p)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/serialize.py", line 633, in setGhostState
    state = self.getState(pickle)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/serialize.py", line 626, in getState
    return unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 6: ordinal not in range(128)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/Connection.py", line 795, in setstate
    self._reader.setGhostState(obj, p)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/serialize.py", line 633, in setGhostState
    state = self.getState(pickle)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py3.6.egg/ZODB/serialize.py", line 626, in getState
    return unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 6: ordinal not in range(128)

I have no idea what this is other than an instance of OISet. Unfortunately I can't use pbauer's solution of clearing the obj in python 2 first since I've already committed to the python 3 version before realizing it was a problem. Attempting to call clear() triggers the same problem. Is there some other way to remove an object by oid?

This looks like an "oid" (= Object ID). Nothing should try to convert it into an str (at least not with the ascii encoding). I assume that the latin1` fallback may have caused a misinterpretation.

In your followup post, you show that a load fails with a similar error in an unpickle. You could use Python's pickletoolsto look into the corresponding pickle. This might show what zodbupdate did wrong.

It is definitely an oid: running zodbupdate with the "-v" flag originally showed that it was 0x02eb03

>>> p64(0x02eb03)
b'\x00\x00\x00\x00\x00\x02\xeb\x03'

Slightly different ending but possibly just a display difference. I tried removing it with a method similar to one outlined here https://www.nathanvangheem.com/posts/2011/05/25/fixing-broken-zodb-object-references.html but this led to a ValueError and I am loathe to seriously attempt fixing the DB with this black magic.

Maybe the best course of action at this point is to cut my losses and keep "pack-gc = false" set for the future for this DB.

Plone Foundation Code of Conduct