Zodbverify: Porting Plone with ZopeDB to Python3

My first Attempt to upgrade a small PloneSite to 5.2 on Py3. All is running fine under 5.2-Py2.

I run zodbverify

>$ ./py2/bin/zodbverify -f var/lexikon/filestorage/Data.fs

I get many Log-Output-Entries like this:

INFO:zodbverify:
Could not process <class 'plone.registry.field.Dict'> record '\x00\x00\x00\x00\x00\x11\x13A':
INFO:zodbverify:'\x80\x02cplone.registry.field\nDict\nq\x01.\x80\x02}q\x02(U\nmin_lengthq\x03K\x00U\x08requiredq\x04\x88U\x0bdescriptionq\x05czope.i18nmessageid.message\nMessage\nq\x06(X1\x00\x00\x00Maps skin names to lists of resource bundle namesq\x07U\x05ploneq\x08NNtRq\tU\x05titleq\nh\x06(X\x1b\x00\x00\x00Resource bundles for themesq\x0bh\x08NNtRq\x0cU\x07defaultq\rNU\x08__name__q\x0eU\x05valueq\x0fU\x17_Element__tagged_valuesq\x10}q\x11U\x08key_typeq\x12U\x08\x00\x00\x00\x00\x00\x11\x13Bq\x13cplone.registry.field\nASCIILine\nq\x14\x86QU\x08readonlyq\x15\x89U\nmax_lengthq\x16NU\tfieldNameq\x17U\x18resourceBundlesForThemesq\x18U\x0edefaultFactoryq\x19NU\tinterfaceq\x1acProducts.ResourceRegistries.interfaces.settings\nIResourceRegistriesSettings\nq\x1bU\nvalue_typeq\x1cU\x08\x00\x00\x00\x00\x00\x11\x13Cq\x1dcplone.registry.field\nList\nq\x1e\x86QU\x07__doc__q\x1fXN\x00\x00\x00Resource bundles for themes\n\nMaps skin names to lists of resource bundle namesq U\rinterfaceNameq!UKProducts.ResourceRegistries.interfaces.settings.IResourceRegistriesSettingsq"u.'
INFO:zodbverify:Traceback (most recent call last):
  File "/Development/Plone/coredev52multipy/py2/eggs/zodbverify-1.0.1-py2.7.egg/zodbverify/verify.py", line 60, in verify_record
    unpickler.load()
ImportError: No module named ResourceRegistries.interfaces.settings

and a Summary at the End:

INFO:zodbverify:Done! Scanned 260308 records. 
Found 190 records that could not be loaded. 
Exceptions and how often they happened: 
ImportError: No module named ResourceRegistries.tools.CSSRegistry: 1
ImportError: No module named Product: 2
AttributeError: 'module' object has no attribute 'IPersistentExtra': 94
ImportError: No module named ResourceRegistries.interfaces.settings: 2
AttributeError: 'module' object has no attribute 'IUndoSupport': 1
ImportError: No module named ResourceRegistries.tools.JSRegistry: 1
ImportError: No module named imagecropping.dx: 89

My Question: What is the general procedure to solve such errors ?

Check this info:

I added

Products.ResourceRegistries
plone.app.imagecropping

to my Buildout and run it. Now the zodbverify script show this output:

Found 185 records that could not be loaded. 
Exceptions and how often they happened: 
AttributeError: 'module' object has no attribute 'IPersistentExtra': 182
ImportError: No module named Product: 2
AttributeError: 'module' object has no attribute 'IUndoSupport': 1

But IPersistentExtra and IUndoSupport lives in https://github.com/zopefoundation/Zope/blob/2.13.29/src/App/interfaces.py
but not in the master https://github.com/zopefoundation/Zope/blob/master/src/App/interfaces.py

I'm confused, My only Idea is to checkout the Zope-Master an add this two Interface Definitions.

1 Like

I'm also seeing these IPersistentExtra and IUndoSupport errors. I didn't find anything when googling them and the Zope repo doesn't yield much info either. Any ideas?

I added this to App/interfaces.py locally

# XXX: might contain non-API methods and outdated comments;
#      not synced with ZopeBook API Reference;
#      based on App.PersistentExtra.PersistentUtil
class IPersistentExtra(Interface):
    pass

# XXX: might contain non-API methods and outdated comments;
#      not synced with ZopeBook API Reference;
#      based on App.Undo.UndoSupport
class IUndoSupport(Interface):
    pass

Assuming you make these objects loadable, anyone know how to actually remove them from the ZODB? This doesn't seem to be something you can just delete in the ZMI?

I temporarily added Products.ResourceRegistries to my buildout so I could load the oid

>>> app._p_jar.get('\x00\x00\x00\x00\x00\x02K\x01')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/Connection.py", line 247, in get
    p, _ = self._storage.load(oid)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/mvccadapter.py", line 143, in load
    r = self._storage.loadBefore(oid, self._start)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/FileStorage/FileStorage.py", line 564, in loadBefore
    pos = self._lookup_pos(oid)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/FileStorage/FileStorage.py", line 523, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x024b01

Not too surprising, I think we assumed this was junk. I've seen a few guides such as https://github.com/zopefoundation/zodbdocs/blob/master/documentation/articles/multi-zodb-gc.rst which show how to replace that object with an instance of the Persistent class using that same oid, but I'm wary of using any guide with the line "Now we can be real evil and create a new fake object in place of the missing one". https://sixfeetup.com/blog/poskeyerror-debugging-and-fixing-cookbook doesn't apply because we can't delete it using "the standard Zope way" because it doesn't appear to be something we can traverse to.

Did you ever find a solution to this one or just live with it? It is looking for App.Product.Product which no longer exists in Zope 4. I believe this is related to the old /Control_Panel/Products in the ZMI, which also no longer exists in Zope 4. My Plone 5.1 environment has nothing in this area though. This is on a mounted database if that matters.

No, i found no solution and live with it. but it was a test, its not in production.

The oids that led to the App.ProductFolder.Product import fail point to app.Control_Panel and app.Control_Panel.Products in my 5.1 environment. Zope 4 changes how the Control Panel is setup significantly. Looks like instead of being a persistent object sitting on the root of the Zope app, it is instead a property of the app itself.

5.1:

>>> app.Control_Panel
<ApplicationManager at /Control_Panel>

5.2:

>>> app.Control_Panel
<App.ApplicationManager.ApplicationManager object at 0x7ff6b7017398>

So what happens to the old persistent Control_Panel? It looks like it is removed at startup https://github.com/zopefoundation/Zope/blob/master/src/OFS/Application.py#L206

@1letter I am going to guess that, like me, your test that found this error was not the main database, but a db mounted on main? What I was doing was taking a new main database and mounting on it backup copies from my 5.1 environment. I am surprised that my mounted database even has a record for App.Control_Panel but that appears to be the case. And it would make sense here because I am trying to mount a db onto a db that already has no persistent Control_Panel, so on the mounted db it just sits orphaned. This may be a non-problem on the actual production migration - in that case I would be starting with a main db that has mount points and I think the cleanup step on initialization would delete the entry in all mounted dbs.

Yes, i mount all my plone-sites via seperate mountpoints to a main database. i will test your hints soon. thanks for the hint.

I'm just getting the same issue you are and trying to work through it.

This may be a non-problem on the actual production migration - in that case I would be starting with a main db that has mount points and I think the cleanup step on initialization would delete the entry in all mounted dbs.

Unfortunately this did not seem to work.

  • You need to pack the Database to 0 days before migrating to Python 3..
  • I had to add a alias for IPersistentExtra to be able to migrate one DB. I've not investigated why:
    from plone.app.upgrade.utils import alias_module
    try:
        from App.interfaces import IPersistentExtra
        IPersistentExtra  # noqa
    except ImportError:
        alias_module('App.interfaces.IPersistentExtra', IDummy)
    
  • alias_module is great to fix such issues. I choose to live with these.
  • I tend to not put too much stock into the output of zodbverify as long as the site and all it's content works. Make sure you can pack the DB after migrating and are able to rebuild the catalog. Then you should be mostly fine.

Plone Foundation Code of Conduct