Zodbverify: Porting Plone with ZopeDB to Python3

1letter · July 17, 2019, 1:15pm

My first Attempt to upgrade a small PloneSite to 5.2 on Py3. All is running fine under 5.2-Py2.

I run zodbverify

>$ ./py2/bin/zodbverify -f var/lexikon/filestorage/Data.fs

I get many Log-Output-Entries like this:

INFO:zodbverify:
Could not process <class 'plone.registry.field.Dict'> record '\x00\x00\x00\x00\x00\x11\x13A':
INFO:zodbverify:'\x80\x02cplone.registry.field\nDict\nq\x01.\x80\x02}q\x02(U\nmin_lengthq\x03K\x00U\x08requiredq\x04\x88U\x0bdescriptionq\x05czope.i18nmessageid.message\nMessage\nq\x06(X1\x00\x00\x00Maps skin names to lists of resource bundle namesq\x07U\x05ploneq\x08NNtRq\tU\x05titleq\nh\x06(X\x1b\x00\x00\x00Resource bundles for themesq\x0bh\x08NNtRq\x0cU\x07defaultq\rNU\x08__name__q\x0eU\x05valueq\x0fU\x17_Element__tagged_valuesq\x10}q\x11U\x08key_typeq\x12U\x08\x00\x00\x00\x00\x00\x11\x13Bq\x13cplone.registry.field\nASCIILine\nq\x14\x86QU\x08readonlyq\x15\x89U\nmax_lengthq\x16NU\tfieldNameq\x17U\x18resourceBundlesForThemesq\x18U\x0edefaultFactoryq\x19NU\tinterfaceq\x1acProducts.ResourceRegistries.interfaces.settings\nIResourceRegistriesSettings\nq\x1bU\nvalue_typeq\x1cU\x08\x00\x00\x00\x00\x00\x11\x13Cq\x1dcplone.registry.field\nList\nq\x1e\x86QU\x07__doc__q\x1fXN\x00\x00\x00Resource bundles for themes\n\nMaps skin names to lists of resource bundle namesq U\rinterfaceNameq!UKProducts.ResourceRegistries.interfaces.settings.IResourceRegistriesSettingsq"u.'
INFO:zodbverify:Traceback (most recent call last):
  File "/Development/Plone/coredev52multipy/py2/eggs/zodbverify-1.0.1-py2.7.egg/zodbverify/verify.py", line 60, in verify_record
    unpickler.load()
ImportError: No module named ResourceRegistries.interfaces.settings

and a Summary at the End:

INFO:zodbverify:Done! Scanned 260308 records. 
Found 190 records that could not be loaded. 
Exceptions and how often they happened: 
ImportError: No module named ResourceRegistries.tools.CSSRegistry: 1
ImportError: No module named Product: 2
AttributeError: 'module' object has no attribute 'IPersistentExtra': 94
ImportError: No module named ResourceRegistries.interfaces.settings: 2
AttributeError: 'module' object has no attribute 'IUndoSupport': 1
ImportError: No module named ResourceRegistries.tools.JSRegistry: 1
ImportError: No module named imagecropping.dx: 89

My Question: What is the general procedure to solve such errors ?

rubencho · July 18, 2019, 10:11am

Check this info:

1letter · July 18, 2019, 1:52pm

I added

Products.ResourceRegistries
plone.app.imagecropping

to my Buildout and run it. Now the zodbverify script show this output:

Found 185 records that could not be loaded. 
Exceptions and how often they happened: 
AttributeError: 'module' object has no attribute 'IPersistentExtra': 182
ImportError: No module named Product: 2
AttributeError: 'module' object has no attribute 'IUndoSupport': 1

But IPersistentExtra and IUndoSupport lives in https://github.com/zopefoundation/Zope/blob/2.13.29/src/App/interfaces.py
but not in the master https://github.com/zopefoundation/Zope/blob/master/src/App/interfaces.py

I'm confused, My only Idea is to checkout the Zope-Master an add this two Interface Definitions.

reinhardt · July 26, 2019, 8:17am

I'm also seeing these IPersistentExtra and IUndoSupport errors. I didn't find anything when googling them and the Zope repo doesn't yield much info either. Any ideas?

1letter · July 26, 2019, 8:36am

I added this to App/interfaces.py locally

# XXX: might contain non-API methods and outdated comments;
#      not synced with ZopeBook API Reference;
#      based on App.PersistentExtra.PersistentUtil
class IPersistentExtra(Interface):
    pass

# XXX: might contain non-API methods and outdated comments;
#      not synced with ZopeBook API Reference;
#      based on App.Undo.UndoSupport
class IUndoSupport(Interface):
    pass

Esoth · October 1, 2019, 2:56pm

Assuming you make these objects loadable, anyone know how to actually remove them from the ZODB? This doesn't seem to be something you can just delete in the ZMI?

Esoth · October 1, 2019, 3:43pm

I temporarily added Products.ResourceRegistries to my buildout so I could load the oid

>>> app._p_jar.get('\x00\x00\x00\x00\x00\x02K\x01')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/Connection.py", line 247, in get
    p, _ = self._storage.load(oid)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/mvccadapter.py", line 143, in load
    r = self._storage.loadBefore(oid, self._start)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/FileStorage/FileStorage.py", line 564, in loadBefore
    pos = self._lookup_pos(oid)
  File "/sprj/btp_zope_plone5/plone-btp-dev-02/buildouts/eggs/ZODB-5.5.1-py2.7.egg/ZODB/FileStorage/FileStorage.py", line 523, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x024b01

Not too surprising, I think we assumed this was junk. I've seen a few guides such as https://github.com/zopefoundation/zodbdocs/blob/master/documentation/articles/multi-zodb-gc.rst which show how to replace that object with an instance of the Persistent class using that same oid, but I'm wary of using any guide with the line "Now we can be real evil and create a new fake object in place of the missing one". https://sixfeetup.com/blog/poskeyerror-debugging-and-fixing-cookbook doesn't apply because we can't delete it using "the standard Zope way" because it doesn't appear to be something we can traverse to.

Esoth · October 15, 2019, 5:20pm

Did you ever find a solution to this one or just live with it? It is looking for App.Product.Product which no longer exists in Zope 4. I believe this is related to the old /Control_Panel/Products in the ZMI, which also no longer exists in Zope 4. My Plone 5.1 environment has nothing in this area though. This is on a mounted database if that matters.

1letter · October 15, 2019, 6:49pm

No, i found no solution and live with it. but it was a test, its not in production.

Esoth · October 16, 2019, 3:47pm

The oids that led to the App.ProductFolder.Product import fail point to app.Control_Panel and app.Control_Panel.Products in my 5.1 environment. Zope 4 changes how the Control Panel is setup significantly. Looks like instead of being a persistent object sitting on the root of the Zope app, it is instead a property of the app itself.

5.1:

>>> app.Control_Panel
<ApplicationManager at /Control_Panel>

5.2:

>>> app.Control_Panel
<App.ApplicationManager.ApplicationManager object at 0x7ff6b7017398>

So what happens to the old persistent Control_Panel? It looks like it is removed at startup https://github.com/zopefoundation/Zope/blob/master/src/OFS/Application.py#L206

@1letter I am going to guess that, like me, your test that found this error was not the main database, but a db mounted on main? What I was doing was taking a new main database and mounting on it backup copies from my 5.1 environment. I am surprised that my mounted database even has a record for App.Control_Panel but that appears to be the case. And it would make sense here because I am trying to mount a db onto a db that already has no persistent Control_Panel, so on the mounted db it just sits orphaned. This may be a non-problem on the actual production migration - in that case I would be starting with a main db that has mount points and I think the cleanup step on initialization would delete the entry in all mounted dbs.

1letter · October 16, 2019, 6:24pm

Yes, i mount all my plone-sites via seperate mountpoints to a main database. i will test your hints soon. thanks for the hint.

Esoth · October 16, 2019, 7:22pm

I'm just getting the same issue you are and trying to work through it.

This may be a non-problem on the actual production migration - in that case I would be starting with a main db that has mount points and I think the cleanup step on initialization would delete the entry in all mounted dbs.

Unfortunately this did not seem to work.

pbauer · October 17, 2019, 7:28am

You need to pack the Database to 0 days before migrating to Python 3..

I had to add a alias for IPersistentExtra to be able to migrate one DB. I've not investigated why:

from plone.app.upgrade.utils import alias_module
try:
    from App.interfaces import IPersistentExtra
    IPersistentExtra  # noqa
except ImportError:
    alias_module('App.interfaces.IPersistentExtra', IDummy)

alias_module is great to fix such issues. I choose to live with these.
I tend to not put too much stock into the output of zodbverify as long as the site and all it's content works. Make sure you can pack the DB after migrating and are able to rebuild the catalog. Then you should be mostly fine.

mauritsvanrees · November 26, 2019, 11:47pm

I tried zodbverify on a small site and ran into similar errors as above.
In a site-specific package I now have a file with these patches, which make zodbverify and zodbupdate work:

from plone.alterego.interfaces import IDynamicObjectFactory
from plone.app.upgrade.utils import alias_module
from plone.dexterity.schema import SchemaModuleFactory
from zope.component.hooks import getSiteManager
from zope.interface import Interface


try:
    from App.interfaces import IPersistentExtra  # noqa
except ImportError:
    class IPersistentExtra(Interface):
        pass
    alias_module('App.interfaces.IPersistentExtra', IPersistentExtra)

try:
    from App.interfaces import IUndoSupport  # noqa
except ImportError:
    class IUndoSupport(Interface):
        pass
    alias_module('App.interfaces.IUndoSupport', IUndoSupport)

try:
    from Products.ResourceRegistries.interfaces.settings import IResourceRegistriesSettings  # noqa
except ImportError:
    class IResourceRegistriesSettings(Interface):
        pass
    alias_module('Products.ResourceRegistries.interfaces.settings.IResourceRegistriesSettings', IResourceRegistriesSettings)

try:
    from webdav.interfaces import IFTPAccess  # noqa
except ImportError:
    class IFTPAccess(Interface):
        pass
    alias_module('webdav.interfaces.IFTPAccess', IFTPAccess)

try:
    from webdav.EtagSupport import EtagBaseInterface  # noqa
except ImportError:
    class EtagBaseInterface(Interface):
        pass
    alias_module('webdav.EtagSupport.EtagBaseInterface', EtagBaseInterface)


# Register the dynamic schema utility.
sm = getSiteManager()
sm.registerUtility(factory=SchemaModuleFactory, name="plone.dexterity.schema.generated")

See Migration to Plone 5.2 - ZODB to py3 - zodbverify for a small bit of information about the utility at the end.

For the rest, I use alias_module. For that, I am wondering which is better: create a specific dummy interface (for example IPersistentExtra in the first patch), or simply use zope.interface.Interface.

With a specific interface:

Good: You can use the specific interface to search for items that have this interface, so you have a chance to clean it up.
Bad: you need to keep the new specific interface around forever. You could remove the alias though.

With simply Interface:

Good: you only need the alias during migration. You can remove the code afterwards.
Bad: any code that checks if an object implements for example IPersistentExtra, will find that all objects implement it, because all objects implement Interface. But after migration, you could remove the alias.

How are people handing this?

mauritsvanrees · November 29, 2019, 11:57am

For zodbupdate you can define rename rules. These seem to work for me:

rename_dict = {
    'App.interfaces IPersistentExtra': 'zope.interface Interface',
    'App.interfaces IUndoSupport': 'zope.interface Interface',
    'Products.ResourceRegistries.interfaces.settings IResourceRegistriesSettings': 'zope.interface Interface',
    'webdav.EtagSupport EtagBaseInterface': 'zope.interface Interface',
    'webdav.interfaces IFTPAccess': 'zope.interface Interface',
    # Add extra renames that seem only needed to avoid warnings:
    'webdav.interfaces IDAVCollection': 'zope.interface Interface',
    'webdav.interfaces IDAVResource': 'zope.interface Interface',
    'webdav.interfaces IWriteLock': 'zope.interface Interface',
}

Without this, zodbupdate failed with:

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package

Maybe that was with an alias_module for webdav already in place. My notes are inconclusive there.

But this is with a small Plone Site that started in Plone 5.0, then 5.1, now 5.2. No Archetypes. I expect that these renames are needed for all sites. So I wonder why these rewrites are not somewhere in core Plone already? Apparently people have successfully migrated Plone Sites without these. Haven't others run into the same?

avoinea · December 13, 2019, 10:56pm

Ok, we added most of these aliases to eea.aliases and released it in order for others to avoid the struggle

mauritsvanrees · March 3, 2020, 6:35pm

For the problem with IPersistentExtra I opened an issue in Zope. Conclusion from there: core Zope does not have this problem, so it is Plone specific. And it is not even necessarily a problem in core Plone: a Plone 5.1 site with the standard content does not have this issue when migrated to 5.2.

I did not like my aliases solution, so I did not want to use the eea.aliases package based on that. Instead, I went with only a rename dictionary for zodbupdate, and loading the dexterity utility.

So here is another tool that you can use: zest.zodbupdate. See PyPI and source code.

My thinking now is: some of the problems that zodbverify complains about, are problems that zodbupdate will solve. So the idea is now (from the readme of zest.zodbupdate):

First with Python 2.7, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff! This will detect and apply several explicit and implicit rename rules.
Then run bin/instance zodbverify . If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate.

When all is well, on Python 3 run:

bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8

For good measure, on Python 3 run bin/instance zodbverify .

Vlnni · August 30, 2023, 9:55am

Hello,
In which file i need to write this rename_dict. Where to create setup.py file and rename dictionary, decode dictionary. Do I need to write this in any config file or Do I need to create any .py files in which folder. For plone we have zinstance,buildout-cache,plone-docs. I don't know where to include these files or statements. I have seen the zodbupdate documentation but I didnot get how to do this zodb migration from py2 to py3. Thanking you and hoping for a reply.

mauritsvanrees · August 31, 2023, 11:34pm

This rename_dict would go in a separate Python package, for example created with plonecli.
You could use the zest.zodbupdate package as an example.
Then make sure your package I is installed in your Plone environment (with pip or buildout).