Migrating a ZODB from Python 2.7 to Python 3: how to delete the objects that cause problem? [Resolved]

Plone Support
, , ,
1

I'm in the process of migrating my ZODB from Python 2.7 to Python 3 (I hope it worth it!).

Unfortunately, problems pop up when I verify the integrity of the database in Python 2 and running ./bin/instance zodbverify -D indeed "result[s] in PDB with the pickle-data and a decompiled pickle in place[...]":

$ ./bin/instance zodbverify -D
2019-09-13 11:28:54,602 WARNING [Init:89][MainThread] Class Products.CMFFormController.ControllerPythonScript.ControllerPythonScript has a security declaration for nonexistent method 'ZPythonScriptHTML_changePrefs'
2019-09-13 11:28:54,606 WARNING [Init:89][MainThread] Class Products.CMFFormController.ControllerValidator.ControllerValidator has a security declaration for nonexistent method 'ZPythonScriptHTML_changePrefs'
2019-09-13 11:28:56,883 WARNING [plone.behavior:172][MainThread] Specifying 'for' in behavior 'Tiles' if no 'factory' is given has no effect and is superfluous.
2019-09-13 11:28:57,873 INFO    [Zope:45][MainThread] Ready to handle requests
2019-09-13 11:28:57,874 INFO    [zodbverify:22][MainThread] Scanning ZODB...
2019-09-13 11:28:57,874 INFO    [zodbverify:64][MainThread] 
Could not process <class 'App.ApplicationManager.ApplicationManager'> record '\x00\x00\x00\x00\x00\x00\x00\x03':
2019-09-13 11:28:57,874 INFO    [zodbverify:65][MainThread] '\x80\x02cApp.ApplicationManager\nApplicationManager\nq\x01.\x80\x02}q\x02U\x08Productsq\x03U\x08\x00\x00\x00\x00\x00\x00\x00\x04q\x04cApp.Product\nProductFolder\nq\x05\x86Qs.'
2019-09-13 11:28:57,874 INFO    [zodbverify:66][MainThread] Traceback (most recent call last):
  File "/home/bitouze/site-gte-5-2/buildout-cache/eggs/zodbverify-1.0.1-py2.7.egg/zodbverify/verify.py", line 60, in verify_record
    unpickler.load()
ImportError: No module named Product

    0: \x80 PROTO      2
    2: }    EMPTY_DICT
    3: q    BINPUT     2
    5: U    SHORT_BINSTRING 'Products'
   15: q    BINPUT     3
   17: U    SHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\x00\x04'
   27: q    BINPUT     4
   29: c    GLOBAL     'App.Product ProductFolder'
   56: q    BINPUT     5
   58: \x86 TUPLE2
   59: Q    BINPERSID
   60: s    SETITEM
   61: .    STOP
highest protocol among opcodes = 2
> /home/bitouze/site-gte-5-2/buildout-cache/eggs/zodbverify-1.0.1-py2.7.egg/zodbverify/verify.py(79)verify_record()
-> msg = "{}: {}".format(e.__class__.__name__, str(e))
(Pdb)

but I have no idea how "[t]his enables solving the problem by either adding a stub class in the code or by deleting the object in the ZODB".

Many thanks in advance if you can help me (and actually do it! :slight_smile:)

2

Had similar errors while migrating a site from 5.1.
zodbverify will complain about anything that was once installed, doesn't matter if the addon is used anymore or not.
Just try the zodbupdate as written in the docs and see what really fails. I had no problems converting.

3

Do you mean zodbupdate? I first couldn't run it because of a typo in the code of "Prepare Your Buildout For Migrating The Database To Python 3" ( parts += instead of parts =+, as reported here).

Then a first try led to a lot of errors. As advised, I deleted Data.fs.index and, now, I succeeded:

Plone 5.2.0 (5207)
CMF 2.4.0
Zope 4.1.1
Python 3.7.3 (default, Apr 4 2019, 08:55:10) [GCC 8.3.1 20190329]
PIL 6.1.0 (Pillow)
WSGI : Allumé
Serveur : waitress 1.3.0

Thanks!

4

yes, zodbupdate :smile:

5

It would still be nice to know how to delete these objects, or figure out what is referencing them. I've had a similar problem where it will find objects from old packages that should have been migrated away from. Presumably if they were once installed but no longer relevant, there is still an object somewhere that is referencing them? I don't know where it is - some kind of history storage?

Here is one that I get

(Pdb) c
INFO:zodbverify:
Could not process <class 'Products.ATContentTypes.tool.metadata.MetadataTool'> record '\x00\x00\x00\x00\x00\x00\x003':
INFO:zodbverify:'cProducts.ATContentTypes.tool.metadata\nMetadataTool\nq\x01.}q\x02(U\x12__ac_local_roles__q\x03}q\x04U\nadmin_ericq\x05]q\x06U\x05Ownerq\x07asU\x04DCMIq\x08(U\x08\x00\x00\x00\x00\x00\x00\x02\x9ecProducts.CMFDefault.MetadataTool\nMetadataSchema\nq\ttq\nQU\x05titleq\x0bU0Controls metadata like keywords, copyrights, etcq\x0cu.'
INFO:zodbverify:Traceback (most recent call last):
  File "/sprj/st_zope_plone5/buildouts/eggs/zodbverify-1.0.2-py2.7.egg/zodbverify/verify.py", line 60, in verify_record
    unpickler.load()
ImportError: No module named CMFDefault.MetadataTool

    0: }    EMPTY_DICT
    1: q    BINPUT     2
    3: (    MARK
    4: U        SHORT_BINSTRING '__ac_local_roles__'
   24: q        BINPUT     3
   26: }        EMPTY_DICT
   27: q        BINPUT     4
   29: U        SHORT_BINSTRING 'admin_eric'
   41: q        BINPUT     5
   43: ]        EMPTY_LIST
   44: q        BINPUT     6
   46: U        SHORT_BINSTRING 'Owner'
   53: q        BINPUT     7
   55: a        APPEND
   56: s        SETITEM
   57: U        SHORT_BINSTRING 'DCMI'
   63: q        BINPUT     8
   65: (        MARK
   66: U            SHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\x02\x9e'
   76: c            GLOBAL     'Products.CMFDefault.MetadataTool MetadataSchema'
  125: q            BINPUT     9
  127: t            TUPLE      (MARK at 65)
  128: q        BINPUT     10
  130: Q        BINPERSID
  131: U        SHORT_BINSTRING 'title'
  138: q        BINPUT     11
  140: U        SHORT_BINSTRING 'Controls metadata like keywords, copyrights, etc'
  190: q        BINPUT     12
  192: u        SETITEMS   (MARK at 3)
  193: .    STOP
highest protocol among opcodes = 1
> /sprj/st_zope_plone5/buildouts/eggs/zodbverify-1.0.2-py2.7.egg/zodbverify/verify.py(79)verify_record()
-> msg = "{}: {}".format(e.__class__.__name__, str(e))

My sites no longer use Archetypes but this looks like it refers to the portal_metadata tool which still exists on these older sites. CMFDefault is no longer a part of my buildout but if I look at the code for Products.ATContentTypes.tool.metadata.MetadataTool it doesn't actually reference CMFDefault anymore. So I'm not sure what's going on. I tried deleting portal_metadata anyway, packing the db, and rerunning zodbverify but it's still catching this. I don't know how to "find" this object and remove it.