Upgrade from 5.1.5 to 5.2 (python2.7): OK but the latter to 5.2 (python3): not OK [Resolved]

dbitouze · September 12, 2019, 8:13pm

I succeeded to upgrade a (rather small, with no extra add-ons) site from Plone 5.1.5 to Plone 5.2 (python2.7), just by copying the blobstorage folder and the Data.fs file: I got only numerous DeprecationWarnings.

But when I tried to do the same from the new Plone 5.2 (python2.7) upgraded site to a Plone 5.2 (python3) one, I get the error below.

Is it hopeless to migrate a Plone 5.2 (python2.7) site to a Plone 5.2 (python3) one?

instance: /home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Products.CMFUid-3.0.1-py3.7.egg/Products/CMFUid/UniqueIdHandlerTool.py:24: DeprecationWarning: InitializeClass is deprecated. Please import from AccessControl.class_init.
  from App.class_init import InitializeClass
/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Products.CMFEditions-3.3.2-py3.7.egg/Products/CMFEditions/StandardModifiers.py:46: DeprecationWarning: ComponentLookupError is deprecated. Import from zope.interface.interfaces
  from zope.component.interfaces import ComponentLookupError
/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Products.CMFFormController-4.1.0-py3.7.egg/Products/CMFFormController/ControllerPythonScript.py:41: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2019-09-12 22:05:58,388 WARNING [Init:89][MainThread] Class Products.CMFFormController.ControllerPythonScript.ControllerPythonScript has a security declaration for nonexistent method 'ZPythonScriptHTML_changePrefs'
2019-09-12 22:05:58,392 WARNING [Init:89][MainThread] Class Products.CMFFormController.ControllerValidator.ControllerValidator has a security declaration for nonexistent method 'ZPythonScriptHTML_changePrefs'
/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Products.CMFPlacefulWorkflow-2.0.0-py3.7.egg/Products/CMFPlacefulWorkflow/permissions.py:5: DeprecationWarning: setDefaultRoles is deprecated. Please use addPermission from AccessControl.Permission.
  from Products.CMFCore.permissions import setDefaultRoles
Traceback (most recent call last):
  File "/home/bitouze/test-plone-5-2-python3/zinstance/parts/instance/bin/interpreter", line 272, in <module>
    exec(compile(__file__f.read(), __file__, "exec"))
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/serve.py", line 252, in <module>
    sys.exit(main() or 0)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/serve.py", line 248, in main
    return command.run()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/serve.py", line 189, in run
    global_conf=vars)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/serve.py", line 217, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 253, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 278, in loadobj
    return context.create()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 715, in create
    return self.object_type.invoke(self)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 209, in invoke
    app = context.app_context.create()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 715, in create
    return self.object_type.invoke(self)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/loadwsgi.py", line 152, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/PasteDeploy-2.0.1-py3.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/run.py", line 71, in make_wsgi_app
    starter.prepare()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/starter.py", line 41, in prepare
    self.startZope()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/starter.py", line 98, in startZope
    Zope2.startup_wsgi()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/__init__.py", line 50, in startup_wsgi
    _startup()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/App/startup.py", line 117, in startup
    DB = dbtab.getDatabase('/', is_root=1)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/datatypes.py", line 266, in getDatabase
    db = factory.open(name, self.databases)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/datatypes.py", line 159, in open
    DB = self.createDB(database_name, databases)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/datatypes.py", line 156, in createDB
    return ZODBDatabase.open(self, databases)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/ZODB-5.5.1-py3.7.egg/ZODB/config.py", line 131, in open
    storage = section.storage.open()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/ZODB-5.5.1-py3.7.egg/ZODB/config.py", line 213, in open
    base = self.config.base.open()
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/ZODB-5.5.1-py3.7.egg/ZODB/config.py", line 207, in open
    return FileStorage(config.path, **options)
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/ZODB-5.5.1-py3.7.egg/ZODB/FileStorage/FileStorage.py", line 306, in __init__
    read_only=read_only,
  File "/home/bitouze/test-plone-5-2-python3/buildout-cache/eggs/ZODB-5.5.1-py3.7.egg/ZODB/FileStorage/FileStorage.py", line 1619, in read_index
    raise FileStorageFormatError(name)
ZODB.FileStorage.FileStorage.FileStorageFormatError: /home/bitouze/test-plone-5-2-python3/zinstance/var/filestorage/Data.fs
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/bitouze/test-plone-5-2-python3/zinstance/var/filestorage/Data.fs.lock' mode='r+' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed file <_io.BufferedRandom name='/home/bitouze/test-plone-5-2-python3/zinstance/var/filestorage/Data.fs.tmp'>
sys:1: ResourceWarning: unclosed file <_io.BufferedRandom name='/home/bitouze/test-plone-5-2-python3/zinstance/var/filestorage/Data.fs'>

dieter · September 12, 2019, 9:02pm

The pickle formats are incompatible between Python 2 and Python 3. Therefore, a Python 3 Plone cannot read the storage file created under Python 2. There is an external program (zodbupdate when I remember right) which can convert the storage. Watch out for corresponding documentation.

jaroel · September 12, 2019, 9:25pm

https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html is quite the read, but it should cover the conversion to python3

dbitouze · September 13, 2019, 9:36am

Thanks. Problems pop up when I verify the integrity of the database in Python 2 but that's another story to be told in another thread

espenmn · September 13, 2019, 11:52am

Just an idea: if someone that knows more could comment it would be great:

What if one did…

upgrade site to 5.2 on python 2.7… do all the migration etc.
Add a new 5.2 / python 3 instance and include the same add-ons
Export a ZEXP from 1)
Import the ZEXP in 2)

erral · September 13, 2019, 12:31pm

I would never do a ZEXP import/export from python 2 to 3. To tell the truth I would never do a ZEXP import/export.

There is a documented procedure to migrate a python 2 Plone to python 3, that includes migrating the ZODB.

You can get the documentation here: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

dbitouze · September 13, 2019, 12:45pm

OK but, unfortunately, not that easy.

erral · September 13, 2019, 1:00pm

It could be in some cases, I migrated 2 databases without issues.

espenmn · September 13, 2019, 1:02pm

If you have identical Plone versions and identical buildout I would say it is OK (I do it 'a lot', never had any problems).

With different python versions I have no idea.

agitator · September 13, 2019, 1:30pm

Be careful with zexp export/imports https://github.com/plone/Products.CMFPlone/issues/2916