Plone Migration 4.3.4 to 5.0 Errors

cthomas · February 2, 2016, 6:35pm

I get the following errors in the console while attempting to upgrade.

First Traceback:

Traceback (innermost last):
Module ZPublisher.Publish, line 146, in publish
Module Zope2.App.startup, line 301, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 329, in commit
Module transaction._transaction, line 443, in _commitResources
Module ZODB.Connection, line 567, in commit
Module ZODB.Connection, line 623, in _commit
Module ZODB.Connection, line 658, in _store_objects
Module ZODB.serialize, line 422, in serialize
Module ZODB.serialize, line 431, in _dump
PicklingError: Can't pickle <class 'plone.app.portlets.portlets.events.IEventsPortlet'>: attribute lookup plone.app.portlets.portlets.events.IEventsPortlet failed

The second Traceback:

Traceback (most recent call last):
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/OFS/SimpleItem.py", line 242, in raise_standardErrorMessage
v = s(**kwargs)
File "/data/plone/plone_env/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/FSPythonScript.py", line 127, in call
return Script.call(self, *args, **kw)
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/Shared/DC/Scripts/Bindings.py", line 322, in call
return self._bindAndExec(args, kw, None)
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/Shared/DC/Scripts/Bindings.py", line 359, in _bindAndExec
return self._exec(bound_data, args, kw)
File "/data/plone/plone_env/buildout-cache/eggs/Products.PythonScripts-2.13.2-py2.7.egg/Products/PythonScripts/PythonScript.py", line 344, in _exec
result = f(*args, **kw)
File "Script (Python)", line 35, in standard_error_message
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/Shared/DC/Scripts/Bindings.py", line 322, in call
return self._bindAndExec(args, kw, None)
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/Shared/DC/Scripts/Bindings.py", line 359, in _bindAndExec
return self._exec(bound_data, args, kw)
File "/data/plone/plone_env/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/FSPageTemplate.py", line 237, in _exec
result = self.pt_render(extra_context=bound_names)
File "/data/plone/plone_env/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/FSPageTemplate.py", line 177, in pt_render
self, source, extra_context
File "/data/plone/plone_env/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/Products/PageTemplates/PageTemplate.py", line 87, in pt_render
showtal=showtal)
File "/data/plone/plone_env/buildout-cache/eggs/zope.pagetemplate-3.6.3-py2.7.egg/zope/pagetemplate/pagetemplate.py", line 132, in pt_render
strictinsert=0, sourceAnnotations=sourceAnnotations
File "/data/plone/plone_env/buildout-cache/eggs/five.pt-2.2.3-py2.7.egg/five/pt/engine.py", line 93, in call
return self.template.render(**kwargs)
File "/data/plone/plone_env/buildout-cache/eggs/z3c.pt-3.0.0a1-py2.7.egg/z3c/pt/pagetemplate.py", line 163, in render
return base_renderer(**context)
File "/data/plone/plone_env/buildout-cache/eggs/Chameleon-2.22-py2.7.egg/chameleon/zpt/template.py", line 258, in render
return super(PageTemplate, self).render(**vars)
File "/data/plone/plone_env/buildout-cache/eggs/Chameleon-2.22-py2.7.egg/chameleon/template.py", line 191, in render
raise_with_traceback(exc, tb)
File "/data/plone/plone_env/buildout-cache/eggs/Chameleon-2.22-py2.7.egg/chameleon/template.py", line 171, in render
self._render(stream, econtext, rcontext)
File "7a08a03d8c95ae7cb4b1c77b909f1c4f.py", line 1492, in render
File "bfa9222644a5dfae03fec75db410dc47.py", line 800, in render_master
File "/data/plone/plone_env/buildout-cache/eggs/z3c.pt-3.0.0a1-py2.7.egg/z3c/pt/expressions.py", line 71, in render_content_provider
cp.update()
File "/data/plone/plone_env/buildout-cache/eggs/zope.viewlet-3.7.2-py2.7.egg/zope/viewlet/manager.py", line 112, in update
self._updateViewlets()
File "/data/plone/plone_env/buildout-cache/eggs/zope.viewlet-3.7.2-py2.7.egg/zope/viewlet/manager.py", line 118, in _updateViewlets
viewlet.update()
File "/data/plone/plone_env/buildout-cache/eggs/plone.app.layout-2.5.15-py2.7.egg/plone/app/layout/viewlets/common.py", line 131, in update
self.metatags = plone_utils.listMetaTags(context).items()
File "/data/plone/plone_env/buildout-cache/eggs/Products.CMFPlone-5.0-py2.7.egg/Products/CMFPlone/PloneTool.py", line 967, in listMetaTags
ISecuritySchema, prefix='plone')
File "/data/plone/plone_env/buildout-cache/eggs/plone.registry-1.0.2-py2.7.egg/plone/registry/registry.py", line 74, in forInterface
"for which there is no record." % (interface.identifier, name))
KeyError: 'Interface Products.CMFPlone.interfaces.controlpanel.ISecuritySchema defines a field enable_user_folders, for which there is no record.'

pbauer · February 2, 2016, 7:21pm

What do you mean by "while attempting to upgrade". Does it happen when you run the migration from 4.3.4 to 5 or at some other time?
BTW: It is a good practice to migrate from the newest release of one version to the newest release of the other (i.e. from 4.3.7 to 5.0.2 rather than from 4.3.4 to 5.0) since the newe releases often have bugfixes for migration-issues.

cthomas · February 2, 2016, 7:38pm

Hello,

This happens when upgrading. For sanity, here are the steps used.

Clean Install of Plone 5.0
Copy database from existing server (plone 4.3.4), along with blobstorage to the Plone 5.0 server.
Run migration (during this, the errors pop up in the console)

Hope this provides clarity. If you think migrating to 4.3.7 is best, I will attempt this to see if this corrects the issues.

pbauer · February 3, 2016, 9:30am

Use 5.0.2 (https://plone.org/products/plone/releases/5.0.2). How can you get two tracebacks when running the migration? It should stop at the first traceback. Also: When posting a traceback please use three ` - characters in the lines before and after so it is readable.

Like this:

import logging
log = logging.getLogger(__name__)

This is text

cthomas · February 3, 2016, 8:11pm

Update:

Upgrading to 4.3.7, then to 5.0 seems to have done the trick. I do not recall seeing anything in the docs suggesting this will make migration less painful. Correct me if I'm off base here.

pbauer, I could be wrong, but what appeared in the console looked like two separate tracebacks. What I posted, albeit incorrect by your standards, was copied and pasted directly from my console in the order it was displayed.

vangheem · February 3, 2016, 8:29pm

I think @pbauer is suggesting there was a bug in logging the error, not you?

pbauer · February 4, 2016, 7:55am

@cthomas great! I'm happy that the migration worked now. And you are right we should add a note that migrations should best be done from the lastest version of 4.x to the lastes version of 5.x.

Regarding the log you pasted it absolutely right and there may be a valid reason for the two tracebacks in a row. I simply suggested to add three `-characters in the lines before and after a traceback so it is formatted like source-code on http://community.plone.org to make it readable.

1letter · February 4, 2016, 2:15pm

Hi, i have upgrading from 4.3.7 to 5.0.2. In my buildout i set version.cfg to Index of /release/5-latest/ and the find-links option also. Now i get the following "Traceback" when i run the upgrade with "Dry Run" option in the zmi.

Dry run selected.
Traceback (most recent call last):
2016-02-04 15:09:33 ERROR plone.app.upgrade Upgrade aborted. Error:
Traceback (most recent call last):
File "/home/Development/Plone/Plone-5.0r1/buildout-cache/eggs/Products.CMFPlone-5.0.2-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
step['step'].doStep(setup)
File "/home/Development/Plone/Plone-5.0r1/buildout-cache/eggs/Products.GenericSetup-1.8.1-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
self.handler(tool)
File "/home/Development/Plone/Plone-5.0r1/buildout-cache/eggs/plone.app.upgrade-1.3.20-py2.7.egg/plone/app/upgrade/v50/betas.py", line 544, in to50rc3
settings = registry.forInterface(IIterateSettings)
File "/home/Development/Plone/Plone-5.0r1/buildout-cache/eggs/plone.registry-1.0.2-py2.7.egg/plone/registry/registry.py", line 74, in forInterface
"for which there is no record." % (interface.identifier, name))
KeyError: 'Interface plone.app.iterate.interfaces.IIterateSettings defines a field checkout_workflow_policy, for which there is no record.'
2016-02-04 15:09:33 INFO plone.app.upgrade End of upgrade path, main migration has finished.
2016-02-04 15:09:33 ERROR plone.app.upgrade The upgrade path did NOT reach current version.
2016-02-04 15:09:33 ERROR plone.app.upgrade Migration has failed
2016-02-04 15:09:33 INFO plone.app.upgrade Dry run selected, transaction aborted

What should i do? Any ideas?

Ok, I found a solution. I delete 'checkout_workflow_policy' in the Plone 4.3.7 zmi-> portal_properties -> site_properties.
and now the upgrade works fine.

cthomas · February 4, 2016, 2:27pm

@pbauer thank you for your suggestions and pointers. Migration went fine, but I'm not seeing blob related issues that I am looking into. This site was pretty "clean" in regards to products and was a good candidate for us to use here. The blob related issues happened when I was attempting to migrate to Dexterity Types via the built in utility. I'll post back here as I learn and find what is causing this to fail.

pbauer · February 4, 2016, 3:53pm

@1letter you need to run a upgrade-step of plone.app.iterate (working copy support) while you are still in Plone 4 (.../portal_setup/manage_upgrades). You could also simply reinstall it in portal_quickinstaller. That one is actually covered in https://youtu.be/bQ-IpO-7F00?t=23m29s

pbauer · February 4, 2016, 3:54pm

@cthomas if you have more info please report a issue at https://github.com/plone/plone.app.contenttypes/issues

1letter · March 4, 2016, 12:55pm

Hi all, now i have the same error like @cthomas,
KeyError: 'Interface Products.CMFPlone.interfaces.controlpanel.ISecuritySchema defines a field enable_user_folders, for which there is no record.'
but i upgrade from 4.3.7 to 5.0.2, how @pbauer explained. What is the best way to add the record to the registry? The record come from an interface wich ist not present in Plone 4.3.7. is there an possibility or a workaround to add the record?
best regards

tkimnguyen · March 5, 2016, 3:10pm

Please include the full error message / stack trace

1letter · March 7, 2016, 8:35am

Ok, i have found the the error. My traceback gave me an error in the function fix_select_all_existing_collections of plone.app.querystring, "no attribute found...". I added the following snippet to the code in Line 46: if hasattr(obj, 'query'):, and the upgrade runs successfully.

jwimmer · July 12, 2016, 9:25am

Hi,

i'm trying to upgrade a Plone 4.3.10 Site to Version 5.0.5 and get the following error, when doing a dry-run:

Dry run selected.
Starting the migration from version: 4313
Ran upgrade step: Miscellaneous
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Upgrade aborted. Error:
Traceback (most recent call last):
File "/opt/plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
step['step'].doStep(setup)
File "/opt/plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
self.handler(tool)
File "/opt/plone/buildout-cache/eggs/plone.app.upgrade-1.3.25-py2.7.egg/plone/app/upgrade/v50/alphas.py", line 34, in to50alpha3
loadMigrationProfile(context, 'profile-plone.app.upgrade.v50:to50alpha3')
File "/opt/plone/buildout-cache/eggs/plone.app.upgrade-1.3.25-py2.7.egg/plone/app/upgrade/utils.py", line 134, in loadMigrationProfile
context.runAllImportStepsFromProfile(profile, purge_old=False)
File "/opt/plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 379, in runAllImportStepsFromProfile
dependency_strategy=dependency_strategy)
File "/opt/plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 1414, in _runImportStepsFromContext
message = self._doRunImportStep(step, context)
File "/opt/plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 1226, in _doRunImportStep
return handler(context)
File "/opt/plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/resources/exportimport/bundles.py", line 28, in combine
combine_bundles(site)
File "/opt/plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/resources/browser/combine.py", line 109, in combine_bundles
if PRODUCTION_RESOURCE_DIRECTORY not in container:
TypeError: argument of type 'NoneType' is not iterable
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed
Dry run selected, transaction aborted

When trying to view the site, i get:

KeyError('Interface
Products.CMFPlone.interfaces.controlpanel.ISecuritySchema defines a
field enable_user_folders, for which there is no record.',) (Also, the
following error occurred while attempting to render the standard error
message, please see the event log for full details: Interface
Products.CMFPlone.interfaces.controlpanel.ISecuritySchema defines a
field enable_user_folders, for which there is no record.)

Before running the upgrade i removed all add-ons and copied the filestorage and blobstorage folders to the new Plone instanze. I also tried with Themes -> unstyled but still receive the error.
Any suggestion would be great.

Regards,

Jan

jwimmer · July 12, 2016, 11:36am

Hi,

i solved it. Just had to enable Diazo Theme Support in Plone 4.3.10.
The migration went without problems then.

tkimnguyen · July 12, 2016, 4:01pm

Thanks! I ran into that a couple of weeks ago and was totally blocked.

acsr · August 2, 2016, 8:31am

Here an additional migration log from a current (almost vanilla) Plone 4.3.10 migrated to Plone 5.0.5:

Prerequisite

We wanted to import content via WebDAV in Plone 5.0.5 for an easy shot. This did not work as expected at once, so we tried to do that in Plone 4 where we knew it worked in the past for reference.
Then we would try to migrate that generated content in place to Plone 5.
Migration dry run failed with (detailed traceback follows below):

...
Step languagetool has an invalid import handler
Upgrade aborted.
...

When googling error messages we found misleading things and finally came upon this this thread.

Solution: The approach Activating Diazo add-on in Plone 4 before migration from Plone 4 to 5, mentioned above in this post worked for us.

Purpose of the long answer

We recorded a detailled step-by-step followup what we did.
Improve this migration experience with this hints for the future.
Help to make the migration log more explicit on errors.

Outlook

Some distilled conclusion should end up in the Plone Migration Documentation after some review by others.

Hopefully we expect you may not get this in later releases.

Steps in Detail

We started with Plone 5 from an addon test and try to use WebDAV

Plone 4.3.10 was not the starting point (crazy?, not!)
The initial Plone was generated as a Plone 5 Test Setup in a Basic Package my.package generated using Mr.Bob for Plone 5.0.5
Instance bootstrapped using a virtualenv generated in the root of the package my.package
buildout went smooth, instance is working.
WebDAV port was activated
Content was imported using WebDAV via Drag&Drop using Cyberduck as WebDAV Client on Mac OS X El-Capitan 10.11.5
Import results in unusable empty objects in Plone 5 without any further adjustments

We start to try that in Plone 4 for comparison

We went back to the latest Plone 4 -> 4.3.10 using the same setup:

extend line in buildout.cfg pointing to Plone version.cfg file was changed from 5.0.5 to 4.3.10

[buildout]
extends = http://dist.plone.org/release/4.3.10/versions.cfg

Rerun buildout

Error caused by my.package generated for Plone 5

Plone drops an error related to a version constrain error on startup:

Version and requirements information containing plone.testing:
  [versions] constraint on plone.testing: 4.1.1
  Requirement of my.package[test]: plone.testing>=5.0.0
  Requirement of plone.app.robotframework[debug]: plone.testing
While:
  Installing instance.
Error: The requirement ('plone.testing>=5.0.0') is not allowed by your [versions] constraint (4.1.1)

This was due to a version constrain in the setup.py in the package 'my.package' pinning 'plone.testing':

extras_require={
    'test': [
        'plone.app.testing',
        # Plone KGS does not use this version, because it would break
        # Remove if your package shall be part of coredev.
        # plone_coredev tests as of 2016-04-01.
        'plone.testing>=5.0.0',
        'plone.app.contenttypes',
        'plone.app.robotframework[debug]',
    ],

Commenting out the line like this helped:

'plone.testing>=5.0.0', # disabled to backport compatibility to Plone 4.3.10

We have now a working Plone 4.3.10 up and running
we add the line weddav-adress=[Port] to the [instance] section of the buildout.cfg to activate WebDAV:

[instance]
recipe = plone.recipe.zope2instance
user = :
http-address = 8080
webdav-address = 8091
eggs =
...

Rerun Buildout

Success!

Import via WebDAV (works fine)

Connect to the instance using Cyberduck using unencrypted local WebDAV (not using the HTTPS one!)
Drag & Drop the HTML content structure into the Plone root in Cyberduck
et voila it works.
Clear and rebuild the catalog in the ZMI portal_catalog -> Advanced tab to avoid issues with content not introduced from the Plone machinery.
Finished content creation in Plone

In place migration to Plone 5 (with error on dry run)

Leave the package my.package untouched!
uninstall my.package in Plone 4 before you change to Plone 5 to avoid any interference. In case you have added any code related to the operation, keep it and make sure it works in Plone 5.
Backup your filestorage and blobstorage directories in case you want to migrate again after unsatifying results.
Move to Plone 5 again.
Extend line in buildout.cfg pointing to Plone version.cfg file from 4.3.10 to 5.0.5

[buildout]
extends = http://dist.plone.org/release/5.0.5/versions.cfg

Rerun buildout
Your site looks broken
Visit your instance root
Plone offers the migration
Important: Start the Plone migration as dry run
Scroll down to see the log
If there is no success You may get the error below

Failure Log

Dry run selected.
Starting the migration from version: 4313
Ran upgrade step: Miscellaneous
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Upgrade aborted. Error:
Traceback (most recent call last):
File "/Applications/Plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
step['step'].doStep(setup)
File "/Applications/Plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
self.handler(tool)
File "/Applications/Plone/buildout-cache/eggs/plone.app.upgrade-1.3.25-py2.7.egg/plone/app/upgrade/v50/alphas.py", line 34, in to50alpha3
loadMigrationProfile(context, 'profile-plone.app.upgrade.v50:to50alpha3')
File "/Applications/Plone/buildout-cache/eggs/plone.app.upgrade-1.3.25-py2.7.egg/plone/app/upgrade/utils.py", line 134, in loadMigrationProfile
context.runAllImportStepsFromProfile(profile, purge_old=False)
File "/Applications/Plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 379, in runAllImportStepsFromProfile
dependency_strategy=dependency_strategy)
File "/Applications/Plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 1414, in _runImportStepsFromContext
message = self._doRunImportStep(step, context)
File "/Applications/Plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/tool.py", line 1226, in _doRunImportStep
return handler(context)
File "/Applications/Plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/resources/exportimport/bundles.py", line 28, in combine
combine_bundles(site)
File "/Applications/Plone/buildout-cache/eggs/Products.CMFPlone-5.0.5-py2.7.egg/Products/CMFPlone/resources/browser/combine.py", line 109, in combine_bundles
if PRODUCTION_RESOURCE_DIRECTORY not in container:
TypeError: argument of type 'NoneType' is not iterable
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed
Dry run selected, transaction aborted

Some Remarks on the Migration Log

The irritating thing in this migration log is that the line

...
Step languagetool has an invalid import handler
...

is not related to the error but the last successfully executed step.

Suggestion 1

If there would be a line that claims what will be the next step to try, the log would be much more easier to read on failure.

One method to improve that is to write such an attempt line to the log before you start an operation and then only after success strip that line and replace that with the successful message.

This is more sophisticated than to just append loglines, and having this too verbose may draw back programmers who love to be brief in standard logs. But this spoils the overall very good reputation of the Plone Migration machinery. Maybe things are obvious for coders, but "being explicit" especially on failure is part of the "Zen of Python".

Suggestion 2

In Unix there is the stdout vs. stderr. make a similar apporach and split success and error log.

How we solved the issue

We applied the above trick and installed Diazo in Plone 4 before migration.

Stop the site
Change back the extend line in buildout.cfg to 4.3.10

[buildout]
extends = http://dist.plone.org/release/4.3.10/versions.cfg

Rerun Buildout
Copy back your backup of filestorage and blobstorage to the final Plone 4.3 step if you alterered the data already in Plone 5
Start the site
Install "Diazo theming support" in the add-on section of the site configuration but not applying any theme with it
Continue with the steps to migrate in place to Plone 5 above
Now the dry run should go smooth and you may do the actual run
If you still have issues, try to delete imported content using nested intervals disabling until you discover the devil.
for me it works now

Indetail documentation helps others to find something when they search

1letter · August 2, 2016, 9:14am

Ok, today i have open an Issue, because i run in the same Error. The Problem is the follwing:

if PRODUCTION_RESOURCE_DIRECTORY not in container:
TypeError: argument of type 'NoneType' is not iterable

My Solution, add a small check to:

"""
Products/CMFPlone/resources/browser/combine.py
"""
def combine_bundles(context):
    container = get_override_directory(context)
    # add check if container is None
    if not container:
        return
    if PRODUCTION_RESOURCE_DIRECTORY not in container:
        container.makeDirectory(PRODUCTION_RESOURCE_DIRECTORY)
    production_folder = container[PRODUCTION_RESOURCE_DIRECTORY]

    # store timestamp
    fi = StringIO()
    fi.write(datetime.now().isoformat())
    production_folder.writeFile("timestamp.txt", fi)

    # generate new combined bundles
    write_js(context, production_folder, 'default')
    write_js(context, production_folder, 'logged-in')
    write_css(context, production_folder, 'default')
    write_css(context, production_folder, 'logged-in')

1letter · August 9, 2016, 12:37pm

Another Solution for me is: install plone.resources via portal_quickinstaller in Plone 4. This worked for me, because the IResourceDirectory is then available in my LocalComponents. Check your Tab 'Components' in the ZMI and search to IResourceDirectory.