[SOLVED] Existing content migration to use Dexterity failed

Plone Support Deployment & hosting
, ,
1

System: Debian 9.4
From plone 4.3.17 to plone 5.1.1 ( Plone 5.0.8 installation failed. There will be another post.)

After a success migration to plone 5.1.1, it stuck in the step of "Upgrade your existing content to use Dexterity".
I checked all the folders and files and can not find any issue to tell.

Here is the error output:

Traceback (innermost last):

    Module ZPublisher.Publish, line 138, in publish
    Module ZPublisher.mapply, line 77, in mapply
    Module ZPublisher.Publish, line 48, in call_object
    Module z3c.form.form, line 233, in __call__
    Module z3c.form.form, line 228, in update
    Module plone.app.z3cform.csrf, line 22, in execute
    Module z3c.form.action, line 98, in execute
    Module z3c.form.button, line 315, in __call__
    Module z3c.form.button, line 170, in __call__
    Module plone.app.contenttypes.migration.browser, line 353, in handle_migrate
    Module plone.app.contenttypes.migration.browser, line 188, in __call__
    Module plone.app.contenttypes.migration.migration, line 352, in migrate_folders
    Module plone.app.contenttypes.migration.migration, line 52, in migrate
    Module Products.contentmigration.basemigrator.walker, line 146, in go
    Module Products.contentmigration.basemigrator.walker, line 225, in migrate

MigrationError: MigrationError for obj at /keti/switch/shbpsh/2010/2010ybsb (Folder -> Folder):
Traceback (most recent call last):
  File "/opt/plone5/buildout-cache/eggs/Products.contentmigration-2.1.19-py2.7.egg/Products/contentmigration/basemigrator/walker.py", line 194, in migrate
    migrator.migrate()
  File "/opt/plone5/buildout-cache/eggs/Products.contentmigration-2.1.19-py2.7.egg/Products/contentmigration/basemigrator/migrator.py", line 220, in migrate
    method()
  File "/opt/plone5/buildout-cache/eggs/plone.app.contenttypes-1.4.9-py2.7.egg/plone/app/contenttypes/migration/migration.py", line 183, in migrate_contentrules
    copy_contentrules(self.old, self.new)
  File "/opt/plone5/buildout-cache/eggs/plone.app.contenttypes-1.4.9-py2.7.egg/plone/app/contenttypes/migration/utils.py", line 181, in copy_contentrules
    assign_rule(target_object, rule_id)
  File "/opt/plone5/buildout-cache/eggs/plone.app.contentrules-4.0.18-py2.7.egg/plone/app/contentrules/api.py", line 36, in assign_rule
    insert_assignment(storage[rule_id], path)
  File "/opt/plone5/buildout-cache/eggs/zope.container-3.11.2-py2.7-linux-x86_64.egg/zope/container/ordered.py", line 89, in __getitem__
    return self._data[key]
KeyError: 'rule-1'

Regards.

2

this insert_assignment call seems a bit weird with this storage[rule_id] that seems to get out of thin air.
I know lot of Zope code looks weird, but there should be some 'magic here!' comment no ? Also, it is supposed to be an api. Was introduced here:

Developer promised to document the assign_rule Api. Promised doc don't seem to be in the doco, though :frowning:

In the changelog for this file there is a commit that looks a bit ominous (all following commits are stylistic):

It seems implausible no one could have experienced a migration failure since this time. Unless you have done something extra special with your rules no one else has done.

In the interest of research maybe you could try to revert this specific change and try to restart migration in a test setup. Unless someone more informed has a better idea. I can't promise anything, I don't understand how any of this is even supposed to work:frowning:. If I had a test case it would be a lot easier, but by just reading code I have never looked at before it's little more than wild guess.

so try to add a 
    from plone.app.contentrules.rule import get_assignments
and replace
    insert_assignment(storage[rule_id], path)
by
    get_assignments(storage[rule_id]).insert(path)

If you have no idea about changing python code don't even try, of course !

3

In my 4.3.17 instance, there is no any rules displayed ( I can not remember any modification :frowning: earlier ） and I disabled rule globally.

4

seems stranger still that migration could fail in migrating non existing rules then.
I'd give it another try with 5.08 if I were you since direct migration from 4.3 to 5.1 has probably not been widely experienced at this time.

5

I downgraded setuptools and installed 5.0.8 :slight_smile:
But when I try to migrate my existing instance(plone 4.3.17) to 5.0.8, got error output:

The following list shows which upgrade steps are going to be run. Upgrading sometimes performs a catalog/security update, which may take a long time on large sites. Be patient.

(4399 → 5000)

        Run to50alpha1 upgrade profile
        Use lowercase for email login

(5000 → 5001)

        Miscellaneous

(5001 → 5003)

        Run to50alpha3 upgrade profile
        Upgrade editing control panel settings
        Upgrade maintenance control panel settings
        Upgrade navigation control panel settings
        Upgrade search control panel settings
        Upgrade site control panel settings

(5003 → 5004)

        Run to50beta1 upgrade profile
        Upgrade markup control panel settings
        Upgrade mail control panel settings
        Upgrade security control panel settings
        Remove Language Tool and migrate settings

(5004 → 5005)

        Run to50beta2 upgrade profile

(5005 → 5006)

        Run to50beta3 upgrade profile

(5006 → 5007)

        Run to50beta4 upgrade profile
        Run to50beta4 upgrade profile

(5007 → 5008)

        Run to50rc1 upgrade profile

(5008 → 5009)

        Run to50rc2 upgrade profile

(5009 → 5010)

        Run to50rc3 upgrade profile
        Unmark installed profiles that are no longer available.
        Mark products as installed for installed uninstallable profiles
        Cleanup uninstalled products

(5010 → 5011)

        Run to500 upgrade profile

(5011 → 5012)

        Unmark installed profiles that are no longer available.
        Mark products as installed for installed uninstallable profiles
        Cleanup uninstalled products
        Cleanup the skins tool.
        Remove fake kupu tool and related settings and resources.
        Run to501 upgrade profile.

(5012 → 5013)

        Run to502 upgrade profile.

(5013 → 5014)

        Run to503 upgrade profile.

(5014 → 5015)

        Miscellaneous

(5015 → 5016)

        Miscellaneous

(5016 → 5017)

        Run to507 upgrade profile.

(5017 → 5018)

        Fix double shared maxage
        Run to508 upgrade profile.

Dry run mode
Run the upgrade and show the result without actually writing anything to the database.
Upgrade report

Starting the migration from version: 4319
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Utilities registered.
Skins tool imported.
Archetype tool imported.
Properties tool imported.
Imported.
Step languagetool has an invalid import handler
Stylesheet registry imported.
Javascript registry imported.
Portlets imported
Cleaned up the toolset registry.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Control panel imported.
Archetype tool imported.
Browser layers imported
Step languagetool has an invalid import handler
Types tool imported.
'Folder' type info imported.
'Plone Site' type info imported.
'Collection' type info imported.
Stylesheet registry imported.
Javascript registry imported.
Cannot register portlet type portlets.Calendar because it is already registered.
Cannot register portlet type portlets.Events because it is already registered.
Portlets imported
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Browser layers imported
Step languagetool has an invalid import handler
Theme barceloneta applied
Theme enabled
upgrading keyring
add keyring to zope root if not done already
Ran upgrade step: Run to50alpha1 upgrade profile
Ran upgrade step: Use lowercase for email login
Ran upgrade step: Miscellaneous
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50alpha3 upgrade profile
Ran upgrade step: Upgrade editing control panel settings
Ran upgrade step: Upgrade maintenance control panel settings
Ran upgrade step: Upgrade navigation control panel settings
Ran upgrade step: Upgrade search control panel settings
Ran upgrade step: Upgrade site control panel settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta1 upgrade profile
Ran upgrade step: Upgrade markup control panel settings
Ran upgrade step: Upgrade mail control panel settings
Ran upgrade step: Upgrade security control panel settings
Ran upgrade step: Remove Language Tool and migrate settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta2 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Control panel imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-accessibility.init.
Removed record plone.resources/mockup-patterns-accessibility.export.
Removed record plone.resources/mockup-patterns-accessibility.deps.
Removed record plone.resources/mockup-patterns-accessibility.conf.
Removed record plone.resources/mockup-patterns-accessibility.url.
Removed record plone.resources/mockup-patterns-accessibility.css.
Removed record plone.resources/mockup-patterns-accessibility.js.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta3 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-tablesorter.init.
Removed record plone.resources/mockup-patterns-tablesorter.export.
Removed record plone.resources/mockup-patterns-tablesorter.deps.
Removed record plone.resources/mockup-patterns-tablesorter.conf.
Removed record plone.resources/mockup-patterns-tablesorter.url.
Removed record plone.resources/mockup-patterns-tablesorter.css.
Removed record plone.resources/mockup-patterns-tablesorter.js.
Removed record plone.resources/resource-collection-css.init.
Removed record plone.resources/resource-collection-css.export.
Removed record plone.resources/resource-collection-css.deps.
Removed record plone.resources/resource-collection-css.conf.
Removed record plone.resources/resource-collection-css.url.
Removed record plone.resources/resource-collection-css.css.
Removed record plone.resources/resource-collection-css.js.
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Upgrade aborted. Error:
Traceback (most recent call last):
  File "/opt/plone5/buildout-cache/eggs/Products.CMFPlone-5.0.8-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
    step['step'].doStep(setup)
  File "/opt/plone5/buildout-cache/eggs/Products.GenericSetup-1.8.8-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
    self.handler(tool)
  File "/opt/plone5/buildout-cache/eggs/plone.app.upgrade-2.0.4-py2.7.egg/plone/app/upgrade/v50/betas.py", line 367, in to50rc1
    migrate_linkintegrity_relations(portal)
  File "/opt/plone5/buildout-cache/eggs/plone.app.linkintegrity-3.1-py2.7.egg/plone/app/linkintegrity/upgrades.py", line 21, in migrate_linkintegrity_relations
    source_obj, target_obj, relationship=referencedRelationship)
  File "/opt/plone5/buildout-cache/eggs/Products.Archetypes-1.10.18-py2.7.egg/Products/Archetypes/ReferenceEngine.py", line 345, in deleteReference
    tID, tobj = self._uidFor(target)
  File "/opt/plone5/buildout-cache/eggs/Products.Archetypes-1.10.18-py2.7.egg/Products/Archetypes/ReferenceEngine.py", line 538, in _uidFor
    raise ReferenceException, "%r not referenceable" % uobject
ReferenceException: None not referenceable
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed

Will check it later: https://github.com/plone/plone.app.linkintegrity/issues/48

6

yes the invalid content workaround is in 5.1 not in 5.0 so you could add it yourself to Plone code to try to get further at least in Plone 5.0

7

I rebuilt the reference_catalog, but failed again with a AttributeError: 51436570636e. These are the full error output:

Starting the migration from version: 4319
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Utilities registered.
Skins tool imported.
Archetype tool imported.
Properties tool imported.
Imported.
Step languagetool has an invalid import handler
Stylesheet registry imported.
Javascript registry imported.
Portlets imported
Cleaned up the toolset registry.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Control panel imported.
Archetype tool imported.
Browser layers imported
Step languagetool has an invalid import handler
Types tool imported.
'Folder' type info imported.
'Plone Site' type info imported.
'Collection' type info imported.
Stylesheet registry imported.
Javascript registry imported.
Cannot register portlet type portlets.Calendar because it is already registered.
Cannot register portlet type portlets.Events because it is already registered.
Portlets imported
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Browser layers imported
Step languagetool has an invalid import handler
Theme barceloneta applied
Theme enabled
upgrading keyring
add keyring to zope root if not done already
Ran upgrade step: Run to50alpha1 upgrade profile
Ran upgrade step: Use lowercase for email login
Ran upgrade step: Miscellaneous
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50alpha3 upgrade profile
Ran upgrade step: Upgrade editing control panel settings
Ran upgrade step: Upgrade maintenance control panel settings
Ran upgrade step: Upgrade navigation control panel settings
Ran upgrade step: Upgrade search control panel settings
Ran upgrade step: Upgrade site control panel settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta1 upgrade profile
Ran upgrade step: Upgrade markup control panel settings
Ran upgrade step: Upgrade mail control panel settings
Ran upgrade step: Upgrade security control panel settings
Ran upgrade step: Remove Language Tool and migrate settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta2 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Control panel imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-accessibility.init.
Removed record plone.resources/mockup-patterns-accessibility.export.
Removed record plone.resources/mockup-patterns-accessibility.deps.
Removed record plone.resources/mockup-patterns-accessibility.conf.
Removed record plone.resources/mockup-patterns-accessibility.url.
Removed record plone.resources/mockup-patterns-accessibility.css.
Removed record plone.resources/mockup-patterns-accessibility.js.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta3 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-tablesorter.init.
Removed record plone.resources/mockup-patterns-tablesorter.export.
Removed record plone.resources/mockup-patterns-tablesorter.deps.
Removed record plone.resources/mockup-patterns-tablesorter.conf.
Removed record plone.resources/mockup-patterns-tablesorter.url.
Removed record plone.resources/mockup-patterns-tablesorter.css.
Removed record plone.resources/mockup-patterns-tablesorter.js.
Removed record plone.resources/resource-collection-css.init.
Removed record plone.resources/resource-collection-css.export.
Removed record plone.resources/resource-collection-css.deps.
Removed record plone.resources/resource-collection-css.conf.
Removed record plone.resources/resource-collection-css.url.
Removed record plone.resources/resource-collection-css.css.
Removed record plone.resources/resource-collection-css.js.
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50rc1 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50rc2 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/resource-plone-app-discussion-stylesheets.init.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.export.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.deps.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.conf.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.url.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.css.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.js.
could not upgrade calendar_starting_year property
could not upgrade calendar_future_years_available property
Ran upgrade step: Run to50rc3 upgrade profile
Setting installed version of profile Products.PloneLanguageTool:PloneLanguageTool as unknown.
Setting installed version of profile plone.app.jquerytools:default as unknown.
Setting installed version of profile Products.TinyMCE:upgrade_12_to_13 as unknown.
Setting installed version of profile plone.app.jquery:default as unknown.
Setting installed version of profile Products.TinyMCE:TinyMCE as unknown.
Setting installed version of profile collective.z3cform.datetimewidget:default as unknown.
Setting installed version of profile archetypes.referencebrowserwidget:default as unknown.
Setting installed version of profile plone.app.search:default as unknown.
Setting installed version of profile archetypes.querywidget:default as unknown.
Setting installed version of profile Products.CMFPlone:plone-content as unknown.
Setting installed version of profile plonetheme.sunburst:default as unknown.
Ran upgrade step: Unmark installed profiles that are no longer available.
Ran upgrade step: Mark products as installed for installed uninstallable profiles
Ran upgrade step: Cleanup uninstalled products
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to500 upgrade profile
Ran upgrade step: Unmark installed profiles that are no longer available.
Ran upgrade step: Mark products as installed for installed uninstallable profiles
Ran upgrade step: Cleanup uninstalled products
Removed no longer existing path CMFPlacefulWorkflow from skin selection Plone Default.
Removed no longer existing path classic_styles from skin selection Plone Default.
Removed no longer existing path CMFPlacefulWorkflow from skin selection Sunburst Theme.
Removed no longer existing path sunburst_images from skin selection Sunburst Theme.
Removed no longer existing path sunburst_templates from skin selection Sunburst Theme.
Removed no longer existing path sunburst_styles from skin selection Sunburst Theme.
Removed no longer existing path classic_styles from skin selection Sunburst Theme.
Ran upgrade step: Cleanup the skins tool.
Ran upgrade step: Remove fake kupu tool and related settings and resources.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Upgrade aborted. Error:
Traceback (most recent call last):
  File "/opt/plone5/buildout-cache/eggs/Products.CMFPlone-5.0.8-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
    step['step'].doStep(setup)
  File "/opt/plone5/buildout-cache/eggs/Products.GenericSetup-1.8.8-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
    self.handler(tool)
  File "/opt/plone5/buildout-cache/eggs/plone.app.upgrade-2.0.4-py2.7.egg/plone/app/upgrade/v50/final.py", line 84, in to501
    refresh_getIcon_catalog_metadata(context)
  File "/opt/plone5/buildout-cache/eggs/plone.app.upgrade-2.0.4-py2.7.egg/plone/app/upgrade/v50/final.py", line 72, in refresh_getIcon_catalog_metadata
    obj = brain._unrestrictedGetObject()
  File "/opt/plone5/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/CatalogBrains.py", line 80, in _unrestrictedGetObject
    return parent.unrestrictedTraverse(self.getPath())
  File "/opt/plone5/buildout-cache/eggs/Zope2-2.13.26-py2.7.egg/OFS/Traversable.py", line 285, in unrestrictedTraverse
    raise e
AttributeError: 51436570636e
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed

Any way to debug this to find the cause?

Regards.

8

There is progress (looking at the bright side always), you have now passed the 5.0 step, you are really in 5.0 (joy) it's the step to 5.01 that is failing.

It seems that there is still something hosed in your db, even after the catalog rebuilding.

Plone migration code is accessing the whole catalog, getting at the object, and doing some mystical magic with it to update the catalog to reflect the fact that icons have completely mutated in Plone 5.

The mystical magic is not the problem, it's the getting at the object that is failing. If is done with an internal function.
I have done simple code like this before for another poster, I just have to update it to do another version with the internal features that Plone migration code use, so that you can reproduce the problem but this time with some indication on what is the problematic object. At this point you should know if you can either fix the object in the Plone 4.3 site (it could be enough to access it and to save it again in the Plone UI) or to delete it altogether if it's not necessary.

If you want to ask why after a catalog rebuild Plone code can still fail on an object access by the catalog, it's a very good question, unfortunately I don't have the answer right now :slight_smile:

Here the code that you will have to run using bin/instance run myfile.py (just save the code in 'myfile.py'). If you don't have a bin/instance you should have something like that. I assume that you know how to run a python script in Plone, if not, ask, I'll provide details (note that DuckDuckGo or other search engines know also about that)

plone=app.Plone
from zope.component.hooks import setSite
setSite(plone)
pc = plone.portal_catalog
zcatalog = pc._catalog

lstbad = []

docs = zcatalog.unrestrictedSearchResults()

for d in docs:
    try:
        doc=d._unrestrictedGetObject()
    except AttributeError:
        lstbad.append((d.getPath(), 'invalid object'))

print('finished, %d good docs, %d bad docs' % (len(docs)-len(lstbad), len(lstbad)))
for x in lstbad:
    print(x)

You should run this code on your Plone 5 instance,
The idea is to find the failing objects in the half migrated database, fix in Plone 4, migrate again.

Hope it will run well for you and not return thousands of objects.
Now if you excuse me after having to read this part of Plone code I want soft music

9

Thank you very much :smile:
Could you tell me how to run myfile.py using bin/instance? Where location should the file be saved?

P.S. There are client1 and client2 in directory /opt/plone508/zeocluster/bin/. I had ever run "bin/client1 debug" to find some info.

10

bin/client1 run myfile.py

you can place myfile.py everywhere plone_daemon user has reading rights, in plone_daemon home, in /tmp, in your home directory... don't matter.

11

Here is what I run and output:

hugo@ubuntu:/opt/plone5/zeocluster$ sudo nano myfile.py
hugo@ubuntu:/opt/plone5/zeocluster$ sudo bin/client1 run myfile.py
finished, 563 good docs, 1 bad docs
('/keti/switch/51436570636e/753362a54e0e8bc45ba1768451436570636e/copy_of_201259296cb3533a89c45212987976ee/2011/xkt2011/czwl/59296cb3533a8bfe98987f514e0a753362a55b8963924e0e64cd4f5c8bf4660e', 'invalid object')

But I can not find the invalid object in my plone 4. There is only one folder in switch.

     You are here: Home / switch
        This page does not seem to exist…
        We apologize for the inconvenience, but the page you were trying to access is not at this address. You can use the links below to help you find what you are looking for.

        If you are certain you have the correct web address but are encountering an error, please contact the Site Administration.
12

did you try to actually delete this 'switch' folder in the UI ? I have no idea if Plone will allow it but it would solve the problem in a really simple way if it did. If it succeeds do another catalog rebuild and you should be good.

13

1.The only folder named shbpsh under the switch folder. No anything named 51436570636e.
I think there may be some error in indexed and catalog.
How can I make sure all are rebuilt.

  1. I copy the folder shbpsh to a new-build folder, then try to delete the folder switch (with pages including links to some links outside of my plone instance) , I get a message:

    Potential link breakage
    By deleting this item, you will break links that exist in the items listed below. If this is indeed what you want to do, we recommend that you remove these references first.

Then I click "yes to all", I get a message:

Info
    /keti/switch could not be deleted.

But I can not see "switch" in UI and ZMI.
Where it could be? Maybe this is the cause of error.

14

1.I built a new folder call cundang and copied all contents from switch to cundang.
2.Cause of "could not be deleted" info, I clear and rebuild catalog in portal_catalog
3. Upgrade to 5.0.8 DONE :smile:
4. After a success migration to plone 5.0.8, it stuck in the step of “Upgrade your existing content to use Dexterity”, the same error:

Traceback (innermost last):

    Module ZPublisher.Publish, line 138, in publish
    Module ZPublisher.mapply, line 77, in mapply
    Module ZPublisher.Publish, line 48, in call_object
    Module z3c.form.form, line 233, in __call__
    Module z3c.form.form, line 228, in update
    Module plone.app.z3cform.csrf, line 21, in execute
    Module z3c.form.action, line 98, in execute
    Module z3c.form.button, line 315, in __call__
    Module z3c.form.button, line 170, in __call__
    Module plone.app.contenttypes.migration.browser, line 397, in handle_migrate
    Module plone.app.contenttypes.migration.browser, line 223, in __call__
    Module plone.app.contenttypes.migration.migration, line 347, in migrate_folders
    Module plone.app.contenttypes.migration.migration, line 50, in migrate
    Module Products.contentmigration.basemigrator.walker, line 144, in go
    Module Products.contentmigration.basemigrator.walker, line 223, in migrate

MigrationError: MigrationError for obj at /keti/cundang/shbpsh/2010/2010ybsb (Folder -> Folder):
Traceback (most recent call last):
  File "/opt/PPP/buildout-cache/eggs/Products.contentmigration-2.1.16-py2.7.egg/Products/contentmigration/basemigrator/walker.py", line 192, in migrate
    migrator.migrate()
  File "/opt/PPP/buildout-cache/eggs/Products.contentmigration-2.1.16-py2.7.egg/Products/contentmigration/basemigrator/migrator.py", line 220, in migrate
    method()
  File "/opt/PPP/buildout-cache/eggs/plone.app.contenttypes-1.2.23-py2.7.egg/plone/app/contenttypes/migration/migration.py", line 178, in migrate_contentrules
    copy_contentrules(self.old, self.new)
  File "/opt/PPP/buildout-cache/eggs/plone.app.contenttypes-1.2.23-py2.7.egg/plone/app/contenttypes/migration/utils.py", line 198, in copy_contentrules
    assign_rule(target_object, rule_id)
  File "/opt/PPP/buildout-cache/eggs/plone.app.contentrules-4.0.16-py2.7.egg/plone/app/contentrules/api.py", line 36, in assign_rule
    insert_assignment(storage[rule_id], path)
  File "/opt/PPP/buildout-cache/eggs/zope.container-3.11.2-py2.7-linux-x86_64.egg/zope/container/ordered.py", line 89, in __getitem__
    return self._data[key]
KeyError: 'rule-1'
15

Coulď it be possible to create å content rule (rule-1) and migrate. Then delete the content and the rule

16

I forgot if I had ever created rule. It's been online for 7 years.
Now there is no any rule on the panel and checked 'disabled globally'.

What can I do? Need your help.

17

My suggestion is that you add a rule (with name rule-1), at your site//@@rules-controlpanel

then migrate/upgrade and delete the rule

18

well, follow @espenm suggestion if it works it could be a real simple solution. Just enable the rules and add a new one without picking a content-type or even an action. Then restart the Dex migration.

19

Good news, the error of rule-1 is resolved. Thanks, @espenm and @gp54321
Another problem in the step of “Upgrade your existing content to use Dexterity” (this time, I copied all contents of my pruduction instance to this instance for test), but I did clear and rebuild the catalog in portal_catalog.

Traceback (innermost last):
Module ZPublisher.Publish, line 138, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 48, in call_object
Module z3c.form.form, line 233, in call
Module z3c.form.form, line 228, in update
Module plone.app.z3cform.csrf, line 21, in execute
Module z3c.form.action, line 98, in execute
Module z3c.form.button, line 315, in call
Module z3c.form.button, line 170, in call
Module plone.app.contenttypes.migration.browser, line 397, in handle_migrate
Module plone.app.contenttypes.migration.browser, line 162, in call
Module plone.app.contenttypes.migration.utils, line 284, in store_references
Module plone.app.contenttypes.migration.utils, line 320, in get_all_references
Module plone.app.contenttypes.migration.utils, line 309, in catalog_get_all
Module Products.ZCatalog.Lazy, line 132, in getitem
IndexError: 0

20

amazing that there is a problem like that after a catalog rebuild.

Another 2 cents code then:

from Products.Archetypes.config import REFERENCE_CATALOG
from Products.CMFCore.utils import getToolByName
plone=app.Plone
from zope.component.hooks import setSite
setSite(plone)
catalog = getToolByName(plone, REFERENCE_CATALOG, None)
unique_idx='UID'
for it in catalog._catalog.data:
    try:
        x1 = catalog._catalog.getIndexDataForRID(it)
    except:
        print('bad thing with ' + str(it))
    else:
        try:
            x2 = x1[unique_idx]
            x3 = catalog({ unique_idx: x2 })
        except:
            print('another bad thing with ' + str(x1))
        else:
            if len(x3) != 1:
                print('problem with :' + str(x1))