Migration: Transmogrifier, collective.jsonify and Plone 5

Plone Support
1

I exported my content as json files using collective.jsonify and I've created a migration.package and included the exported folder with json files as a 'content'
folder. So far so good.

Next I borrowed ideas from: marr weblog: Transmogrifier Revisited
I configured the package to read my transmogrifier pipeline using a setuphandler.py in my package.

Here's my pipeline:

[transmogrifier]
pipeline =
    source
    disable_versioning
    constructor
    enable_versioning
    deserializer
    schemaupdater
    reindexobject
    transitionsinserter
    workflowupdater
    logger

[source]
blueprint = collective.jsonmigrator.jsonsource
path = migration.package:content/

[disable_versioning]
blueprint = plone.app.transmogrifier.versioning.disable

[constructor]
blueprint = collective.transmogrifier.sections.constructor

[enable_versioning]
blueprint = plone.app.transmogrifier.versioning.enable

[deserializer]
blueprint = transmogrify.dexterity.deserializer

[schemaupdater]
blueprint = transmogrify.dexterity.schemaupdater

[reindexobject]
blueprint = plone.app.transmogrifier.reindexobject

[transitionsinserter]
blueprint = collective.transmogrifier.sections.inserter
key = string:_transitions
value = string:publish
condition = python:'_transitions' not in item

[workflowupdater]
blueprint = plone.app.transmogrifier.workflowupdater

[logger]
blueprint = collective.transmogrifier.sections.logger
level = INFO
name = Site migration log
key = id

My setuphandlers.py looks like this:

# -*- coding: utf-8 -*-
from Products.CMFPlone.interfaces import INonInstallable
from zope.interface import implementer
from collective.transmogrifier.transmogrifier import Transmogrifier


@implementer(INonInstallable)
class HiddenProfiles(object):

    def getNonInstallableProfiles(self):
        """Hide uninstall profile from site-creation and quickinstaller"""
        return [
            'my.migration:uninstall',
        ]


def initcontent(portal):
    transmogrifier = Transmogrifier(portal)
    # profile is defined in my/migration/transmogrify.zcml
    transmogrifier('my.jsoncontent.importer')

    
def post_install(context):
    """Post install script"""
    if context.readDataFile('mymigration_default.txt') is None:
        return
    portal = context.getSite()
    initcontent(portal)
    

def uninstall(context):
    """Uninstall script"""
    if context.readDataFile('mymigration_uninstall.txt') is None:
        return
    # Do something during the uninstallation of this package

When I install my special add-on I get lots of output like this in my log:

2016-01-26 14:52:58 WARNING collective.transmogrifier.constructor Container Plo
ne/tours/images does not exist for item /Plone/tours/images/r-main.jpg │
2016-01-26 14:52:58 INFO Site migration log ur-main.jpg
2016-01-26 14:52:58 WARNING collective.transmogrifier.constructor Container Plone/tours/images does not exist for item /Plone/tours/images/ka-main.jpg
2016-01-26 14:52:58 INFO Site migration log ka-main.jpg
2016-01-26 14:52:58 WARNING collective.transmogrifier.constructor Container Plone/tours/images does not exist for item /Plone/tours/images/museum-main.jpg

At the end of all of that, no new content in my site. Would appreciate guidance on this.

2

Apparently you are trying to import some contents before their containing folders exist (in your case for instance, you try to import /Plone/tours/images/r-main.jpg before /Plone/tours/images is created).
It depends on the order of the JSON source data you import, which depends on your export script.

So you need to fix your export script so it produces data in the proper order.
collective.jsonify should produce something right, did you make any change ? maybe you have ignored folders (and export only images for instance) ?

3

As I didn't alter the way it works, I'm wondering if this suggests a flaw in the collective.jsonmigrator code?

In the meantime to deal with the folder creation issue I've added a step that creates dependent folders before attempting to add content see below ** for emphasis.

[transmogrifier]
pipeline =
    source
    **folders**
    disable_versioning
    constructor
    enable_versioning
    deserializer
    schemaupdater
    reindexobject
    transitionsinserter
    workflowupdater
    logger
    
[source]
blueprint = collective.jsonmigrator.jsonsource
path = flyjamaica.migration:content/

 **[folders]**
 **blueprint = collective.transmogrifier.sections.folders**
  ...

Now I get a different error (it may be related to the fact that one of my json files has properties for the Plone Site object).

Here's the traceback:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFPlone.controlpanel.browser.quickinstaller, line 212, in __call__
  Module Products.CMFQuickInstallerTool.QuickInstallerTool, line 686, in installProducts
  Module Products.CMFQuickInstallerTool.QuickInstallerTool, line 603, in installProduct
   - __traceback_info__: ('my.migration',)
  Module Products.GenericSetup.tool, line 374, in runAllImportStepsFromProfile
   - __traceback_info__: profile-my.migration:default
  Module Products.GenericSetup.tool, line 1268, in _runImportStepsFromContext
  Module Products.GenericSetup.tool, line 1113, in _doRunImportStep
   - __traceback_info__: my.migration-postInstall
  Module my.migration.setuphandlers, line 28, in post_install
  Module my.migration.setuphandlers, line 20, in initcontent
  Module collective.transmogrifier.transmogrifier, line 63, in __call__
  Module collective.transmogrifier.sections.logger, line 39, in __iter__
  Module plone.app.transmogrifier.workflowupdater, line 27, in __iter__
  Module collective.transmogrifier.sections.inserter, line 19, in __iter__
  Module plone.app.transmogrifier.reindexobject, line 33, in __iter__
  Module transmogrify.dexterity.schemaupdater, line 59, in __iter__
  Module transmogrify.dexterity.serializer, line 67, in __iter__
  Module plone.app.transmogrifier.versioning, line 63, in __iter__
  Module collective.transmogrifier.sections.constructor, line 61, in __iter__
  Module Products.CMFCore.TypesTool, line 546, in _constructInstance
  Module Products.CMFCore.TypesTool, line 462, in _getFactoryMethod
ValueError: Product factory for Plone Site was invalid
4

To fix the ValueError that I was getting I removed two json files that had "Plone Site" as their content types.
After that transmogrifier pipeline ran successfully.

Now I have a folder named "Plone" which reflects the folder structure of the source site.
However most images have not come across successfully and all my items have been named with a "-1" appended to it. So if it was called "/myfolder/item" in the source site, it is now called /myfolder/item-1".

5

I guess transmogrifier imagined that "item" was already existing so it looked for anotehr id and picked "item-1".

Migrations are always difficult, and debugging is not easy.

My approach is:
1 - use a very small subset of data which produce the failure case,
2 - tune my pipeline and my blueprints properly to fix it,
3 - run the migration on a bigger subset,
4 - find a new problem (there is always a new problem),
5 - go back to step 1, until there is no problem anymore (see step 4: there is always a new problem) or until you are too tired to continue :slight_smile:

6

I don't know what kind of content or how much you're trying to export, but in our users group meeting last week we came up with a crazy workaround after we couldn't get transmogrifier to work.

I need to move a folder from Plone 4.3.2 to Plone 5 site. The folder had Archtypes and Dexterity content types and custom views. We exported the folder (.zexp), imported it into local vanilla Plone 4.3.2, then disabled workflows, content rules, and custom views. Then we took the data.fs and blobstorage, put them into a local vanilla Plone 5 site and migrated the site to Plone 5. Then we exported the folder and imported it into the site it needed to be in. The custom views were copy and pasted and the Dexterity content types were exported and imported directly, without issues.

Probably not relevant to what you're trying to do, but maybe this will help somebody else.

7

At this point, since I have all the json files, I've started work on my own import script. Feels like a few more hours of work but it is working so far.