Which product should I use to migrate sites to Plone 5.2?

If I want to migrate a site from Plone 4 to Plone 5.2, without using the traditional migration of Data.fs from Plone, or migrate from another CMS, which product is recommended to use?

collective.transmogrifier?
collective.exportimport?

I saw that there is documentation on how to migrate using collective.transmogrifier:

https://training.plone.org/5/transmogrifier/

But I also saw that an official release of collective.transmogrifier with Python 3 support has not yet been released. I even opened a PR with the Python 3 branch:

collective/collective.transmogrifier#22

I saw that the collective.exportimport has a lot more activity.

So, which product should I use? Which one will be easier to use and will have more support?

@mauritsvanrees @pbauer @zopyx @cdw9?

You may check my blog entries on export/import migrations:

@pbauer 's new collective.exportimport module is clearly a good choice from what I have seen (but no experience with it) - at least for particular sort of migration requirements. So the choice of the migration approach may differ and depend on the particular situation.

2 Likes

Just noticed that I need to write about the importing part from ArangoDB into Plone.
The importer code is here: GitHub - collective/collective.plone5migration: collective.plone5migration

But it likely needs some more love (from my side) and it is usually required to adopt and adjust the code to each new migration. Each migration is different, each migration has its specialties.

I'm generally a fan of in-place migrations but dealing with no-longer supported addons and manual configuration can be really tough. I've been deep in just about every rabbit-hole there is and put most of what I learned into GitHub - collective/collective.migrationhelpers: Some helpers and examples to use during migrations and various pieces of documentation.

But starting fresh without any baggage is a great benefit of exporting old data and importing it into a new site. Not importing what you don't need is much easier that trying to get rid of it.
Since I wanted to see how far I got using the restapi I wrote GitHub - collective/collective.exportimport: Export and import content and other data from and to Plone.

A admin who wants a simple solution can use the forms to export and import data without any development. But for advanced use-cases it is relatively easy to extend and customize. I'm currently migrating a complex >80GB Intranet with 200.000 content items and it is going well.

Watch my talk about it and read the Readme:

Transmogrify is more mature. More people will have experience with it. I still want to use this in combination with collective.jsonify to migrate my personal site to Plone 5.2. It currently is Plone 2.5 so collective.exportimport is no option for the export. I might want to try it for the import actually.

At Zest we are using collective.exportimport to migrate one customer site from 4.3 to 5.2, and will likely use it more often, certainly for more sites of this customer. So far it is working fine, and me and colleague @fredvd have contributed to the package the last few weeks.

In my experience, transmogrify is hard to configure, you have to setup a pipeline with hooks from various packages, and probably write a few of your own. exportimport is newer and probably lacks some parts, but it builds on the solid foundation of plone.restapi. I would go with collective.exportimport.

The standard inline migration can still be a good option (I did that recently), but in your first sentence you say that you are looking for something else, so I don't go into that. (I will have to write a blog post on that migration some time.)

If you want to migrate from another CMS, then the flexibility and already available modules for transmogrifier will be a much better match. You'll also have to do much more work anyway by mapping fields/content/concepts/structure from another CMS to Plone.

As @mauritsvanrees wrote, him and me have tested and contributed to collective.exportimport in the last few weeks to fix the common migration case of Plone 4 with Archetypes content to Dexterity Plone 5.2 content.

technical thingie:

Collective.exportimport uses under the hood the (de-)serializers of plone.restapi to export and import content. But there are field name and small structural changes between the Archetypes and Dexterity serializer code. With a check box on the export content form you can activate some extra patches/renames on the exported json from Archetypes content items so it matches the Dexterity exported json and both should import the same using the import_content dexterity deserializer.

We'll have to add more documentation as it is a kind of 'manual' ETL migration for smaller projects. You have to stick to a certain order and some best practice, but with it you can migrate most of the default content.

We also together 'fixed' some of the 'only for small sites' limitations by adding a sequential json reader/parser and support also json files from the server. With this I was able to migrate 8Gb of .json serialized Files (base64) and as Philip writes above he now wants to take on 80Gb :open_mouth: :slight_smile:

It's kinda amazing that the plone.restapi improvements made in the last years to support the new Volto frontend in Plone 6 now indirectly also contribute to having this simplified ETL migration add'on. This can skip a lot of unknown unkowns in the migration of older sites to Python3 Plone 5.2/6 for the bulk of default content. to quote Philip:

But starting fresh without any baggage is a great benefit of exporting old data and importing it into a new site. Not importing what you don't need is much easier that trying to get rid of it.

This has bitten me very hard with a previous attempt last year to migrate this Plone 4 site with in place migration. The site is 13-14 years old, was and has collected so many small rabbit holes and bit rot, that you keep fixing edge cases in your migration code until it has sucked all motivation out of you to face another new traceback. :cold_face: :cold_face:

I'm just skipping old Style Collections (Topic)and will recreate them manually. There are like 40 of them in this project where most of them are small variants. You can pick and choose the most efficient route for individual CT's. fix/customise the export/import code, recreate manually, modifiy the .json.

The 'feedback loop' with collective.exportimport is also much quicker. Export, import, something breaks, fix a field modifier, or test by modifiying the json. export/import again, maybe on a part of the json. But I have some 'feature requests' to improve on this further, for example see import_content: Dry run option with more verbose deserializer validation feedback · Issue #15 · collective/collective.exportimport · GitHub.

3 Likes