I've been hearing how the Django community is trying to make its upgrades "super slick easy" (and they're already pretty easy) and I feel a bit sad about our own Py3 story. The process is complicated and full of black magic. I say this knowing how much it has been worked on by so many of you.
Forcing our installed base to go through this in-place upgrade process is not very encouraging.
I think we should kill the in-place upgrade story and go with transmogrifier only.
The in-place migration is likely more suited for "smaller" sites where you want to preserve the existing configuration and content and when there are no changes in functionality, code or whatever. The main issue with in-place migration is the major pain in the a** factor in Plone projects called ZODB. The ZODB neither a decent database solution nor a reasonable database regarding migration. In a recent Plone 4.3 -> 5.2 migration we have high migration costs directly related to the ZODB. One part of the cost is related to an inefficient AT -> DX migration of Dexterity, the other part is related to the ZODB format migration. There is no offense included with this statement. I appreciated all work and efforts done here, in particular many thanks to Philipp.
An export-import migration is more suited when you have to reorganize your site anyway or doing large changes,
The overall problem (which is related to efforts and costs) is performance. I am currently helping with the migration of the University Ghent website. 90.000 content objects, 60 GB of data..some would call it "big data", I call this peanuts. Importing the data takes 36 hour - perhaps it might be possible bring the duration done with some optimizations. But out of the box performance without trickery is far away from being called "enterprise-level" CMS. Importing the same data (based on their JSON representation) into a document database takes 10 minutes.
So the decision of in-place vs. export-import migration depends on various factors. There is likely room for optimizations with the in-place migration and its complexity...but an export-import migration also is a complex operation.
I tend to agree that the official way should the lowest risk way. And the lowest risk way is not inplace unfortunately. Any reasonable size site we haven't done in place. There are just too many things that can bite you.
But transmogrifier is not simple either, or as pointed out, not fast. I suspect simplicity and speed could be improved on but not sure by who.
Things like turning off creation events and delaying indexing and other ways to simplify bulk creating objects might speed things up for large imports.
I think it's a bit comparing two tomatoes to a fruit salad. Does that make sense? Mabye not
Django is a low level framework, Plone an 'end product'. If you migrate a Django project, The 'Django Project' has a much smaller responsability footprint: they have/had to upgrade/fix their ORM, but the data is in a relational database, which was chosen by the end developers using Django in their project. Also everything built on top of the Django foudation in any Django project is the end developers task. Maybe you should compare the migration stories of DjangoCMS, FeinCMS and other cms'es built with Django.
Plone manages and is responsible to a very high level for your data with its own database.
And as Andreas correclty remarks, there is a lot of room for improvement there which we cannot share with other projects.
Curiosity: how fast is an in place migration if you already use Relstorage with Postgres/Mysql. Would it be much faster?
The complete Relstorage thing just moves the pain or burden down the street to a different layer. I would assume that a migration on Relstorage is slower due to a higher overhead. Nothing beats a ZEO server in simplicity, efficiency and stupidity.
What you are really complaining about is three other issues:
The ZODB is not forgiving if you change or remove code and do not properly update objects that reference that code. It's not simple to know all the possible places where references happen (IntIds and RelationCatalog!). https://www.starzel.de/blog/zodb-debugging discusses these problems in detail.
The migration from Archetypes to Dexterity (especially folder with a lot of content) can take a very long time. But the code for that has been around and in use since 2013. Complaining about it in 2020 is a little weird. There are alternative options to do these migrations if a in-place migration is not required. In these cases I usually choose a export/import using restapi serializers and deserializers.
Migrations between Plone-versions can be hard if you have addons that are no longer supported or have changed a lot. The reason is customization and addons that are used and written without properly thinking about uninstalling and upgrades. https://github.com/collective/collective.migrationhelpers contains helpers to deal with most issues that might arise there.
Imagine you want to migrate a Plonesite including configured Content-Rules, all users dashboard-configuration and custom workflow-variables created in the ZMI and everything else that might be in the site. Then your best and only option is a in-place migration. If at the same time you want to update from Plone 4 to 5.2 from AT to DX and from Py2 to Py3 then you should hire someone who knows what to do (wink) because your are asking to upgrade your car from diesel to electric and from red to yellow while the passengers are sitting in it.
If you plan to convert a RelStorage from Python 2 to 3, prepare a DB configuration snippet like so (this one is meant to be used in my collective.recipe.template section):
My case: an organization with just one Plone site. But it has 100.000 content items and a lot of custom add-ons. We have to migrate to Plone 5 / Python 3 without losing any functionality.
I think in this case the in-place migration path is more recommend than transmogrifier.
In-place migration has its advantages and should not be killed. Please
I do agree with Rafael. In-place migration is way better than Transmogrifier when you need to keep all the existing features, security settings, user information and so on.
Same here with a huge intranet, it works. But it was a lot of work. I have automate the migration from 2.5 to 5.2.2/py3, all inplace. 2 Weeks try and error with many hacks and many things that i learned.
Any migration scenario that does not include all of the above is like saying that your car can be upgraded, but only if it has not left the assembly line yet.
The issue, in my opinion, is not the technical challenge, but how to price such a migration for the customer so that a) the migration project does not cost more than a brand new site and b) I don't end up working for free. In the end, we should aim to bring as many of the remaining existing Plone sites up to date as possible and that means eliminating as many roadblocks to the upgrade as possible, including cost.
It pains me to say this, but I have a couple of WordPress sites continuously running hands-off for almost 15 years, with lots of user accounts, and sometimes I forget I have them because they keep auto-upgrading themselves and I only ever had one or two very minor security (mostly spam) issues.
The key point is risk.
inplace migration works until it doesn't and some weird left over data starts screwing things up.
i.e. starting with a clean site and a known set of data (ie just content + users) will be more work to get back to the same functionality but the chance of it biting you in the ass is less.
This also goes to @fulv point of quoting. Risk turns into a higher price for any fixed price quote. Upgrades are already hard to justify. You don't want to lose money on them as well.
Maybe its not a matter of "getting rid" of inplace but having an officially supported external migration story that is the more foolproof recommended way for newbies? ie you can have a painless riskfree migration of content only. Everything else you have to rebuild.
