What's the best way to merge filestorages when migrating to relstorage?

Plone Support
,
1

Hello plone community!

We're finally moving into this century in tiny steps.
I'm testing out Relstorage (3.5.0) and zodbconvert at the moment and it seems to work fine.
But we currently have 2 filestorages. A data.fs and an extra catalog.fs.
What would be the best strategy to get them both into the relstorage?
zodbconvert doesn't accept multiple paths, but it there any other way, that i could achieve that? Or how would you go about it?

Site: Plone 5.2(py2.7)
Relstorage: 3.5.0 (psycopg2 2.8.3) and postgres

Your input will be appreciated!
Cheers
Ivo

2

I don't think you can do that with zodbconvert. Possible alternatives:

  1. Export the catalog as a .zexp file and reimport it into the new site.
  2. Clear the catalog, export/import it as .zexp, then reindex the catalog in the new site.
  3. Manually create the catalog and then reindex it.
3

I would not fight with this, the catalog can be recreated.

My process would look like:

  • in old site remove the catalog in ZMI,
  • convert the data.fs to new RelStorage site
  • in new site, recreate the catalog (i.e. by creating a vanilla site at /PloneVanilla, copy the catalog from ther to the old site,
  • re-apply all profiles of the old site to install indexes from add-ons/custom code),
  • clear and rebuild the catalog.

This may take a while, but you end up with a clean shiny catalog.

4

We did go the route of exporting the catalog.fs and importing it to the normal filestorage. Exporting is slightly faster, with almost 6h, than rebuilding it with over 8h.

The only tricky part was the removal of some catalog subscriber, that prevented us from deleting portal_catalog. We had to do that via debugger by hand.

Importing was seamless (took 30min)
and then i converted the whole thing for 50h into RelStorage.

Copied transactions: 3875978/3875978, 100%, 4.95 MB/s 21.29 True/s, 901946.33 MB

real    3107m9.961s
user    117m17.899s
sys     96m8.099s

This is mainly due to the use of NFS shares. In production we would make use of --increment.

The "step-by-step" route allows us to have 2 smaller downtimes* for

  • incorporating 2nd filestorage into first
  • switching instances after short --increment downtime*

instead of one giant one.

If you have any optimizations feel free to comment

Cheers
Ivo

*downtime for editors

5

We had this very same problem and solved like this (I can provide the scripts):

(on a server still with ZEO and the mount point configuration):

  • delete the mount point
  • unregister the catalog utility
  • copy a portal catalog from a temporal (empty) plone website (i.e. /Plone and /Plone-temporal)

(on the same server but with a configuration that does not have the mount point)

  • register the catalog utility

(on yet another server configuration)

  • run zodbconvert

(now configure and start your Zope workers)

  • clear and rebuild the catalog
    • to speed it up, we did a custom reindexing, to prioritize the objects that are more likely to be needed first
    • this way once the critical objects are reindexing you can put the website up again

Let me know if this is enough, or more details would be helpful, we are happy to share them, just that it takes time to put them in a way that makes sense out of our specific context :smile:

Fortunately we kept documentation and scripted everything so we could test and try the migration several times before we actually run it :crossed_fingers: