Migrating Plone content using JSON format

Plone Support Add-on Development
, , ,
1

one of our customer wants to migrate the content of one of its Plone sites using a script that needs to get content based on location and tags, move that content to a new location and apply some new tags to it.

at first I was planning to use plone.importexport but after reading the requirements I decided that it would be better to write a custom script that could be using plone.restapi.

I read part of plone.importexport code and seems relatively easy to write such script, and my plan would be something like this:

  • make a catalog call to get the content tagged on the location specified
  • transform objects into a json string using the ISerializeToJson adapter
  • clean up the resulting JSON by removing the attributes that are not useful
  • recursively do the same thing on any object inside folderish content (mainly images and files), b64encoding their data
  • add information about possible alias of the content
  • creating a ZIP file containing all objects serialized in JSON format

on the importing part I would

  • read the ZIP file to get all serialized data
  • create the objects in the new location (or in the old location and move them after that) (I don't know if this can be done using also plone.restapi using the IDeserializeFromJson adapter)
  • create the associated aliases including a new one referencing the old location
  • recursively create all contained objects by b64decoding their data

I want to ask if anybody has done this before and if there is something I need to be aware.

any hints are highly appreciated.

2

To do this sort of things, in the past we have used collective.jsonify to export the contents from the old site and use collective.transmogrifier to import it in the new site.

3

thanks! I took a look on it but seemed to me a little abandoned; is it playing well with Dexterity-based content types?

I really want to avoid Transmogrifier; yes, it's cool but setting it up is quite complex.

4

Yes it should work exporting dexterity objects: https://github.com/collective/collective.jsonify/blob/master/collective/jsonify/wrapper.py#L96

We have used it exporting Archetypes objects from Plone 2.5, 3.x and 4.x without problems.

5

yes, I was testing it and seems to do the work; I made 2 fixes to it may be you can help me reviewing and merging them:

I would love a new release after that; @thet can you help on that?

1 Like
6

@erral thank you, very much, for your help! today I was able to do a full export and import of almost 200 Dexterity-based content objects among 2 different versions of the project I'm working on, and everything seems to be working as expected.

I'm using master branch of collective.jsonify to do the export and I was even able to process the redirect aliases I implemented yesterday.

the working transmogrifier pipline can be seen here:

I have now a couple of doubts: I need to change one of the content types default view during the migration; is there a pipeline section that can help me on that task?

the other problem seems to be a little bit more complicated: the collective.cover content type is being migrated except for the tiles configuration because they are stored in annotations in the object:

def __init__(self, context, request, tile):
    …
    self.tile = tile
    self.annotations = IAnnotations(self.context)
    self.key = '{0}.{1}'.format(ANNOTATIONS_KEY_PREFIX, tile.id)

    def set_configuration(self, configuration):
        self.annotations[self.key] = PersistentDict(configuration)

is there any other pipeline section for such use case, or do I have to create mine?

have a nice weekend!

1 Like
7

That's easy! Just use the inserter blueprint at an appropriate point in your pipeline, for example:

[change_view]
blueprint = collective.transmogrifier.sections.inserter
key = string:_layout
value = string:listing_view
condition = python:item['_type'] == 'Collection'

In case I misunderstood you there's also a '_defaultpage' key (rather than layout)

1 Like
8

@hvelarde Was working on plone.importexport at the sprint on the weekend. It would be good to know what could be added to make it fit your needs.
Seems like adding a search criteria during export is the only thing thats missing. The changes of path and tags you'd have to do anyway using excel or csv script.
Anything else required for this specific usecase to work well?

BTW I also considered if a search criteria widget for the import dialog might also make sense. ie it would only replace content included in the search result and drop everything else. Not sure how useful that would be.

9

I think the goal of the add-on is different from what we need and that has been the story since the beginning. our use case is simple: migrate a Plone site from an older version to a new one.

plone.importexport seem to be oriented to a different audience; we, as integrators, feel easier to migrate site using Transmogrifier.

1 Like
10

Wow. Thats exactly zero help.

11

I'm sorry, Dylan; you're totally right. my head is in another place and I better shouldn't have answered your question now.

at the end, it's the same I stated in this thread: GSoC 2018 Ideas: Improve Plone's import/export story

12

I think support for the jsonify format would be a nice to have. Possibly as an external script rather than as part of plone.importexport. We recently had to write a script to create a CSV of some of the jsonify data just to make it more usable for an export to another system. A similar script could turn an on disk jsonify dump into a format compatible with plone.importexport @hvelarde ? possibly even a transmogrify blueprint to output a p.importexport compatible format.

13