every now and then we need to do some upgrade steps that have to reindex lots of content (100k to 1Milion objects).
We have been using different approaches so far:
- write the UUID of each reindexed object in a file and upon a server restart load the file again to check if it needs to be reindexed (downsides: you need to clear the file once it finished, make sure the file is not removed...)
- add a marker interface on the object and discriminate for that interface on the catalog query (downside: you need to reindex all the objects again to remove the marker interface once the upgrade is done)
But today I thought about something different:
- create an ASCIILine registry record that holds a 'START' value
- the upgrade step checks the record value and keeps updating its value as long as its successfully reindexing objects, writing the value of how many objects have been reindexed so far
- if it finishes, updates the value to 'DONE' so that at the very beginning of the upgrade step it can check if it's already in that state and do nothing
The upsides of this is that you can control and see the value on /plone_registry itself and specially it survives restarts happily without having to use marker interfaces or external files that need clean up afterwards.
Do you have other approaches? Any downsides?