Upgrade steps that need to survive server restarts

[gforcada]

Hi all,

every now and then we need to do some upgrade steps that have to reindex lots of content (100k to 1Milion objects).

We have been using different approaches so far:

• write the UUID of each reindexed object in a file and upon a server restart load the file again to check if it needs to be reindexed (downsides: you need to clear the file once it finished, make sure the file is not removed...)
• add a marker interface on the object and discriminate for that interface on the catalog query (downside: you need to reindex all the objects again to remove the marker interface once the upgrade is done)

But today I thought about something different:

• create an ASCIILine registry record that holds a 'START' value
• the upgrade step checks the record value and keeps updating its value as long as its successfully reindexing objects, writing the value of how many objects have been reindexed so far
• if it finishes, updates the value to 'DONE' so that at the very beginning of the upgrade step it can check if it's already in that state and do nothing

The upsides of this is that you can control and see the value on /plone_registry itself and specially it survives restarts happily without having to use marker interfaces or external files that need clean up afterwards.

Do you have other approaches? Any downsides?

[hvelarde]

check this thread:

Best practices on reindexing the catalog

we are working on some upgrade steps for an add-on to make some changes on stuff like updating static resources and interfaces used and provided by a Dexterity-based content type. as part of the upgrade process we need to reindex all items in the catalog for this specific content type and that could be tricky sometimes as sites could have tens of thousands of object instances. my main concerns here are memory usage, speed and resiliency. this is the code we have right know: def get_valid_obj…

[dieter (Dieter Maurer)]

@gforcada

Do you have other approaches?

This sounds like a typical queueing task (your file based approach in fact implements a queue). There are solutions to implement a queue in the ZODB (e.g. zc.queue, z3c.taskqueue). With an appropriate design (i.e. a general task executor), there would be no need to delete the queue at the end; it would just become empty.

[dieter (Dieter Maurer)]

@gforcada

Any downsides?

If your Zope accepts requests before the reindexing is complete, new objects may be created. While they would not need reindexing, the reindexing process may not know that. As a consequence, you may not know the total number of reindexed objects and your must be careful with the termination check.