CMFEditions/ZopeVersionControl issues in cleaning up uninstalled content

1

I'm having a bit of an issue with CMFEditions (or ZopeVersionControl, which it's built on) when migrating some sites from 4 to 5. On Plone 4, I had some old products that created content objects. They get migrated to whatever new Dexterity type they need to be and the old product gets uninstalled. At this point I can remove the old product from the buildout entirely and the site works fine.

But if you try to export as a zexp and import it into (an identical) environment, it is revealed that the uninstall of the old product wasn't as complete as you expected. CMFEditions/ZVC still knows about some old versions of those objects and when you try to import the site it will try to unpickle them - which causes the whole import to fail, because those classes do not exist. I don't want to have to include the old product in my buildout just to get this import to work, I want everything about it gone.

I did not find any cleanup tool offered by CMFEditions or if there is I couldn't find documentation on it. So below are my efforts to clean up as much as possible without breaking history on good content, or otherwise deleting more than is necessary. I'll be honest, I don't fully understand how CMFEditions/ZVC stores history and whatever shadow storage does. So this attempt is necessarily hacky. Ideally, CMFEditions should be updated to provide an interface or at least an API a product could reference in its uninstall method.

There are three types of objects I'm trying to delete:

  1. Corrupted objects that throw a POSKeyError when you try to access them

  2. PersistentBroken objects, from old classes that no longer exists

  3. Objects from classes that will not exist in the environment I want to move this to. In this example I have Event, EventFolder, and WebProject that are deprecated. The content from them was already migrated, but I need to clean out their histories here so that I can move this site to an environment that doesn't have their associated products in the buildout at all.

    def zvc_cleanup(dry=False):
    """ Removes broken content from CMFEditions repos (zvc = Zope version control)
    This has happened where an old product was migrated and removed, but objects of the old modules were
    left unmigrated in those repos. It tends to only be an error when pickling, such as on imports
    Also removes any POSKeyError objects and PersistentBroken objects in the history
    """
    from Products.IMSEvents.Event import Event
    from Products.IMSEvents.EventFolder import EventFolder
    from Products.IMSDocumentation.imsdoc import WebProject

    logger = logging.getLogger('ims.migrations')
    tool = plone.api.portal.get_tool('portal_historiesstorage')

    def is_deprecated_class(obj):
    deprecated_classes = [EventFolder, Event, WebProject]
    return [dc for dc in deprecated_classes if isinstance(obj.aq_base, dc)]

    bad_repo_ids = set()
    for sequence in tool.zvc_repo._histories:
    for version in tool.zvc_repo[sequence]._versions:
    try:
    obj = tool.zvc_repo[sequence].getVersionById(version)._data._object.object
    except POSKeyError:
    bad_repo_ids.add(sequence)
    logger.warn('POSKey error on bad object from zvc repo: %s' % sequence)
    else:
    if isinstance(obj, PersistentBroken):
    bad_repo_ids.add(sequence)
    logger.warn('Removing broken object from zvc repo: %s' % obj.module)
    elif is_deprecated_class(obj):
    bad_repo_ids.add(sequence)
    logger.warn('Removing AT content from zvc repo: %s' % obj.aq_base.module)

    if not dry:
    # can't delete them while traversing a generator anyway
    for bid in bad_repo_ids:
    del tool.zvc_repo._histories[bid]

    remove deleted items of deprecated class objects from shadow storage

    deleted =
    total_hids =
    hidhandler = plone.api.portal.get_tool('portal_historyidhandler')
    for hid in tool._getShadowStorage(autoAdd=False)._storage:
    workingCopy = hidhandler.unrestrictedQueryObject(hid)
    if not workingCopy:
    try:
    obj = tool.retrieve(hid).object.object
    except KeyError:
    logger.warn('Could not retrieve history id %s, removing from shadow storage' % hid)
    deleted.append(hid)
    else:
    if is_deprecated_class(obj):
    deleted.append(hid)
    total_hids.append(hid)

    logger.warn('Removing %d out of %d history ids in ZVC storage' % (len(deleted), len(total_hids)))
    if not dry:
    for hid in deleted:
    del tool._getShadowStorage(autoAdd=False)._storage[hid]

1 Like
2

I recommend you to try collective.revisionmanager, a Plone add-on that lets you manage Products.CMFEditions histories.

1 Like