Maintaining zope intids on zexp imports

Plone Support
, ,
1

I've been having several issues moving Plone sites via zexp exports (yes it is from identical buildouts which is the only way this is supposed to be supported). This latest error seems to be a KeyError because it can't find the intid key reference for an object when trying to create a 'Related Item'.

I have tested this with a blank, out-of-the-box Plone site (5.0.5) and get a result like this
Traceback (innermost last):

    Module ZPublisher.Publish, line 138, in publish
    Module ZPublisher.mapply, line 77, in mapply
    Module ZPublisher.Publish, line 48, in call_object
    Module plone.z3cform.layout, line 66, in __call__
    Module plone.z3cform.layout, line 50, in update
    Module plone.dexterity.browser.add, line 130, in update
    Module plone.z3cform.fieldsets.extensible, line 59, in update
    Module plone.z3cform.patch, line 30, in GroupForm_update
    Module z3c.form.group, line 145, in update
    Module plone.app.z3cform.csrf, line 21, in execute
    Module z3c.form.action, line 98, in execute
    Module z3c.form.button, line 315, in __call__
    Module z3c.form.button, line 170, in __call__
    Module plone.dexterity.browser.add, line 105, in handleAdd
    Module z3c.form.form, line 263, in createAndAdd
    Module plone.dexterity.browser.add, line 72, in create
    Module z3c.form.form, line 51, in applyChanges
    Module plone.app.relationfield.widget, line 141, in set
    Module five.intid.intid, line 41, in getId
    Module zope.intid, line 89, in getId

KeyError: <Folder at /ootb/news>

Interestingly, any existing related items are fine, I am just having problems creating new ones. So it can't just be that intids were never registered for all of these objects. Indeed if I try to run the profile for plone.app.intids I get "Assigned intids to 0 content objects, 6 objects already had intids."

I dug a bit deeper into zope.intid.IntIds.getId (this is getUtility(IIntIds).getId). That method actually does two things, first it tries to do IKeyReference(ob), and then sees if that result is indexed in getUtility(IIntIds).ids (a btree). That interface is indeed provided by the object, but the result is not indexed in that btree. That causes the method to raise a KeyError. So here's an example where I looked into one particular object on my original site.

>>> intids = getUtility(IIntIds)
>>> list(intids.ids)[0].object
<ContactsFolder at contacts>
>>> IKeyReference(portal.contacts)
<five.intid.keyreference.KeyReferenceToPersistent object at 0x7f9bd1f35410>
>>> list(intids.ids)[0]
<five.intid.keyreference.KeyReferenceToPersistent object at 0x7f9bd2456090>

You can see that the two objects are at different locations, and the intids.ids index is keyed off of objects. My only guess is that this is from moving databases? That explains why it's not indexed though. It also explains why plone.app.intids doesn't reregister it - the install method for that profile first checks to see if the object's path already matches the path of an existing key reference and indeed it does. One thing I am looking into is to do "intids.clear()" and then rebuild the whole thing by reinstalling plone.app.intids. That seems to work but probably needs more testing

At this point I have several questions:

  • Am I doing something weird/unsupported in the first place? How do people move a Plone site that has zc.relations from one environment to another?
  • Is clearing the intids utility and rebuilding likely to create any caveats I am missing?
  • Would it be worth changing the behavior of plone.app.intid's profile step to just always clear and rebuild instead of checking for existing indexes?
  • Is zope.intid just not something that will work if moving an entire site?
2

The problem lies in five.intid.keyreference.KeyReferenceToPersistent.__cmp__. It compares based on the (remembered) "oid" of an object. Of course, export+import changes an object's oid -- the KeyReferenceToPersistents stored in the zope.intid persistent utility become unreliable.

If you clear and rebuild the zope.intid persistent utility, all objects will get new "intid"s. This implies, that all references to those objects based on "intid"s become invalid and must be rebuild as well. This, likely, will affect the ZCatalogs at least.

1 Like
3

Thanks for the tip, I investigated that and thought I could perhaps just update the oid of the keyreferences with something like this

def repair_intids():
    intids = getUtility(IIntIds)
    for key_ref in intids.ids:
        if key_ref.object and key_ref.oid and key_ref.object._p_oid != key_ref.oid:
            key_ref.oid = key_ref.object._p_oid

That actually does not work, although I would have thought it did. Also key_ref.dbname is the same on both.

Perhaps this is moot because my previous suggestion of doing intids.ids.clear() and rerunning the plone.app.intids does appear to preserve my relations. If I check intids.ids afterwards, it has correctly put my z3c.relationfield.relation.RelationValue objects in there, and both my existing relations and new relations appear to work as expected.

4

@Esoth That's interesting. If clear and rebuild for intids after import should generate new intid to each object, how do the old relations still work?