Complex form widget with DataGridField and relations

I thought it might be of interest to share my experience with a complex form widget I needed. I do not yet have a full solution - I feel like I'm 99% there, but that last bit may end up requiring a major redesign. The use case is that we have some type of Person content type that can belong to one or more OrganizationalUnit types. For each of those org units, they can have one or more titles. A DataGridField (collective.z3cform.datagridfield) seemed liked the natural choice.

The widget can actually do the majority of the work already, with some important caveats I will get to later.

class IOrganizationalUnit(model.Schema):
    directives.widget(ou=SelectFieldWidget)
    ou = RelationChoice(
        title='Organizational Unit',
        vocabulary='foo.organizational_units',
        required=False
    )
    directives.widget(title=SelectFieldWidget)
    title = schema.List(
        title='Title',
        value_type=schema.Choice(vocabulary='foo.contact_jobs_source'),
        required=False
    )

and

directives.widget(org_units=DataGridFieldFactory)
org_units = OrgUnitList(
    title="Organizational Units" ,
    value_type=DictRow(title="Organizational Unit" , schema=IOrganizationalUnit),
    required=**False** ,
)

The OrgUnitList is just a subclass of RelationList so that it could have its own data manager class. This is needed so that the widget will render the relations as relations, instead of just the objects themselves.

class IOrgUnitList(IRelationList):
    """ """


@implementer(IOrgUnitList)
class OrgUnitList(RelationList):
    """ """


@adapter(Interface, IOrgUnitList)
class OrgUnitDataManager(AttributeField):
    """A data manager which sets a list of relations"""

    def get(self):
        """Gets the target"""
        rel_list = []

        # Calling query() here will lead to infinite recursion!
        try:
            rel_list = super().get()

        except AttributeError:
            rel_list = None

        if not rel_list:
            return []

        resolved_list = []
        for rel in rel_list:
            resolved_dict = {}
            for key, value in rel.items():
                if isinstance(value, RelationValue):
                    if value.isBroken():
                        continue
                    resolved_dict[key] = value.to_object
                else:
                    resolved_dict[key] = value
            resolved_list.append(resolved_dict)
        return resolved_list

    def set(self, value):
        """Sets the relationship target"""
        value = value or []
        new_relationships = []
        intids = getUtility(IIntIds)
        for item in value:
            new_item = {}
            for item_key, item_value in item.items():
                if isinstance(item_value, DexterityContent):
                    to_id = intids.getId(item_value)
                    relation = RelationValue(to_id)
                    new_item[item_key] = relation
                else:
                    new_item[item_key] = item_value
            new_relationships.append(new_item)
        super().set(new_relationships)

This mostly works with one huge caveat that might not be immediately obvious - these relations are not going to be indexed in the zc.relation catalog. The reason why requires a look at z3c.relationfield and its event.py. By virtue of being a Dexterity content item that provides IHasRelations, these events will scan the schema (I think plone.dexterity also extends this to look at behaviors) for any Relation or RelationList fields and extract that content and create relations. This will handle adding new relations to the catalog, updating, breaking, etc. The problem is that it really only understands those two data structures, but what we're sending it is something that looks like

{'ou': <relation_value_class>, 'title': ['list', 'of', 'titles']}

I had hoped to use a custom event listener for this content type that looks like this

from zope import component

from z3c.relationfield.event import _setRelation, _potential_relations
from z3c.relationfield.interfaces import IHasOutgoingRelations
from z3c.relationfield.interfaces import IRelationValue
from zc.relation.interfaces import ICatalog
from zope.intid.interfaces import IIntIds


def addRelations(obj, event):
    """ see z3c.relationfield.event
    """
    for name, relation in _relations(obj):
        _setRelation(obj, name, relation)


def addRelationsEventOnly(obj, event):
    """ see z3c.relationfield.event
    """
    if not IHasOutgoingRelations.providedBy(obj):
        return
    addRelations(obj, event)


def removeRelations(obj, event):
    """ This is the same as z3c.relationfield except we use our _relations
    """
    catalog = component.queryUtility(ICatalog)
    if catalog is None:
        return

    for name, relation in _relations(obj):
        if relation is not None:
            try:
                catalog.unindex(relation)
            except KeyError:
                # The relation value has already been unindexed.
                pass


def updateRelations(obj, event):
    """ This is the same as z3c.relationfield except we use our addRelations
    """
    catalog = component.queryUtility(ICatalog)
    intids = component.queryUtility(IIntIds)

    if catalog is None or intids is None:
        return

    # check that the object has an intid, otherwise there's nothing to be done
    try:
        obj_id = intids.getId(obj)
    except KeyError:
        # The object has not been added to the ZODB yet
        return

    # remove previous relations coming from id (now have been overwritten)
    # have to activate query here with list() before unindexing them so we don't
    # get errors involving buckets changing size
    rels = list(catalog.findRelations({'from_id': obj_id}))
    for rel in rels:
        if hasattr(obj, rel.from_attribute):
            catalog.unindex(rel)

    # add new relations
    addRelations(obj, event)


def _relations(obj):
    for name, index, relation in _ou_potential_relations(obj):
        if IRelationValue.providedBy(relation):
            yield name, relation


def _ou_potential_relations(obj):
    # we need to also get the non ou items since the update resets all
    for item in _potential_relations(obj):
        yield item
    counter = 0
    for ou in obj.org_units:
        if ou['ou']:
            yield 'org_units', counter, ou['ou']
            counter += 1

(and accompanying zcml). This "works" but we have a race condition. The original event handlers in z3c.relationfield are still going to fire too, and I believe the order in which these events fire is non-deterministic. There's no way to make the z3c.relationfield event handlers happen first, or to suppress them that I know of. So if it happens after my custom event handler, it will simply wipe out my changes and we still have no index in the zc relation catalog.

At this point I am stuck and would welcome any suggestions. One possible workaround that requires a very different approach would be to have a hidden field that is just a regular RelationList and this would be the only place that we store relations. The DataGridField would simply store an intid or whatever in lieu of what used to be a relation. To make this work, we would have to do some work both on form load and on form submit. When the form is loaded we would need to check those intids to make sure they reference objects that still exist - or maybe this can be done with an event listener. When the form is saved we would need to update our hidden field based on the options selected in our grid field. It's a bit of a mess.

I believe to remember that the documentation (--> zope.interface) specifies an activation order for subscribers: when I read the documentation, the text got it wrong but the corresponding example has been correct. Thus, you might be able to control the activation order.

However, it may be more straight forward to z3c.unconfigure the original subscriber and register your own one in place - integrating the original handler and your own in a proper way.

I don't see anything in the zope.interface about that, but z3c.unconfigure sounds promising, thanks!

Sorry, maybe I'm totally wrong but what about registering your own indexers in the relation catalog? So you can index correctly your Objects and then call the standard indexer for others. I can't find any example using an indexer for the relation catalog, all refers to the main catalog. But here:

"Customising indexers based on the catalog type"

"It is possible to provide a custom indexer for a different type of catalog."

Also, did you check:

I am familiar with custom indexers, but I do not follow how it would be appropriate here.