Catalog Multiplex Implementation for Dexterity Contents?

Hi all,

Is there a way to register Dexterity Contents with another catalog than portal_catalog?

In Archetypes based contents there was the archetype_tool, where it was possible to map portal types to catalogs. The logic behind was handled by Products.Archetypes.CatalogMultiplex.

At the moment I have this requirement to register Dexterity contents with a specific catalog and I'm thinking about creating a mix-in class for these contents similar to CatalogMultiplex, because it does not look like that it can be achieved with a behaviour.

I also tried to override this code in my content class, coming from Products.CMFCore.CatalogAware:

    # The following method can be overridden using inheritance so that it's
    # possible to specify another catalog tool for a given content type
    def _getCatalogTool(self):
        return getToolByName(self, 'my_catalog', None)

Unfortunately, this raises this error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module plone.z3cform.layout, line 66, in __call__
  Module plone.z3cform.layout, line 50, in update
  Module plone.dexterity.browser.add, line 118, in update
  Module plone.z3cform.fieldsets.extensible, line 59, in update
  Module plone.z3cform.patch, line 30, in GroupForm_update
  Module z3c.form.group, line 145, in update
  Module plone.app.z3cform.csrf, line 21, in execute
  Module z3c.form.action, line 98, in execute
  Module z3c.form.button, line 315, in __call__
  Module z3c.form.button, line 170, in __call__
  Module plone.dexterity.browser.add, line 101, in handleAdd
  Module z3c.form.form, line 265, in createAndAdd
  Module plone.dexterity.browser.add, line 80, in add
AttributeError: 'NoneType' object has no attribute 'id'

Did anyone ever had a similar requirement or has an idea how to approach it?

Any input from you is highly appreciated.

Thanks, Ramon

That's the most recent implementation:

So, you can register your own catalog-utility and it will be picked up instead of the default.

Another - IMO better - way to hook in is to write a Products.CMFCore.interfaces.IIndexQueueProcessor implementation. This is used i.e. by collective.solr and some elasticsearch integrations like collective.elastic.plone.

But: I really can not remember well why it was good to have multiple catalogs. IIRC the idea was to have multiple mount points in the ZODB. But later on in production and site life-cycle it was a mess to work with.

What is your actual use case?

Hi Jens,

nice to meet you again and thanks for answering.

I'm coming from the Senaite LIMS project (formerly bika.lims), where multiple catalogs were primarily used to deal more performant with the huge amount of laboratory data stored in the system.

While I agree with you that this adds more complexity (and likely better ways exist than this approach), we have to somehow deal with it at the moment to focus on getting the system from Plone 4 to Plone 5 and migrating the existing contents to Dexterity.

Probably the best way then is to write an ICatalogTool utility and mix in a base class with the new implementation of _getCatalogTool into the dexterity contents. At least we could drop then later the mix-in class when we achieved the upgrade to Plone 5.

The downside of an implementation of a Products.CMFCore.interfaces.IIndexQueueProcessor would probably be, that the catalog indexing would be additional. So that the data is indexed in portal_catalog and the specialised custom catalog, which would decrease performance again I guess.

However, not sure if I'm missing something here at the moment...

Hi Jens

Thanks again for pointing out the approach with the IIndexQueueProcessor.
I was able to implement it as a behavior for Plone 4 with collective.indexing as follows:

class CatalogMultiplexProcessor(object):
    """A catalog multiplex processor
    """
    if USE_COLLECTIVE_INDEXING:
        implements(IIndexQueueProcessor)

    def get_catalogs_for(self, obj):
        catalogs = getattr(obj, "_catalogs", [])
        for rc in REQUIRED_CATALOGS:
            if rc in catalogs:
                continue
            catalogs.append(rc)
        return map(api.get_tool, catalogs)

    def supports_multi_catalogs(self, obj):
        """Check if the Multi Catalog Behavior is enabled
        """
        if IMultiCatalogBehavior(obj, None) is None:
            return False
        return True

    def index(self, obj, attributes=None):
        if attributes is None:
            attributes = []

        if not self.supports_multi_catalogs(obj):
            return

        catalogs = self.get_catalogs_for(obj)
        url = api.get_path(obj)

        for catalog in catalogs:
            logger.info(
                "CatalogMultiplexProcessor::indexObject:catalog={} url={}"
                .format(catalog.id, url))
            # We want the intersection of the catalogs idxs
            # and the incoming list.
            indexes = set(catalog.indexes()).intersection(attributes)
            catalog.catalog_object(obj, url, idxs=list(indexes))

    def reindex(self, obj, attributes=None):
        self.index(obj, attributes)

    def unindex(self, obj):
        if aq_base(obj).__class__.__name__ == "PathWrapper":
            # Could be a PathWrapper object from collective.indexing.
            obj = obj.context

        if IMultiCatalogBehavior(obj, None) is None:
            return

        catalogs = self.get_catalogs_for(obj)
        url = api.get_path(obj)

        for catalog in catalogs:
            if catalog._catalog.uids.get(url, None) is not None:
                logger.info(
                    "CatalogMultiplexProcessor::unindex:catalog={} url={}"
                    .format(catalog.id, url))
                catalog.uncatalog_object(url)

    def begin(self):
        pass

    def commit(self):
        pass

    def abort(self):
        pass

The Dexterity contents can then specify additional catalogs like this:

class MyDexterityContent(Item):
    """A dexterity content providing the `IMultiCatalogBehavior` behavior
    """
    _catalogs = ["my_special_catalog"]

However, all contents are still indexed in portal_catalog as well, but that's ok for the moment.

Any comments on this approach are appreciated, thanks.
Ramon

Hi Ramon,

I was investigating about the topic of multiplexing catalogs as well and found your post. I was wondering if you are happy with this approach or can probably report any side-effects?

best, Paul

Hi Paul,

we recently switched to Plone 5.2.3 and switched from collective.indexing.interfaces.IIndexQueueProcessor to Products.CMFCore.interfaces.IPortalCatalogQueueProcessor.

Besides that objects are always indexed in portal_catalog, it seems to work pretty well.

This is our implementation:

Dexterity objects that should be indexed in multiple catalogs need to have this behavior assigned:

  <!-- Dexterity behaviours for this type -->
  <property name="behaviors">
    ...
    <element value="bika.lims.interfaces.IMultiCatalogBehavior"/>
    ...
  </property>

Best regards
Ramon

1 Like