Catalog result count by index value (portal_type)

Hi,

We've got a use case where we would like to show the amount of search results per index value (portal_type in this case), ie

result_count = {
    'File': 4,
    'Images': 6,
    'Droids': 0,
}

Do we happen to have an existing (performant) solution to this, or do I just need to loop over the result set and add a counter?

1 Like

You should fast (enough) iterating over all brains and building a mapping portal_type -> counter.
This should be fast even for some thousands of brains. Eventually itertools.groupby() might be helpful here.

-aj

catalog.Indexes['portal_type'].uniqueValues(withLengths=True)

This is a raw count from inspecting the internals of the index. As such, it requires loading the entire index from the ZODB, and won't work if you also need to apply other indexes to limit the results. But it should be faster than doing a query and looping over the results.

As we do depend on other indexes, I ended up using something like

types = (x.portal_type for x in brains)
counter = collections.Counter(types)

which gives me

Counter({'Folder': 2, 'Document': 1, 'Image': 1})
2 Likes

The catalog search result is in fact a LazyMap, i.e. a sequence (of catalog record ids) and a function. On access to a sequence component, the function is transparently applied to the base component (which gives you the catalog proxy/brain). Using the above internal implementation details, you can access the raw search result (the set of catalog record ids) and "and" it with the sets for the various values in the index your are interested in to obtain the count values.

Whether this is faster than looping over the proxies depends on the size of the result set and the size of the index.

Slightly related:

does this, ( like here: http://www.ektedata.no/innhold/oppgavebank )

That would be something like this, me thinks.

      # This takes the catalog record ids from the result set (_seq) and
        # compares them with the records in the catalog (_index[portal_type])
        portal_catalog = getToolByName(portal, 'portal_catalog')
        _index = portal_catalog.Indexes['portal_type']._index
        # We get a list of tuples when searching.
        _seq = set(
            x[1] if isinstance(x, (list, tuple)) else x
            for x in self.lazy_resultset._seq
        )
        counter = {}
        for portal_type in _index:
            index_values = set(_index[portal_type])
            counter[portal_type] = len(_seq & index_values)

I think this should perform better for large result sets, though I haven't benchmarked this :smiley:

@jaroel There is a builtin function which gives you exactly this efficiently.

Almost completly undocumented but we make use of it in production. We do reports by combining different facets into a combined field and then walking over the histogram of those unique values to give grouping totals.

@djay
Did you see I need to check this against the result set?
As far as I can see, this doesn't do that?

Sorry I missed that. If you want to count a filtered search result then that method doesn't work. What you would need to do is build that into the index you are counting. What I did was create a special groupby index with values like "Male|USA|White" etc and only had values on items I wanted to count. Then I iterate over the histogram rather than the over the whole DB to count things like number of Males or number of white males. So if you have much less combined groupings than records this works well. The tradeoff is if you need to add in another facet then is a huge reindex.
Not sure if this helps your usecase though.