Products.AdvancedQuery using In() vs MatchGlob()

I have recently upgraded from Plone 5.0.6 up through 5.2.1. I used to rely on a customization I had built for better search ranking that ran on the Products.AdvancedQuery product. See: https://pypi.org/project/Products.AdvancedQuery/#description

I was required to move from AdvancedQuery 3.0.4 to 4.0. My customization attempts to rank as follows:

                r = query['SearchableText']
                lc = self.request.get('LANGUAGE', u'en')
                aq = catalog.makeAdvancedQuery(
                    {'Language': lc, 'SearchableText': r, 'portal_type': query['portal_type']})
                rs = RankByQueries_Sum((In('Title', r), 32), (In('Subject', r), 16), (In('Description', r), 8))
                results = catalog.evalAdvancedQuery(aq, (rs,))

In Plone 5.2.1 with Products.AdvancedQuery 4.0:
RankByQueries_Sum((In('Title', r), 32), (In('Subject', r), 16), (In('Description', r), 8)) no longer works. It gives ParseError: Query contains only common words: u'*'. The r part of the query is in the form 'mysearch*'.

After reading the docs available in the AdvancedQuery package, it appears that wildcards (*) are only supported by MatchGlob. When I swap out for MatchGlob, I get a ComponentLookupError.

  Module Products.AdvancedQuery.eval.adapter, line 69, in getMultiSubscriptionAdapter
ComponentLookupError: ((<Products.ZCTextIndex.ZCTextIndex.ZCTextIndex object at 0x7fbc915fb050 oid 0xde5 in <Connection at 7fbc98b72990>>, <Products.AdvancedQuery.AdvancedQuery.MatchGlob object at 0x7fbc86228e90[Title =~ '*peni*']>), <InterfaceClass Products.AdvancedQuery.eval.interfaces.IQueryConverter>)

It would appear that the ZCTextIndex type indexes do not have a registered adapter for IQueryConverter Is this expected, and what are my options here? Thank you in advance for any help or insight!

If you report errors, you should provide full error information (found e.g. via the error_log object), including the traceback.

Are you aware that in order to use Products.AdvancedQuery (>= 4.0) with Plone, you must also install (and ZCML register) dm.plone.advancedquery; without it, important registrations may be missing.

Again, full error information (including the traceback) would be helpful.

You may have hit a bug in Products.AdvancedQuery: a ZCTestIndex should support MatchGlob queries but maybe a corresponding adapter registration is missing. Come back with the tracebacks and I will hav a look.

You are tremendously helpful, Dieter. To answer your questions the best I can.

  1. Yes, I have added dm.plone.advancedquery as well. I have added the zcml = dm.plone.advancedquery registration to my buildout. I believe that is what you mention: (and ZCML register).

  2. The full stack trace for the ComponentLookupError when I use MatchGlob is as follows:

2020-02-04 13:21:52 ERROR Zope.SiteErrorLog 1580840512.820.233710540399 https://myplonesite.com/en/@@ajax-search
Traceback (innermost last):
  Module ZServer.ZPublisher.Publish, line 144, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZServer.ZPublisher.Publish, line 44, in call_object
  Module myproduct.filtersearch.browser.search, line 277, in __call__
  Module plone.batching.batch, line 161, in __getitem__
  Module ZTUtils.Lazy, line 201, in __getitem__
  Module Products.AdvancedQuery.sorting, line 37, in __getitem__
  Module Products.AdvancedQuery.sorting, line 60, in _sort
  Module Products.AdvancedQuery.ranking, line 25, in group
  Module Products.AdvancedQuery.ranking, line 114, in _group
  Module Products.AdvancedQuery.ranking, line 84, in generate
  Module Products.AdvancedQuery.eval, line 24, in _eval
  Module Products.AdvancedQuery.eval.adapter.cmfcore, line 21, in eval
  Module Products.AdvancedQuery.eval.context, line 43, in eval
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.query.converter, line 340, in transform
  Module Products.AdvancedQuery.eval.adapter.query.converter, line 340, in <genexpr>
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.query.converter, line 46, in transform
  Module Products.AdvancedQuery.eval.adapter, line 69, in getMultiSubscriptionAdapter
ComponentLookupError: ((<Products.ZCTextIndex.ZCTextIndex.ZCTextIndex object at 0x7faf8537bcf8 oid 0xde5 in <Connection at 7faf86776b10>>, <Products.AdvancedQuery.AdvancedQuery.MatchGlob object at 0x7faf75184810[Title =~ '*mysearch*']>), <InterfaceClass Products.AdvancedQuery.eval.interfaces.IQueryConverter>)
  1. The full stack trace for the ParseError when I use In:
2020-02-04 13:38:23 ERROR Zope.SiteErrorLog 1580841503.210.361581771782 https://myplonesite.com/en/@@ajax-search
Traceback (innermost last):
  Module ZServer.ZPublisher.Publish, line 144, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZServer.ZPublisher.Publish, line 44, in call_object
  Module myproduct.filtersearch.browser.search, line 277, in __call__
  Module plone.batching.batch, line 161, in __getitem__
  Module ZTUtils.Lazy, line 201, in __getitem__
  Module Products.AdvancedQuery.sorting, line 37, in __getitem__
  Module Products.AdvancedQuery.sorting, line 60, in _sort
  Module Products.AdvancedQuery.ranking, line 25, in group
  Module Products.AdvancedQuery.ranking, line 114, in _group
  Module Products.AdvancedQuery.ranking, line 84, in generate
  Module Products.AdvancedQuery.eval, line 24, in _eval
  Module Products.AdvancedQuery.eval.adapter.cmfcore, line 21, in eval
  Module Products.AdvancedQuery.eval.context, line 53, in eval
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.tree.evaluator.set_, line 37, in transform
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.tree.evaluator.set_, line 66, in transform
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.tree.evaluator.set_, line 22, in transform
  Module Products.AdvancedQuery.eval.tree, line 113, in as_set
  Module Products.ZCTextIndex.ZCTextIndex, line 203, in _apply_index
  Module Products.ZCTextIndex.ZCTextIndex, line 209, in query_index
  Module Products.ZCTextIndex.QueryParser, line 157, in parseQuery
ParseError: Query contains only common words: u'*'

Thank you for your help, and please let me know if I can provide anything else.

A first analysis for this case: In interpretes its second argument as a sequence and the query is equivalent to an Or over its conponents. In your case, the second argument has the form 'mysearch*'. As a consequence, one of the Or components is the search for * which results in the ParseError (because * is not indexed). It may (by accident) work, if you pass a one element tuple as second argument to In: the traceback seems to indicate that the special ZCTextIndex related optimizations are not in effect (this would explain the MatchGlob problem, too); as a consequence, ZCTextIndex related subqueries are evaluated by ZCTextIndex itself and this supports queries of the form 'myquery*'. If the AdvancedQuery optimizations were in effect, the In would not work (as the * would not be interpreted as you expect it).

I do not yet know why the ZCTextIndex related registrations (among them the optimizations) are not effective.

Can you run the Products.AdvancedQuery test suite in your environment? I use ztest (--> dm.zopepatches.ztest, an extension of zope.testrunner) but in principle, zope.testrunner shoud work as well.

Which version of Plone are you using? Do your text indexes use special configuration?

I am using Plone 5.2.1. There is nothing special to my knowledge with the ZCTextIndex. I believe they are out-of-the-box.

The zope.testrunner results do indicate that 6 tests were skipped for Products.AdvancedQuery.

 test_and (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)
 test_filter (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)
 test_lookup_index (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)
 test_not (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)
 test_or (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)
 test_set (Products.AdvancedQuery.eval.adapter.tree.evaluator.tests.TestISearch) (skipped: ISearch not installed)

Do you know if there is an easy way to see why?

Products.AdvancedQuery

/usr/local/bin/Plone/buildout-cache/eggs/Products.AdvancedQuery-4.0-py2.7.egg/Products/AdvancedQuery/eval/__init__.py:14: DeprecationWarning: LazyCat is deprecated. Please import from ZTUtils.Lazy.
  from Products.ZCatalog.Lazy import LazyCat, LazyMap
/usr/local/bin/Plone/buildout-cache/eggs/Products.AdvancedQuery-4.0-py2.7.egg/Products/AdvancedQuery/eval/__init__.py:14: DeprecationWarning: LazyMap is deprecated. Please import from ZTUtils.Lazy.
  from Products.ZCatalog.Lazy import LazyCat, LazyMap
Running Products.AdvancedQuery.tests.layer.AqzcmlLayer tests:
  Set up Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.216 seconds.
  Ran 88 tests with 0 failures, 0 errors and 6 skipped in 0.037 seconds.
Running Products.AdvancedQuery.tests.TestBase.Layer tests:
  Set up Testing.ZopeTestCase.layer.ZopeLite in 0.023 seconds.
  Set up Products.AdvancedQuery.tests.TestBase.Layer in 0.000 seconds.
  Ran 30 tests with 0 failures, 0 errors and 0 skipped in 0.129 seconds.
Tearing down left over layers:
  Tear down Products.AdvancedQuery.tests.TestBase.Layer in 0.000 seconds.
  Tear down Testing.ZopeTestCase.layer.ZopeLite in 0.000 seconds.
  Tear down Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.000 seconds.
Total: 118 tests, 0 failures, 0 errors and 6 skipped in 0.522 seconds.

dm.plone.advancedquery

/usr/local/bin/Plone/buildout-cache/eggs/Products.AdvancedQuery-4.0-py2.7.egg/Products/AdvancedQuery/eval/__init__.py:14: DeprecationWarning: LazyCat is deprecated. Please import from ZTUtils.Lazy.
  from Products.ZCatalog.Lazy import LazyCat, LazyMap
/usr/local/bin/Plone/buildout-cache/eggs/Products.AdvancedQuery-4.0-py2.7.egg/Products/AdvancedQuery/eval/__init__.py:14: DeprecationWarning: LazyMap is deprecated. Please import from ZTUtils.Lazy.
  from Products.ZCatalog.Lazy import LazyCat, LazyMap
Running dm.plone.advancedquery.tests.Layer tests:
  Set up Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.075 seconds.
  Set up dm.plone.advancedquery.tests.Layer in 0.001 seconds.
  Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds.
Tearing down left over layers:
  Tear down dm.plone.advancedquery.tests.Layer in 0.000 seconds.
  Tear down Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.000 seconds.

The reason is a bug/weakness in Products.AdvancedQuery ("AQ" in the following explanation). The AQ optimizations are implemented via adapters and based on knowledge how an index performs its lookups. Because a derived index can perform lookups differently from its base index, an optimization for a base index cannot always be used for a derived index. Therefore, AQ cannot use standard Zope adapters but uses so called "conditional adapters". A ZCTextIndex is actually not an index on its own but combines a lexicon (mapping terms to word ids) and a word index. AQ's adapters for ZCTextIndex are all registered under the condition that it can "handle" the associated word index. Whether AQ can "handle" a word index is again defined via conditional adapters: it knows how to handle the base word index and registers adapters for derived classes under the condition that they have the same index_doc method as the base word index. Plone uses ZCTextIndex with an Okapi word index and this has its own index_doc. As a consequence, in the standard Plone setup, AQ does not use extensions (such as MatchGlob support) or optimizations for ZCTextIndex but falls back to the ZCatalog interface.

Your options:

  • for the moment, you can use In(..., (r,)). This should work as ZCTextIndex supports * at its ZCatalog interface. Drawback: with a new AQ version, it may no longer work (when AQ has learned to handle an Okapi index)
  • register your own conditional adapters for the Okapi index. You find the definitions for the base index at the end of Products.AdvancedQuery.eval.adapter.query.converter.zctextindex. Appropriate definitions for the Okapi index look identical with all references to BaseIndex replaced by OkapiIndex. Register your own conditional adapters via ZCML similar to how those for the base index have been registered
  • wait for a new AQ release which knows how to handle an Okapi index (I do not yet know when I will make such a release).

Thank you, Dieter, for the detailed explanation. I attempted registering my own conditional adapters for Okapi, but I ran into errors downstream. Ultimately, I tried your first option and passed used In() with the second argument being converted to a single-element tuple. That did work as you mention.

I have just released version 4.1: it implements extensions/optimizations for ZCTextIndex even if this is using an Okapi (and not only a Cosine) base index.

That's fantastic! Thank you for taking swift action. Extremely helpful.

Plone Foundation Code of Conduct