Error when querying SearchableText with AdvancedQuery on Plone5.2

Plone Support Add-on Development
,
1

installed advancedquery by adding dm.plone.advancedquery as dependency and including it in my package's configure.zcml

on a fresh plone site i can query the portal_type index:

>>> from Products.AdvancedQuery import Eq, In
>>> from plone import api
>>> cat = api.portal.get_tool('portal_catalog')
>>> cat.evalAdvancedQuery(Eq("portal_type", "Document"))
[<Products.ZCatalog.Catalog.Catalog.useBrains.<locals>.mybrains object at 0x7f9ce5010540>, ...]

but when i query for SerachableText i get the following error.

>>> cat.evalAdvancedQuery(Eq("SearchableText", "Document"))
*** TypeError: set operation: invalid argument, cannot iterate
Traceback (most recent call last):
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/__init__.py", line 53, in _evalAdvancedQuery
    return _eval(self, query, sortSpecs, withSortValues, restricted=True, **kw)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/__init__.py", line 43, in eval
    rs = _eval(query, catalog, restricted, **kw)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/__init__.py", line 24, in _eval
    return getSubscriptionAdapter(cat, IQueryContext).eval(query, restricted, **kw)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/adapter/cmfcore.py", line 21, in eval
    return super(QueryContext, self).eval(*args, **kw)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/context.py", line 53, in eval
    r = c_context.transform(t)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/transform.py", line 235, in transform
    return self._push(env_params)._transform(t)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/transform.py", line 250, in _transform
    return adapter.transform(t, self)
  File "/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/adapter/tree/evaluator/set_.py", line 49, in transform
    focus = intersection(focus, s)

any hints/tips welcome (@dieter :wink:

2

Something may have changed recently in the BTrees set operation implementation. In will look into this in the next few days.

Please try to run the AdvancedQuery test suite and report whether it is able to reproduce the problem in your context. The suite assumes zope.testrunner (or variant thereof) as runner.

3

thanks for your fast reply @dieter

the test suite passes (6 tests are skipped) - see below.

if there is anything else i can help with just let me know

$ bin/test -s Products.AdvancedQuery
/home/frisi/.buildout/eggs/Products.AdvancedQuery-4.1-py3.8.egg/Products/AdvancedQuery/eval/__init__.py:14: DeprecationWarning: LazyMap is deprecated. Please import from ZTUtils.Lazy.
  from Products.ZCatalog.Lazy import LazyCat, LazyMap
Running Products.AdvancedQuery.tests.layer.AqzcmlLayer tests:
  Set up Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.140 seconds.
  Running:
                  
  Ran 96 tests with 0 failures, 0 errors, 6 skipped in 0.024 seconds.
Running Products.AdvancedQuery.tests.TestBase.Layer tests:
  Set up Testing.ZopeTestCase.layer.ZopeLite in 0.015 seconds.
  Set up Products.AdvancedQuery.tests.TestBase.Layer in 0.000 seconds.
  Running:
                  
  Ran 30 tests with 0 failures, 0 errors, 0 skipped in 0.093 seconds.
Tearing down left over layers:
  Tear down Products.AdvancedQuery.tests.TestBase.Layer in 0.000 seconds.
  Tear down Testing.ZopeTestCase.layer.ZopeLite in 0.000 seconds.
  Tear down Products.AdvancedQuery.tests.layer.AqzcmlLayer in 0.000 seconds.
Total: 126 tests, 0 failures, 0 errors, 6 skipped in 0.373 seconds.
4

I tried to reproduce the problem in my Plone 5.2.1 installation and failed:

>>> p = app.Plone
>>> p
<PloneSite at /Plone>
>>> from Products.AdvancedQuery import Eq
>>> from zope.component.hooks import setSite
>>> setSite(p)
>>> c = p.portal_catalog
>>> c.evalAdvancedQuery(Eq("SearchableText", "Document"))
[]
>>> c.evalAdvancedQuery(Eq("portal_type", "Document"))
[<Products.ZCatalog.Catalog.Catalog.useBrains.<locals>.mybrains object at 0x7f1ad4c80af8>]
>>> from Products import CMFPlone
>>> CMFPlone.__file__
'/home/dieter/plone/buildout-cache/eggs/Products.CMFPlone-5.2.1-py3.6.egg/Products/CMFPlone/__init__.py'
>>> dp = c.evalAdvancedQuery(Eq("portal_type", "Document"))[0]
>>> d=dp.getObject()
>>> d
<Document at /Plone/front-page>
>>> d.SearchableText()
'Willkommen bei Plone Herzlichen Glückwunsch! Sie haben das professionelle Open-Source Content-Management-System Plone erfolgreich installiert.'

Can you use a debugger to find out what focus and s are in the error case?

5

apologies: my example code to reproduce was not run on a completely fresh site

on a german plonesite you should be able to reproduce with a string that results in a match:

(using Plone5.2.4 running on python 3.8)

>>> cat.evalAdvancedQuery(Eq("SearchableText", "Document"))
[]
>>> cat.evalAdvancedQuery(Eq("SearchableText", "Willkommen"))
*** TypeError: set operation: invalid argument, cannot iterate
...

if still not reproduceable for you here the debugger output (hope that helps).
seems focus should be a BTree but is a dict_keys

>>> focus
dict_keys([-613569189])
>>> focus.__class__
<class 'dict_keys'>
>>> focus.__class__.__module__
'builtins'

>>> s
<BTrees.IIBTree.IITreeSet object at 0x7fa88520d0c0 oid 0x5ede in <Connection at 7fa889042a60>>
>>> [k for k in s.keys()]
[-409543018, -409543017]

relevant traceback

  Module Products.AdvancedQuery, line 53, in _evalAdvancedQuery
  Module Products.AdvancedQuery.eval, line 43, in eval
  Module Products.AdvancedQuery.eval, line 24, in _eval
  Module Products.AdvancedQuery.eval.adapter.cmfcore, line 21, in eval
  Module Products.AdvancedQuery.eval.context, line 53, in eval
  Module Products.AdvancedQuery.eval.transform, line 235, in transform
  Module Products.AdvancedQuery.eval.transform, line 250, in _transform
  Module Products.AdvancedQuery.eval.adapter.tree.evaluator.set_, line 49, in transform
TypeError: set operation: invalid argument, cannot iterate
6

It is strange that focus is of type dict_keys (it represents the result of previous searches and therefore should be a kind of IISet). Nevertheless, with BTrees==4.9.2, the intersection seems to work. Thus, a workaround might be to upgrade BTrees.

I will try again to reproduce the problem and (if successful) check where the strange type comes from.

7

thanks for your update @dieter.

i can confirm that a version pin for BTrees=4.9.2 resolves the problem :slight_smile:

however i'm not 100% sure it won't cause any regressions/troubles with the rest of the zope 4.5.5 versions.

8

Upgrading BTrees should not cause problems.

The problem you have reported in this discussion is understood: Products.ZCTextIndex.BaseIndex switches the type it uses as values for its _wordinfo attribute. If a word occurs in few documents, the associated information uses a dict, otherwise a IIBTree (not sure when this type switching has been introduced). Currently, AdvancedQuery's ZCTextIndex adapter (--> Products.AdvancedQuery.eval.adapter.query.converter.zctextindex.BaseIndexLookupTreeIndex) does not yet expect the type switch. A modern BTrees version can handle the unexpected type (however, problems can occur at other places).

I will soon publish a new AdvancedQuery version which copes with the type switch.

1 Like