Eea.facetednavigation and filtering

[espenmn (Espen)]

I have eea.facetednavigation running on Plone 5.

I can filter on indexes, unless they have 'special characters'.
If they have, no items are found.

How can I troubleshoot this (can it be an index problem, or a faceted navigation problem)?

[espenmn (Espen)]

Looks like it could be a unicode problem... any suggestions?

Traceback (most recent call last):
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/browser/app/query.py", line 173, in query
    brains = catalog(self.context, **query)
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/search/catalog.py", line 110, in __call__
    return search(**newquery)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.CMFPlone-5.0.4-py2.7.egg/Products/CMFPlone/CatalogTool.py", line 394, in searchResults
    return ZCatalog.searchResults(self, REQUEST, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
    return self._catalog.searchResults(REQUEST, used, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 1072, in searchResults
    return self.search(args, sort_indexes, reverse, sort_limit, _merge)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 549, in search
    r = _apply_index(query, rs)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py", line 426, in _apply_index
    s = index.get(k, None)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
2016-11-03 17:03:53 ERROR eea.facetednavigation.browser.app.query 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/browser/app/query.py", line 173, in query
    brains = catalog(self.context, **query)
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/search/catalog.py", line 110, in __call__
    return search(**newquery)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.CMFPlone-5.0.4-py2.7.egg/Products/CMFPlone/CatalogTool.py", line 394, in searchResults
    return ZCatalog.searchResults(self, REQUEST, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
    return self._catalog.searchResults(REQUEST, used, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 1072, in searchResults
    return self.search(args, sort_indexes, reverse, sort_limit, _merge)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 549, in search
    r = _apply_index(query, rs)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py", line 426, in _apply_index
    s = index.get(k, None)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
2016-11-03 17:03:54 ERROR eea.facetednavigation.browser.app.query 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/browser/app/query.py", line 173, in query
    brains = catalog(self.context, **query)
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/search/catalog.py", line 110, in __call__
    return search(**newquery)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.CMFPlone-5.0.4-py2.7.egg/Products/CMFPlone/CatalogTool.py", line 394, in searchResults
    return ZCatalog.searchResults(self, REQUEST, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
    return self._catalog.searchResults(REQUEST, used, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 1072, in searchResults
    return self.search(args, sort_indexes, reverse, sort_limit, _merge)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 549, in search
    r = _apply_index(query, rs)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py", line 426, in _apply_index
    s = index.get(k, None)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
2016-11-03 17:03:54 ERROR eea.facetednavigation.browser.app.query 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/browser/app/query.py", line 173, in query
    brains = catalog(self.context, **query)
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/search/catalog.py", line 110, in __call__
    return search(**newquery)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.CMFPlone-5.0.4-py2.7.egg/Products/CMFPlone/CatalogTool.py", line 394, in searchResults
    return ZCatalog.searchResults(self, REQUEST, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
    return self._catalog.searchResults(REQUEST, used, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 1072, in searchResults
    return self.search(args, sort_indexes, reverse, sort_limit, _merge)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 549, in search
    r = _apply_index(query, rs)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py", line 426, in _apply_index
    s = index.get(k, None)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
2016-11-03 17:03:54 ERROR eea.facetednavigation.browser.app.query 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/browser/app/query.py", line 173, in query
    brains = catalog(self.context, **query)
  File "/home/espen/instance8095/Plone506/zinstance/src/eea.facetednavigation/eea/facetednavigation/search/catalog.py", line 110, in __call__
    return search(**newquery)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.CMFPlone-5.0.4-py2.7.egg/Products/CMFPlone/CatalogTool.py", line 394, in searchResults
    return ZCatalog.searchResults(self, REQUEST, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
    return self._catalog.searchResults(REQUEST, used, **kw)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 1072, in searchResults
    return self.search(args, sort_indexes, reverse, sort_limit, _merge)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 549, in search
    r = _apply_index(query, rs)
  File "/home/espen/instance8095/Plone506/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py", line 426, in _apply_index
    s = index.get(k, None)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

[dieter (Dieter Maurer)]

In a standard setup, Python is very picky when an encoded string and unicode comes together. In such a case, Python uses the "default encoding" (which is "ascii", by default) to convert the encoded string to unicode -- and often fails (as in your case).

To avoid problems of this kind, I have a "sitecustomize.py" in my Python installations which sets the "default encoding" to "utf-8" (i.e. the encoding typically used for encoded strings in a Plone context). This way, "UnicodeError"s occur only for encoded strings using an encoding different from "utf-8" (and "ascii").

The alternative is that you ensure that either everything is unicode or everything is an encoded string using the same encoding or that there are not "special character"s (i.e. the "ascii" encoding can handle the transformation to unicode).

[espenmn (Espen)]

@dieter

To avoid problems of this kind, I have a "sitecustomize.py" in my Python installations which sets the "default encoding" to "utf-8" (i.e. the encoding typically used for encoded strings in a Plone context). This way, "UnicodeError"s occur only for encoded strings using an encoding different from "utf-8" (and "ascii").

does this mean that I can also set the default encoding in the facedenavigation py file, for example
eea.facetednavigation.browser.app.query
(or eea.facetednavigation.browser.app.search).

If this is a bug in faceted navigation (as it sounds like(?)), it is probably better to fix it there ?
... or might this break something else?

[dieter (Dieter Maurer)]

@espenmn

does this mean that I can also set the default encoding in the facedenavigation py file, for exampleeea.facetednavigation.browser.app.query(or eea.facetednavigation.browser.app.search).

No. Changing the default encoding is dangerous (it affects the hashing of dict keys; as a consequence, some dict keys may no longer be found after the change). Therefore, Python deletes "setdefaultencoding" from "sys" after startup. In principle, you could save "setdefaultencoding" during startup (e.g. in "sitecustomize") and use it later; however, I strongly discourage such a use due to the potential nasty side effects ("dict"s may stop working).

@espenmn

If this is a bug in faceted navigation (as it sounds like(?)), it is probably better to fix it there ?... or might this break something else?

Likely. "faceted navigation" should conform with respect to "unicode/str" to its Plone environment. Apparently, this is not the case.

"Archetypes" used to store text values (internally) in unicode but presented them towards Plone as encoded "str" - which was suboptimal, from my point of view. The new "dexterity" might use unicode both for storage as well as at its interface. In this case, the indexes may now (Plone 5) contain unicode values whereas they formerly contained "str" values. "ea.facetednavigation" might not yet be adapted.

[espenmn (Espen)]

This seems to work, but I am not sure if it is a 'good way to fix it'.

Fix for unicode index