Vocabulary items pulled from portal_catalog (encode attribute error)

I used to be able to create dynamic vocabularies by pulling items from portal_catalog without any issues in Plone 4 but for the life of me, I cannot get it to work in Plone 5. I want a search term in collections to include NC Counties which has 100 items. However, when I try to list the values of this field as a drop down menu by using vocabularies, the entire search term section will disappear. Without the vocabulary section, hence the drop down menu, the search term section reappears with the NC County as a search term. One just has to type in the county manually instead of being able to choose from a list. This is what I have done for the Dexterity field called nc_county:

Plone 5.1.0.1 (5110)
CMF 2.2.12
Zope 2.13.26
Python 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
PIL 3.4.2 (Pillow)

ncdhhscontenttype.ncssinfo/src/ncdhhscontenttype/ncssinfo/vocabulary.py:

from Products.CMFCore.utils import getToolByName
from zope.interface import implementer
from zope.schema.interfaces import IVocabularyFactory
from zope.schema.vocabulary import SimpleTerm
from zope.schema.vocabulary import SimpleVocabulary

def vocab_from_catalog_index(index_name):
    @implementer(IVocabularyFactory)
    class VocabFactory(object):
        def __call__(self, context):
            catalog = getToolByName(context, 'portal_catalog')
            terms = []
            for value in catalog.Indexes[index_name].uniqueValues():
                if value is None:
                    continue
                encoded_value = value.encode('utf8')
                terms.append(SimpleTerm(
                    value=encoded_value,
                    token=value.encode('raw_unicode_escape'),
                    title=value,
                    ))
            return SimpleVocabulary(terms)
    return VocabFactory
 
nc_county_vocab = vocab_from_catalog_index('nc_county')

ncdhhscontenttype.ncssinfo/src/ncdhhscontenttype/ncssinfo/configure.zcmlconfigure.zcml:

<utility name="ncdhhscontenttype.ncssinfo.vocabulary.nc_county"
    factory="ncdhhscontenttype.ncssinfo.vocabulary.nc_county_vocab" />

ncdhhscontenttype.ncssinfo/src/ncdhhscontenttype/ncssinfo/profiles/default/registry.xml

 <records interface="plone.app.querystring.interfaces.IQueryField"
           prefix="plone.app.querystring.field.nc_county">
    <value key="title">NC County</value>
    <value key="description">A custom NC County index</value>
    <value key="enabled">True</value>
    <value key="sortable">False</value>
    <value key="operations">
        <element>plone.app.querystring.operation.string.is</element>
    </value>
    <value key="vocabulary">ncdhhscontenttype.ncssinfo.vocabulary.nc_county</value>
    <value key="group">Metadata</value>
  </records>

The error seems to be coming from my vocabulary.py file:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module plone.app.content.browser.query, line 13, in __call__
  Module plone.app.querystring.registryreader, line 136, in __call__
  Module plone.app.querystring.registryreader, line 87, in getVocabularyValues
  Module ncdhhscontenttype.ncssinfo.vocabulary, line 17, in __call__
AttributeError: 'set' object has no attribute 'encode'

Line 17: encoded_value = value.encode('utf8')

Any advice or help would be most appreciated. Thank you so much.

Obviously the index referenced through the variable index_name has values indexed as set and not a string that you code expects it. So figure out why this particular index indexes (suddenly?) sets of something instead of pure strings. And take the debugger to figure out what the offending value - the set in this case - contains and why. pdb is your friend.

-aj

I am already in a debug mode and it does not really say what is the matter. Anyway, I when replaced utf-8 with utf-32 the attribute error disappeared. However, another error appeared:

2018-08-15 13:15:57 INFO plone.app.querystring ncdhhscontenttype.ncssinfo.vocabulary.nc_county is missing, ignored.

The search term section appears in Collections and the NC County is showing. However, the NC County items are still now showing.

This is nonsense.

Your error message is

AttributeError: 'set' object has no attribute 'encode'

So you are having a Python set here and not a string. A set has not encode() method. encode() is a string specific method. Any why utf-32? Where do you have to deal with utf-32 encoded content or data? UTF 32 is completely uncommon.

Wild guess, but could it be that you are using the same code for different indexes (some that are sets, and some that are not?), so when you changed the utf-8 you got an error on some code that 'happened before' ? (or that you changed the index type after adding content ?)

Espen, you are brilliant and absolutely right! Yes, it does not like a "set" field that allows people to choose more than one answer. However, it works perfectly when I am dealing with a "choice" field that allows people to choose only one answer. The other thing I had to change was in the operations tag:

<value key="operations">
    <element>plone.app.querystring.operation.selection.any</element>
</value>

So how should I deal with "set" fields? Is it even possible? Any help or advise would be most appreciated.

Check the type of the value retrieved from the index. If it is a Python set the add each value of the set to the vocabulary by looping over the set, if it is a string then proceed as implwmnted at the moment.

Sometimes, it might be difficult to understand that you are dealing with a set or a list.

For example, if you search the catalog for a specific UID, you might expect the result to be 'one item'

But you have to do something like:

my_search = context.portal_catalog.unrestrictedSearchResults(UID=some_uid)
my_item = my_seach[0]

So, if you 'know' that the set should only return one value (like when you search for UID), you can do:

 my_value = value[0]

This is tinkered code and a tinkered explanation.

  1. why do you use unrestricedSearchResults()? This will bypass Zope/Plone security, possibly expose unrelated and unwanted information here.

  2. You assume that a search will return at least one catalog brain. This assumption is in general wrong. You have to check for the length of the result and set retrieve then the 1st item. Otherwise you may encounter an `IndexError``

  3. You assume that a set contains at least one item...this is also an improper assumption.

  4. Regarding the original question: if you field can store multiple values then they are stored as said. If each of the values is relevant for building a vocabulary then you need to iterate over all values of the set (as described earlier) and not only over the 1st item of the set. In addition you need to ensure that duplicate terms won't make it into the vocabularies because otherwise you will receive an error about duplicate vocubulary tokens.

Many Plone add-ons (including the ones of one particular persons) are full of such anti-patterns and basic beginner mistakes...don't repeat them, don't sell them as solution.

-aj

sorry, that was not on purpose, I just wanted to explain, so I copied the line from another post here…

No, I dont assume anything, I just mention something that might be difficult to understand if you have done it before and/or if you are not a programmer.

If you have a field where you can and must enter ONE value
and if you have a search that (can) only return ONE item

it might not be clear that you get a set or a list and you might think that if nothing was found it would return for example 'None'

Of course a lot of add-ons are full of mistakes because one person is trying to scare others from asking questions to get it right by telling them how stupid they are and are not interested in helping, just explaining how smart he is and always using wording that he is sure the person that asked the question will not understand.

That's exactly how most of this particular person's code looks like...exactly the code quality why some customer plan to throw Plone out of the windows...just for the record..but yes, it's all open-source and everyone has the freedom to do or do not.

Others jump the ship because how they are treated when they ask questions

Are you questioning the fact the my analysis was wrong and any of the given hints are wrong.

And yes, badly written software is one of the reasons why customers are sick of Plone, sick of spending money for fixing tinkered add-ons. Of course everything is open-source but majority of existing add-ons and Plone itself are miles away from being an "enterprise" CMS. It once was...this was with Plone 3 and 4.

guys, I'm probably not the best person to say this, but I think your discussion in far from being productive.

@espenmn you become a programmer since the very moment you start coding and giving others advice on it.

there are many mistakes in the code you're proposing, but nothing that can't be enhanced. first you have to understand some basic concepts: catalog searches always return an iterable of results, always. some specific searches, like the one you're proposing using the UID of an object, can return an iterable of zero (if no item was found in the catalog with that UID), one (if there's an item with that UID in the catalog), or even more than one (if you have catalog issues or some objects are inheriting their UID from others). the last possibility is an error that most be fixed.

in Python, errors should never pass silently; so, you want to rise an exception if that's the case. you can use a simple assertion:

assert len(results) in (0, 1)

that could be fine, but if you are running with optimized code, assertions are ignored; it's up to you to decide how to deal with that.

now, sets and list are different kind of iterators: a set differs from a list in the sense that it can't contain duplicated elements. there are many other types of iterators and each one of them is useful in different circumstances. some of them are not mutable, some consume less memory. go to the Python documentation and learn about it.

resuming, you have to be careful and study to become a better programmer. I'll give you an advice: review other people's code; you'll learn a lot from it.

always have in mind that some of our best programmers are not graduated from computer science.

@zopyx relax, you don't have to be aggressive even if you have lost your patience; you better skip the thread: life goes on.

for me, Plone right now is in a transition; people is spending time fighting against time on the Python 3 migration and we have very limited resources. Plone 6 will probably be much better if we can get rid of the broken parts.

and don't assume proprietary software is better just because you can't see the code; open source can be hard, but people will always pay more using proprietary software, and I'm not talking only about money.

1 Like

Thank you all for all your help. It is very much appreciated.