IContextSourceBinder in DatagridField leads to InvalidVocabularyError in zope.schema._field

Hi all, first a bit of background.

I have the form below in Plone 5.0.10 which "almost works" - that is, only the first row of values in the nested DatagridField (article and quantity) are being saved. I found some hints that this is due to a buggy converter in the old version of collective.z3cform.datagridfield. In the Plone6 version this seems to have been fixed (thanks @petschki).

So, I decided to use Plone6 for my app, and stumbled on an InvalidVocabularyError in zope.schema.field for the same field... (Traceback below, edit: updated for Plone 6.0-latest).

The field works if I insert it in the main schema (bottom image)

I have read these two posts:

What am I doing wrong?

Plone 5.0.10

Plone 6

Field in the main schema works...
(Note: I am using a source binder ItalianCitiesSourceBinder that calls a simplified vocabulary for testing here)

Traceback:

2023-05-24 12:45:15,936 ERROR   [Zope.SiteErrorLog:17][waitress-0] InvalidVocabularyError: http://localhost:8888/Plone/api/edi-order
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 181, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 390, in publish_module
  Module ZPublisher.WSGIPublisher, line 285, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module Products.PDBDebugMode.wsgi_runcall, line 60, in pdb_runcall
  Module Shared.DC.Scripts.Bindings, line 333, in __call__
  Module Shared.DC.Scripts.Bindings, line 370, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 338, in _exec
  Module script, line 16, in edi-order
   - <PythonScript at /Plone/api/edi-order>
   - Line 16
  Module z3c.form.form, line 233, in __call__
  Module pnz.erpediem.client.browser.create_edi_order, line 345, in update
  Module plone.z3cform.fieldsets.extensible, line 62, in update
  Module plone.z3cform.patch, line 31, in GroupForm_update
  Module z3c.form.group, line 132, in update
  Module z3c.form.form, line 136, in updateWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.multi, line 63, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 509, in update
  Module Products.CMFPlone.patches.z3c_form, line 46, in _wrapped
  Module z3c.form.widget, line 132, in update
  Module z3c.form.widget, line 504, in value
  Module collective.z3cform.datagridfield.datagridfield, line 154, in updateWidgets
  Module collective.z3cform.datagridfield.datagridfield, line 126, in getWidget
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.object, line 297, in update
  Module collective.z3cform.datagridfield.datagridfield, line 267, in updateWidgets
  Module z3c.form.object, line 222, in updateWidgets
  Module z3c.form.object, line 216, in setupWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.multi, line 63, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 509, in update
  Module Products.CMFPlone.patches.z3c_form, line 46, in _wrapped
  Module z3c.form.widget, line 132, in update
  Module z3c.form.widget, line 504, in value
  Module collective.z3cform.datagridfield.datagridfield, line 148, in updateWidgets
  Module z3c.form.widget, line 445, in updateWidgets
  Module collective.z3cform.datagridfield.datagridfield, line 126, in getWidget
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.object, line 297, in update
  Module collective.z3cform.datagridfield.datagridfield, line 267, in updateWidgets
  Module z3c.form.object, line 222, in updateWidgets
  Module z3c.form.object, line 216, in setupWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.select, line 51, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 233, in update
  Module z3c.form.widget, line 225, in updateTerms
  Module zope.component._api, line 102, in getMultiAdapter
  Module zope.component._api, line 116, in queryMultiAdapter
  Module zope.interface.registry, line 364, in queryMultiAdapter
  Module zope.interface.adapter, line 844, in queryMultiAdapter
  Module z3c.form.term, line 108, in ChoiceTerms
  Module zope.schema._field, line 458, in bind
  Module zope.schema._field, line 448, in _resolve_vocabulary
zope.schema._field.InvalidVocabularyError: Invalid vocabulary <pnz.erpediem.core.vocabularies.edifact.sources.ItalianCitiesSourceBinder object at 0x7f59c87de040>
[55] > /usr/local/Plone6/lib/python3.8/site-packages/zope/schema/_field.py(448)_resolve_vocabulary()
-> raise InvalidVocabularyError(vocabulary).with_field_and_value(
(Pdb++) 
(Pdb++) self
<zope.schema._field.Choice object at 0x7f1a9051d070 pnz.erpediem.client.browser.create_edi_order.IEDIOrderItemsRowSchema.order_item_article>
(Pdb++) self.source
<pnz.erpediem.core.vocabularies.edifact.sources.ItalianCitiesSourceBinder object at 0x7f1a7fed4e50>
(Pdb++) self.source(self.context).getTerm('bologna')
<zope.schema.vocabulary.SimpleTerm object at 0x7f1a81aaca90>
(Pdb++) self.source(self.context).getTerm('bologna')
<zope.schema.vocabulary.SimpleTerm object at 0x7f1a81aacbb0>
(Pdb++) self.source(self.context).getTerm('bologna').title
'Bologna'

My field:

    order_item_article = schema.Choice(
        title    = _(
            'order_item_article_title',
            default=u'Article'
        ),
        required    = True,
        #source = EDIOrderItemsSourceBinder(),
        source = ItalianCitiesSourceBinder(),
        default = '--NOVALUE--'
    )
    directives.widget(
        'order_item_article',
        SelectFieldWidget,
        allowNewItems = False,
    ) 

Which version of zope.schema are you using? in 6.0-latest it's defined as zope.schema=7.0.1 but your line numbers look like a different version ...

Hi @petschki version 6.2.1

So you need to figure out why your vocabulary doesn't provide ISource ... zope.schema/_field.py at 6.2.1 · zopefoundation/zope.schema · GitHub

Thanks, I did add ISource as a @provider earlier during troubleshooting - without effect. Right now I'm stuck in upgrade purgatory, attempting to update my Plone instance to 6.0-latest :wink:

"Should be easy" if I can find where Products.CMFDynamicViewFTI.interface ended up...

File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/standardtiles/navigation.py", line 18, in <module>
    from Products.CMFDynamicViewFTI.interface import IBrowserDefault
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/standardtiles/layout.zcml", line 9.2-18.8
    File "/usr/local/Plone6/instance/etc/site.zcml", line 16.2-16.43
    File "/usr/local/Plone6/lib/python3.8/site-packages/collective/easyform/configure.zcml", line 8.2-8.43
    File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/dexterity/configure.zcml", line 11.2-11.41
    File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/z3cform/configure.zcml", line 11.2-11.41
    File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/widgets/configure.zcml", line 6.2-6.41
    File "/usr/local/Plone6/lib/python3.8/site-packages/Products/CMFPlone/configure.zcml", line 118.2-122.8
    File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/mosaic/configure.zcml", line 11.2-11.47
    File "/usr/local/Plone6/lib/python3.8/site-packages/plone/app/standardtiles/configure.zcml", line 36.2-36.32
    ModuleNotFoundError: No module named 'Products.CMFDynamicViewFTI.interface'

Edit: AHA!

I am out of my league here, adding a @provider decorator to the ItalianCitiesSourceBinder class had no effect.

I then decided to find out what zope.schema._form actually sees when it tries to resolve the vocabulary.

What gives??? Why does it see the same field twice, and the second time without providing ISource? Is this the ghost row (AA ???) that is added by dgf?

self z3c.form.interfaces.IInputForm.method
IContextSourceBinder.providedBy(vocabulary) False
self.context <Products.Five.browser.metaconfigure.SimpleViewClass from /usr/local/Plone6/src/pnz.erpediem.client/src/pnz/erpediem/client/browser/templates/create_edi_order.pt object at 0x7fb209262c70>
ISource.providedBy(vocabulary) True
self z3c.form.interfaces.IInputForm.method
IContextSourceBinder.providedBy(vocabulary) False
self.context <Products.Five.browser.metaconfigure.SimpleViewClass from /usr/local/Plone6/src/pnz.erpediem.client/src/pnz/erpediem/client/browser/templates/create_edi_order.pt object at 0x7fb209262c70>
ISource.providedBy(vocabulary) True
self pnz.erpediem.client.browser.create_edi_order.IEDIOrderItemsRowSchema.order_item_article
IContextSourceBinder.providedBy(vocabulary) True
self.context None
ISource.providedBy(vocabulary) False
self pnz.erpediem.client.browser.create_edi_order.IEDIOrderItemsRowSchema.order_item_quantity
IContextSourceBinder.providedBy(vocabulary) False
self.context None
ISource.providedBy(vocabulary) True
self pnz.erpediem.client.browser.create_edi_order.IEDIOrderItemsRowSchema.order_item_article
IContextSourceBinder.providedBy(vocabulary) True
self.context None
ISource.providedBy(vocabulary) False

Hah, that seems to be the case: if I remove the ISource check and skip validation, the failure seems a bit clearer... @petschki @davisagli @jensens does this look like a bug to you too? If so, I could make a simplified package that shows this in action.

2023-05-24 14:48:45,067 ERROR   [Zope.SiteErrorLog:17][waitress-0] ComponentLookupError: http://localhost:8888/Plone/api/edi-order
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 181, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 390, in publish_module
  Module ZPublisher.WSGIPublisher, line 285, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module Products.PDBDebugMode.wsgi_runcall, line 60, in pdb_runcall
  Module Shared.DC.Scripts.Bindings, line 333, in __call__
  Module Shared.DC.Scripts.Bindings, line 370, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 338, in _exec
  Module script, line 16, in edi-order
   - <PythonScript at /Plone/api/edi-order>
   - Line 16
  Module z3c.form.form, line 233, in __call__
  Module pnz.erpediem.client.browser.create_edi_order, line 346, in update
  Module plone.z3cform.fieldsets.extensible, line 62, in update
  Module plone.z3cform.patch, line 31, in GroupForm_update
  Module z3c.form.group, line 132, in update
  Module z3c.form.form, line 136, in updateWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.multi, line 63, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 509, in update
  Module Products.CMFPlone.patches.z3c_form, line 46, in _wrapped
  Module z3c.form.widget, line 132, in update
  Module z3c.form.widget, line 504, in value
  Module collective.z3cform.datagridfield.datagridfield, line 154, in updateWidgets
  Module collective.z3cform.datagridfield.datagridfield, line 126, in getWidget
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.object, line 297, in update
  Module collective.z3cform.datagridfield.datagridfield, line 267, in updateWidgets
  Module z3c.form.object, line 222, in updateWidgets
  Module z3c.form.object, line 216, in setupWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.multi, line 63, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 509, in update
  Module Products.CMFPlone.patches.z3c_form, line 46, in _wrapped
  Module z3c.form.widget, line 132, in update
  Module z3c.form.widget, line 504, in value
  Module collective.z3cform.datagridfield.datagridfield, line 148, in updateWidgets
  Module z3c.form.widget, line 445, in updateWidgets
  Module collective.z3cform.datagridfield.datagridfield, line 126, in getWidget
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.object, line 297, in update
  Module collective.z3cform.datagridfield.datagridfield, line 267, in updateWidgets
  Module z3c.form.object, line 222, in updateWidgets
  Module z3c.form.object, line 216, in setupWidgets
  Module z3c.form.field, line 274, in update
  Module z3c.form.browser.select, line 51, in update
  Module z3c.form.browser.widget, line 171, in update
  Module z3c.form.widget, line 233, in update
  Module z3c.form.widget, line 225, in updateTerms
  Module zope.component._api, line 104, in getMultiAdapter
zope.interface.interfaces.ComponentLookupError: ((None, <WSGIRequest, URL=http://localhost:8888/Plone/api/edi-order>, <DataGridFieldObjectWidget 'edi_order_form.widgets.orders_data.AA.widgets.order_items_data.0'>, <zope.schema._field.Choice object at 0x7fb8da0bdb20 pnz.erpediem.client.browser.create_edi_order.IEDIOrderItemsRowSchema.order_item_article>, <SelectWidget 'edi_order_form.widgets.orders_data.AA.widgets.order_items_data.0.widgets.order_item_article'>), <InterfaceClass z3c.form.interfaces.ITerms>, '')
[49] > /usr/local/Plone6/lib/python3.8/site-packages/zope/component/_api.py(104)getMultiAdapter()
-> raise ComponentLookupError(objects, interface, name)

The only thing I see is, that the first item in the objects tuple is None which should be a valid context, otherwise the IContextSourceBinder cannot work. So it's likely, that datagridfield fails to resolve the correct context for sub-widgets. I remember fiddling around this when validating a widget in a DictRow but maybe there's missing more when updateing the widgets ... collective.z3cform.datagridfield/row.py at master · collective/collective.z3cform.datagridfield · GitHub

From what I see, it never even gets that far:

    def _validate(self, value):
        # XXX HACK: Can't call the super, since it'll check to
        # XXX see if we provide DictRow.
        # We're only a dict, so we can't.
        # super()._validate(value)

        # Validate the dict against the schema
        # Pass 1 - ensure fields are present

        print ('DGF Validate is called')
        
        if value is NO_VALUE:
            return
        # Treat readonly fields
        for field_name in getFields(self.schema).keys():
            field = self.schema[field_name]
            if field.readonly:
                value[field_name] = field.default
        errors = []
        for field_name in getFields(self.schema).keys():
            if field_name not in value:
                errors.append(AttributeNotFoundError(field_name, self.schema))

        if errors:
            raise WrongContainedType(errors, self.__name__)

        # Pass 2 - Ensure fields are valid
        
        print('DGF pass 2 was reached')

        for field_name, field_type in getFields(self.schema).items():

            print ('field_name %s' % field_name)
            print ('field_type %s' % field_type)
            print ('IChoice.providedBy(field_type) %s' % IChoice.providedBy(field_type))
            
            if IChoice.providedBy(field_type):
                # Choice must be bound before validation otherwise
                # IContextSourceBinder is not iterable in validation
                bound = field_type.bind(self.context)
                
                print ("DGF pass 2, bound is %s" % bound)
                print ("DGF pass 2, context is %s" % self.context)
                
                bound.validate(value[field_name])
            else:
                field_type.validate(value[field_name])

DGF Validate is called
DGF pass 2 was reached
field_name order_id_customer
field_type pnz.erpediem.client.browser.create_edi_order.IEDIOrderCustomerRowSchema.order_id_customer
IChoice.providedBy(field_type) False
field_name shipping_name
field_type pnz.erpediem.client.browser.create_edi_order.IEDIOrderCustomerRowSchema.shipping_name
IChoice.providedBy(field_type) False
field_name shipping_street
field_type pnz.erpediem.client.browser.create_edi_order.IEDIOrderCustomerRowSchema.shipping_street
IChoice.providedBy(field_type) False
field_name shipping_city
field_type pnz.erpediem.client.browser.create_edi_order.IEDIOrderCustomerRowSchema.shipping_city
IChoice.providedBy(field_type) False
field_name shipping_country
field_type pnz.erpediem.client.browser.create_edi_order.IEDIOrderCustomerRowSchema.shipping_country
IChoice.providedBy(field_type) False
2023-05-24 15:35:37,339 ERROR   [Zope.SiteErrorLog:17][waitress-0] InvalidVocabularyError: http://localhost:8888/Plone/api/edi-order

For obvious reason, because your field doesn't provide IChoice

Yes obvious indeed :wink: , it iterates through the first level fields, and never makes it to the DGF with the choice field.

The full schema:


class IEDIOrderCustomerRowSchema(Interface):
    """ EDI Order single row schema for customer / shipping information
    """

    order_id_customer = schema.TextLine(
        title    = _(
            'order_id_customer_title',
            default=u'Your Order Number'
        ),
        required = False,
        default = u'',
    )   

    shipping_name = schema.TextLine(
        title    = _(
            'shipping_name_title',
            default=u'Customer Full Name'
        ),
        required = False,
        default = u'',
    )   

    shipping_street = schema.TextLine(
        title    = _(
            'shipping_street_title',
            default=u'Customer Street and Number'
        ),
        required = False,
        default = u'',
    )   

    shipping_city = schema.TextLine(
        title    = _(
            'shipping_city_title',
            default=u'Postal Code and City'
        ),
        required = False,
        default = u'',
    )   

    shipping_country = schema.TextLine(
        title    = _(
            'shipping_country_title',
            default=u'Country'
        ),
        required = False,
        default = u'',
    )   


class IEDIOrderItemsRowSchema(Interface):
    """ EDI Order row schema for article items and quantities in an order
    """

    """
    order_item_article = schema.Choice(
        title    = _(
            'order_item_article_title',
            default=u'Article'
        ),
        required    = True,
        values=('this field does not work', 'choice 2', 'choice 3'),
        )
    """
    order_item_article = schema.Choice(
        title    = _(
            'order_item_article_title',
            default=u'Article'
        ),
        required    = True,
        #source = EDIOrderItemsSourceBinder(),
        source = ItalianCitiesSourceBinder(),
        default = '--NOVALUE--'
    )
    directives.widget(
        'order_item_article',
        SelectFieldWidget,
        allowNewItems = False,
    ) 
    
    order_item_quantity = schema.Choice(
        title    = _(
            'order_item_quantity_title',
            default=u'Quantity'
        ),
        required = True,
        values = range(0,10),
        default = 0,
    )


class IEDIOrdersRowSchema(Interface):
    """ EDI Orders row schema for customer / order items information
    """
    customer_data = schema.List(
        title       = _(
            'customer_data_title',
            default=u'Customer Data',
            ),
        description = _('customer:data_description',
            default=u'Enter the customer information'),
        required    = True,
        value_type  = DictRow(
                title  = u"tablerow",
                schema = IEDIOrderCustomerRowSchema,
        ),
        default     = [{
            'order_id_customer' : u'',
            'shipping_name'     : u'',
            'shipping_street'   : u'',
            'shipping_city'     : u'',
            'shipping_country'  : u'',
        }],
    )
    directives.widget(
        'customer_data',
        BlockDataGridFieldWidgetFactory,
        allow_reorder = False,
        allow_insert = False,
        allow_delete = False,
        auto_append = False,
    )

    order_items_data = schema.List(
        title       = _(
            'order_items_data_title',
            default=u'Order Items',
            ),
        description = _('order_items_data_description',
            default=u'Enter the ordered articles and quantities'),
        required    = True,
        value_type  = DictRow(
                title  = u"tablerow",
                schema = IEDIOrderItemsRowSchema,
        ),
        default     = [{  
            'order_item_article' : '--NOVALUE--',
            'order_item_quantity' : 0,
            },
        ],
    )
    directives.widget(
        'order_items_data',
        DataGridFieldWidgetFactory,
        auto_append = False,
    )
    

class IEDIOrdersSchema(Interface):
    """ EDI Orders main schema interface
    """
    
    customer_id = schema.TextLine(
        title    = _(
            'customer_id_title',
            default=u'Customer ID'
        ),
        required = True,
        default = u'',
    )   
    directives.mode(customer_id='display')

    order_date = schema.Datetime(
        title=_(
            'date_title',
            default = u'Order Date & Time (UTC)',
        ),
        required = True,
    )   
    directives.mode(order_date='display')
    #directives.mode(order_date='hidden')
    
    orders_data = schema.List(
        title       = _(
            'orders_data_title',
            default=u'Your Orders',
            ),
        description = _('orders_data_description',
            default=u'Enter your orders'),
        required    = True,
        value_type  = DictRow(
                title  = u"tablerow",
                schema = IEDIOrdersRowSchema,
        ),
        default     = [],
    )
    directives.widget(
        'orders_data',
        DataGridFieldWidgetFactory,
        #auto_append = True
    )

    # For testing: this works
    mainform_order_item_article = schema.Choice(
        title    = _(
            'order_item_article_title',
            default=u'Article'
        ),
        required    = True,
        #source = EDIOrderItemsSourceBinder(),
        source = ItalianCitiesSourceBinder(),
        default = '--NOVALUE--'
    )
    directives.widget(
        'mainform_order_item_article',
        SelectFieldWidget,
        allowNewItems = False,
    )