We have a strange issue on a migrated custom Content-Type of ours. We have an old site from Plone 3 which was migrated to Plone 5, within this migration we find the old items and re-create them with the Plone API so that we have a new catalog item. Within this migration we move over the values of the various schema specific fields of the old Content-Type to the new one, however for some reason the indexes of title and in this case, lastname are causing a unicode error when rebuilding the catalog or saving an item. When these indexes are removed, we can both update the field and also rebuild the catalog. If we re-add the indexes, it once again fails.
My understanding of this issue is possibly down to a unicode to string comparison which is failing somewhere, however it's causing quite a headache to fix.
Here is the traceback that happens:
../.buildout/eggs/z3c.form-3.2.11-py2.7.egg/z3c/form/util.py:180: UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if (not dm.canAccess() or dm.query() != value): ../.buildout/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py:193: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal indexRow = self._index.get(entry, _marker)  > ../.buildout/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py(193)insertForwardIndexEntry() -> indexRow = self._index.get(entry, _marker)
Traceback (innermost last): Module ZPublisher.Publish, line 138, in publish Module ZPublisher.mapply, line 77, in mapply Module Products.PDBDebugMode.runcall, line 70, in pdb_runcall Module ZPublisher.Publish, line 48, in call_object Module plone.z3cform.layout, line 66, in __call__ Module plone.z3cform.layout, line 50, in update Module plone.dexterity.browser.edit, line 58, in update Module plone.z3cform.fieldsets.extensible, line 59, in update Module plone.z3cform.patch, line 30, in GroupForm_update Module z3c.form.group, line 145, in update Module plone.app.z3cform.csrf, line 21, in execute Module z3c.form.action, line 98, in execute Module z3c.form.button, line 315, in __call__ Module z3c.form.button, line 170, in __call__ Module plone.dexterity.browser.edit, line 30, in handleApply Module z3c.form.group, line 126, in applyChanges Module zope.event, line 31, in notify Module zope.component.event, line 24, in dispatch Module zope.component._api, line 136, in subscribers Module zope.component.registry, line 321, in subscribers Module zope.interface.adapter, line 585, in subscribers Module zope.component.event, line 32, in objectEventNotify Module zope.component._api, line 136, in subscribers Module zope.component.registry, line 321, in subscribers Module zope.interface.adapter, line 585, in subscribers Module plone.dexterity.content, line 774, in reindexOnModify Module Products.CMFCore.CMFCatalogAware, line 88, in reindexObject Module Products.CMFCore.CatalogTool, line 301, in reindexObject Module Products.CMFPlone.CatalogTool, line 351, in catalog_object Module Products.PDBDebugMode.zcatalog, line 20, in catalog_object Module Products.ZCatalog.ZCatalog, line 476, in catalog_object Module Products.ZCatalog.Catalog, line 360, in catalogObject Module Products.PluginIndexes.common.UnIndex, line 216, in index_object Module Products.PluginIndexes.common.UnIndex, line 242, in _index_object Module Products.PluginIndexes.common.UnIndex, line 193, in insertForwardIndexEntry UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)  > ../.buildout/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/PluginIndexes/common/UnIndex.py(193)insertForwardIndexEntry() -> indexRow = self._index.get(entry, _marker)
The strange thing is that special characters such as umlauts also fail with the field being either encoded or decoded, I have no idea what's going on..?
I also attempted to rectify any string encoding errors but with little success, this was the code I ran on the Content-Type:
def contact_cleanup(self): from plone.protect.interfaces import IDisableCSRFProtection from zope.interface import alsoProvides import transaction alsoProvides(self.request, IDisableCSRFProtection) contacts = api.content.find(portal_type='Contact') contacts = [contact.getObject() for contact in contacts] for item in contacts: setattr(item, 'sortname', '') if isinstance(item.firstname, unicode): setattr(item, 'firstname', item.firstname.encode('utf-8')) if isinstance(item.lastname, unicode): setattr(item, 'lastname', item.lastname.encode('utf-8')) item.reindexObject(idxs=('sortname', 'firstname', 'lastname')) transaction.commit()
If anyone has some ideas why this may be failing I'd appreciate some insight, the main issue is the site is now migrated and while it'd normally be better to go back, debug and fix the issue. We now have to fix this post-migration...