Collective.exportimport: Warning that it cannot deserialize provided json content

I'm using collective.exportimport to import json content. The content was exported from another Plone site using collective.exportimport. Is it safe to ignore the following warning during an import? I'm not 100% sure of what the warning means.

WARNING [collective.exportimport.import_content:365][waitress-1] cannot deserialize http://localhost:8080/Plone/committees/finance/resources/xx-4521-provider-solution: BadRequest([{'message': 'Constraint not satisfied', 'field': 'committee', 'error': 'ValidationError'}])

There's a custom content type called committee, in this case the 'finance' committee and a custom content type called bill, in this case the 'xx-4521' bill.

The associated object from the provided json looks like this:

  "@id": "http://localhost:8080/Plone/committees/finance/resources/xx-4521-provider-solution",
    "@type": "bill",
    "UID": "e99ba9eac2f94a19b95db1a2e099182d",
    "allow_discussion": true,
    "bill_number": "XXXXX",
 "body": {
        "content-type": "text/html",
        "data": "<!DOCTYPE html>\r\n<html>\r\n<head>\r\n</head>\r\n<body>\r\n <p>The Commissioner shall promulgate  To the fullest extent practicable, these standards and procedures shall incorporate the use of<span>\u00a0</span>
... snip ...
<span>blockchain</span></a><span>\u00a0</span>technology, as defined in this act, on a permanent, statewide basis. If making such a recommendation, the report shall also include legislative recommendations to improve the efficiency, effectiveness, and security of such</u></p>\r\n</body>\r\n</html>",
        "encoding": "utf-8"
    },
    "committee": {
        "title": "Finance\t",
        "token": "Finance\\t"
    },
 "contributors": [],
    "copy_of_the_bill": null,
    "created": "2019-01-14T16:38:13+00:00",
    "creators": [
        "04190735"
    ],
    "description": "Relates to electronic return ...pilot program may return ...",
    "effective": "2019-01-09T10:40:00",
    "expires": null,
    "id": "xx-4521",
    "is_folderish": true,
    "jurisdiction": {
        "title": "Virginia ",
        "token": "Virginia "
    },
    "language": {
        "title": "English (USA)",
        "token": "en-us"
    },
    "layout": "view",
... snip ...

The validation error is because the committee field has a value that is not allowed by the field's vocabulary. The root cause depends on the details of how the vocabulary values are constructed.

Is the token in the JSON correct? It looks like it has an escaped tab character at the end.

(We should really improve that "Constraint not satisfied" message to be more specific about what the constraint was. I think it comes from zope.schema, if I remember right.)

Yup... you're right!... I did some investigation and it is a field that links to a named committee, the committee name was changed from what it was originally and "broke" the link and "vocabulary".
Thanks @davisagli

In general: this is a warning (in my understanding). The original value is always assigned back to the related object. The plone.restapi deserializer is used to generate these warnings (for validation purposes). However, it does not avoid the assignment (which is a good thing). You clearly will run into UI issues or warnings if the related value is not available as vocabulary value.

1 Like

I have several migrations with similar problems and have written a two-part solution that I will add to the section FAQ, Tips and Tricks of the docs.

It involves:

  1. Custom serializers for Choice and Collection-Fields that log messages if a value on the object is not allowed in the vocabulary.
  2. A simple setter that imports the data despite that.

As a developer you should to check the result of 1 before implementing 2 so you don't import data that you don't want.

Serializers:

@adapter(ICollection, IDexterityContent, IMigrationMarker)
@implementer(IFieldSerializer)
class CollectionFieldSerializer(DefaultFieldSerializer):
    def __call__(self):
        """Override default serializer:
        1. Export only the value, not a token/title dict.
        2. Do not drop values that are not in the vocabulary.
           Instead log info so you can handle the data.
        """
        # Binding is necessary for named vocabularies
        if IField.providedBy(self.field):
            self.field = self.field.bind(self.context)
        value = self.get_value()
        value_type = self.field.value_type
        if (
            value is not None
            and IChoice.providedBy(value_type)
            and IVocabularyTokenized.providedBy(value_type.vocabulary)
        ):
            for v in value:
                try:
                    value_type.vocabulary.getTerm(v)
                except LookupError:
                    # TODO: handle defaultFactory?
                    if v not in [self.field.default, self.field.missing_value]:
                        logger.info("Term lookup error: %r not in vocabulary %r for field %r of %r", v, value_type.vocabularyName, self.field.__name__, self.context)
        return json_compatible(value)


@adapter(IChoice, IDexterityContent, IMigrationMarker)
@implementer(IFieldSerializer)
class ChoiceFieldSerializer(DefaultFieldSerializer):
    def __call__(self):
        """Override default serializer:
        1. Export only the value, not a token/title dict.
        2. Do not drop values that are not in the vocabulary.
           Instead log info so you can handle the data.
        """
        # Binding is necessary for named vocabularies
        if IField.providedBy(self.field):
            self.field = self.field.bind(self.context)
        value = self.get_value()
        if value is not None and IVocabularyTokenized.providedBy(self.field.vocabulary):
            try:
                self.field.vocabulary.getTerm(value)
            except LookupError:
                # TODO: handle defaultFactory?
                if value not in [self.field.default, self.field.missing_value]:
                    logger.info("Term lookup error: %r not in vocabulary %r for field %r of %r", value, self.field.vocabularyName, self.field.__name__, self.context)
        return json_compatible(value)

Deserializer


SIMPLE_SETTER_FIELDS = {
    "ALL": [],
    "CollaborationFolder": ["allowedPartnerDocTypes"],
    "DocType": ["automaticTransferTargets"],
    "DPDocument": ["scenarios"],
    "DPEvent" : ["Status"],
}

class CustomImportContent(ImportContent):

    def global_dict_hook(self, item):
        # handle fields that should be set using a simple setter instead of a deserializer
        # this works around issues with validation
        simple = {}
        for fieldname in SIMPLE_SETTER_FIELDS.get("ALL", []):
            if fieldname in item:
                value = item.pop(fieldname)
                if value:
                    simple[fieldname] = value
        for fieldname in SIMPLE_SETTER_FIELDS.get(item["@type"], []):
            if fieldname in item:
                value = item.pop(fieldname)
                if value:
                    simple[fieldname] = value
        if simple:
            item["exportimport.simplesetter"] = simple

    def global_obj_hook_before_deserializing(self, obj, item):
        """Hook to modify the created obj before deserializing the data.
        """
        # import simplesetter data before the rest
        for fieldname, value in item.get("exportimport.simplesetter", {}).items():
            setattr(obj, fieldname, value)