TransformError: Error during transformation after upgrade from Plone 5.1.1 to 5.1.4

Plone Support
1

I'm seeing a "TransformError" after an upgrade to Plone 5.1.4.
I had a similar problem a good couple months ago. At the time the issue seemed to be related to the version of lxml. This time the lxml version is fine. I'm still trying to work out the issue.

I'm pretty sure it has something to do with plone.app.textfield's output_relative_to() function.

Traceback

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.Five.browser.metaconfigure, line 485, in __call__
  Module Products.Five.browser.pagetemplatefile, line 125, in __call__
  Module Products.Five.browser.pagetemplatefile, line 59, in __call__
  Module zope.pagetemplate.pagetemplate, line 137, in pt_render
  Module five.pt.engine, line 98, in __call__
  Module z3c.pt.pagetemplate, line 163, in render
  Module chameleon.zpt.template, line 261, in render
  Module chameleon.template, line 191, in render
  Module chameleon.template, line 171, in render
  Module 6469ff6e372688ee93f391ccdc9417b3.py, line 246, in render
  Module a468d21c2e8b73537d024f4c5e5e3c0d.py, line 1223, in render_master
  Module a468d21c2e8b73537d024f4c5e5e3c0d.py, line 420, in render_content
  Module 6469ff6e372688ee93f391ccdc9417b3.py, line 234, in __fill_content_core
  Module 6469ff6e372688ee93f391ccdc9417b3.py, line 137, in render_content_core
  Module plone.app.textfield.value, line 110, in output_relative_to
  Module plone.app.textfield.transform, line 69, in __call__
TransformError: Error during transformation

 - Expression: "python:context.text.output_relative_to(view.context)"
 - Filename:   ... egg/plone/app/contenttypes/browser/templates/newsitem.pt
 - Location:   (line 15: col 29)
 - Source:     ... ucture python:context.text.output_relative_to(view.context)"
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 - Arguments:  repeat: {...} (0)
               template: <ViewPageTemplateFile - at 0x7f4c16b86e10>
               views: <ViewMapper - at 0x7f4c1587eb90>
               modules: <instance - at 0x7f4c1eafcab8>
               args: <tuple - at 0x7f4c29858050>
               here: <ImplicitAcquisitionWrapper rules-as-commodities at 0x7f4c192f8cd0>
               user: <ImplicitAcquisitionWrapper - at 0x7f4c1909dc30>
               nothing: <NoneType - at 0x7ba070>
               container: <ImplicitAcquisitionWrapper rules-as-commodities at 0x7f4c192f8cd0>
               request: <instance - at 0x7f4c1896bd40>
               wrapped_repeat: <SafeMapping - at 0x7f4c15863ec0>
               traverse_subpath: <list - at 0x7f4c18a0ff80>
               default: <object - at 0x7f4c2982ea90>
               loop: {...} (0)
               context: <ImplicitAcquisitionWrapper rules-as-commodities at 0x7f4c192f8cd0>
               view: <SimpleViewClass from /home/ubuntu/buildout-cache/eggs/plone.app.contenttypes-1.4.15-py2.7.egg/plone/app/contenttypes/browser/templates/newsitem.pt newsitem_view at 0x7f4c1587e650>
               translate: <function translate at 0x7f4c160e7410>
               root: <ImplicitAcquisitionWrapper Zope at 0x7f4c19268870>
               options: {...} (0)
               target_language: <NoneType - at 0x7ba070>
2

I decided to pin lxml and plone.app.textfield to their older versions that ship with Plone 5.1.1
(lxml 4.1.1 and plone.app.textfield 1.2.10)

This is additional traceback information that is showing up on the console, interestingly this mentions a UnicodeEncodeError. (I might not have been paying attention, so these tracebacks may have been there before the pinning.)

Traceback (most recent call last):
 File "/home/ubuntu/buildout-cache/eggs/plone.app.textfield-1.2.10-py2.7.egg/plone/app/textfield/transform.py", line 63, in __call__
    return output.decode(value.encoding)
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 261: ordinal not in range(128)
3

So I decided to tinker a bit.
I went into plone.app.textfield transform.py and did something horrible.
I commented out the .decode() and now I don't get the UnicodeEncodeError because I'm totally skipping the problem.

The site works as expected but I'm pretty sure this is the wrong fix and I'm setting myself up for issues in the future.

output = data.getData()
               return output # .decode(value.encoding)
4

plone.app.textfield does not use lxml.
https://github.com/plone/plone.app.textfield/search?q=lxml&unscoped_q=lxml

You're feeding it the wrong kind of data.

5

Agreed (so changing lxml was not based on the current issue, was hoping it was similar to a previous issue, so I tried out the lxml thing before digging deeper).

Basic inspection and it is definitely plone.app.textfield and richtext related.
But why would this become an issue AFTER an upgrade when everything was fine before?

This is my latest "exploration" using ipdb.

> /home/ubuntu/buildout-cache/eggs/plone.app.textfield-1.2.10-py2.7.egg/plone/app/textfield/transform.py(64)__call__()
     63                 import ipdb;ipdb.set_trace()
---> 64                 return output.decode(value.encoding)
     65         except ConflictError:

ipdb> output.format
<built-in method format of unicode object at 0x7f483b1251b0>
ipdb> output.format()
...rich history that we are proud of, but we are not ready to rest or revel in the past.\xa0 We ...more content on our ....</p>&#13;\n<p>\xa0</p>&#13;\n<p>Let me close by telling you that I am...to serving, collaborating and hearing from you.</p>&#13;\n<p>\xa0</p>&#13;\n<p>Very truly yours,</p>&#13;\n<p>... 2017-2018</p>&#13;\n'
ipdb> type(output)
<type 'unicode'>
ipdb> output.decode()
*** UnicodeEncodeError: 'ascii' codec can t encode character u'\xa0' in position 245: ordinal not in range(128)
ipdb> value.encoding
'utf-8'
ipdb> output.decode(value.encoding)
*** UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 245: ordinal not in range(128)
ipdb>
6

A further experiment.

  • I created a new document with just ascii characters and it worked.
  • I then added a unicode smartquote to the document and saved it again.

The result: I got a similar traceback as above. There appears to be a unicode issue that didn't exist before the upgrade.

7

@Rotonen,
Thanks for pointing out the core issue, documented in this thread:

Since it is introduced with a new version of Products.PortalTransforms, I pinned the old version in my buildout versions and now everything works again.

Products.PortalTransforms = 3.1.2 works for me.

8

That really is strange: how can an utf-8 decoding complain about the ascii codec?

9

At that level, we're talking straight python. My current answer? I don't know enough.

10

I understand by now: decode('utf-8') is used to convert an utf-8 encoded (binary) string to unicode. In your observation, it is applied to a unicode object. As the utf-8 decode operation requires a binary string, the unicode is first converted to a binary string (using the default encoding ascii) and this fails if the unicode is not pure ascii.

Others already pointed out a fix: apply the decode only if the value is a binary string, not to a unicode object.

11

@dieter,
Since new versions of Products.PortalTransforms.data.datastream.getData() no longer consistently return strings, a fix needs to be introduced to plone.app.textfield (basically to detect the type before attempting the decode as you suggest).

12

The same problem exists in 5.1.2, incidentally (in case anyone was wondering).

13

I got the same issue after the upgrade to Plone 5.1.4. The pinning
Products.PortalTransforms = 3.1.2
works for me, also.