Hello everyone, our testing matrix start failing recently for Plone 5.1 builds, can someone help us please?
More information here:
Running collective.cover.testing.collective.cover:Integration tests: Set up plone.testing.zca.LayerCleanup in 0.000 seconds. Set up plone.testing.z2.Startup in 0.340 seconds. Set up plone.app.testing.layers.PloneFixture in 9.700 seconds. Set up collective.cover.testing.Fixture in 4.652 seconds. Set up collective.cover.testing.collective.cover:Integration in 0.000 seconds. Running: 7/8 (87.5%)> /home/rodfersou/.projects/collective.cover.p5/src/collective/cover/tiles/richtext.py(103)SearchableText() 102 --> 103 searchable_text = six.text_type(data.getData(), 'utf-8') 104 return searchable_text.strip() # remove leading and trailing spaces ipdb> data <Products.PortalTransforms.data.datastream object at 0x7ff58c4dfb30> ipdb> data.getData() u' My document text... ' ipdb> six.text_type(data.getData(), 'utf-8') *** TypeError: decoding Unicode is not supported
it should be something in the last release 3.1.4 (2018-09-23) https://github.com/plone/Products.PortalTransforms/compare/3.1.3...3.1.4
The quick fix is to change collective.cover's code here:
searchable_text = data.getData() if isinstance(searchable_text, six.binary_type): searchable_text = searchable_text.decode('utf-8')
from Products.CMFPlone.utils import safe_unicode return safe_unicode(data.getData())
Unluckily the output of
convertTo has changed in a non retrocompatible way in a minor release and nobody noticed this.
convertTo should return a
str object in both Python 2 and 3.
That is very bad and I am sorry it happened but it is good that now we know.
Not really sure how to fix this easily and in a fast way upstream.
fortunately we have pretty good tests in
collective.cover that had lead to discover early a couple of issues on the core in the past .
Products.PortalTransforms should be fixed before releasing Plone 5.2a1.
@rodfersou are you able to create a pull request with a failing test in https://github.com/plone/Products.PortalTransforms ? ... since core tests were passing we might have missed something there.
I'd recommend following best practice for transforming RichTextValue here https://github.com/plone/plone.app.contenttypes/blob/master/plone/app/contenttypes/indexers.py#L51 ...
But this method was not changed in the last release.
convertTo method returns an object implementing
IDataStream ... so in detail, the
text/plain transformer is a subclass of
Products.PortalTransforms.libtransforms.retransform.retransform and this returns always a six.text_type since the last change of @pbauer (https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#diff-667edf09d266049d938e4a0bda2a20b1R29) ... maybe the reason why your transform doesn't work anymore.
@pigeonflight I think this is the change you're fighting with in your thread.
So maybe converge here as the root cause is now known.
My quick fix was to pin
Products.PortalTransforms = 3.1.2
Well spotted @petschki!
In this case the fix might be https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#r30973076If somebody has time to open a PR he is welcome.
Otherwise I will do one later/tomorrow
Here it is https://github.com/plone/Products.PortalTransforms/pull/38.
I modified the tests to ensure that the output type is not going to change in the future.
JFTR, Products.PortalTransforms 3.1.5 has been released and it solves the issue:
thanks, @alert, you rock!