Hello everyone, our testing matrix start failing recently for Plone 5.1 builds, can someone help us please?
https://travis-ci.org/collective/collective.cover/jobs/441921182
Hello everyone, our testing matrix start failing recently for Plone 5.1 builds, can someone help us please?
https://travis-ci.org/collective/collective.cover/jobs/441921182
More information here:
Running collective.cover.testing.collective.cover:Integration tests:
Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
Set up plone.testing.z2.Startup in 0.340 seconds.
Set up plone.app.testing.layers.PloneFixture in 9.700 seconds.
Set up collective.cover.testing.Fixture in 4.652 seconds.
Set up collective.cover.testing.collective.cover:Integration in 0.000 seconds.
Running:
7/8 (87.5%)> /home/rodfersou/.projects/collective.cover.p5/src/collective/cover/tiles/richtext.py(103)SearchableText()
102
--> 103 searchable_text = six.text_type(data.getData(), 'utf-8')
104 return searchable_text.strip() # remove leading and trailing spaces
ipdb> data
<Products.PortalTransforms.data.datastream object at 0x7ff58c4dfb30>
ipdb> data.getData()
u' My document text... '
ipdb> six.text_type(data.getData(), 'utf-8')
*** TypeError: decoding Unicode is not supported
@hvelarde it is weird, the last change in this module was 7 months ago, all other changes was years ago.
it should be something in the last release 3.1.4 (2018-09-23) https://github.com/plone/Products.PortalTransforms/compare/3.1.3...3.1.4
The quick fix is to change collective.cover's code here:
searchable_text = data.getData()
if isinstance(searchable_text, six.binary_type):
searchable_text = searchable_text.decode('utf-8')
or
from Products.CMFPlone.utils import safe_unicode
return safe_unicode(data.getData())
Unluckily the output of convertTo
has changed in a non retrocompatible way in a minor release and nobody noticed this.
Apparently convertTo
should return a str
object in both Python 2 and 3.
That is very bad and I am sorry it happened but it is good that now we know.
Not really sure how to fix this easily and in a fast way upstream.
fortunately we have pretty good tests in collective.cover
that had lead to discover early a couple of issues on the core in the past .
IMO, Products.PortalTransforms
should be fixed before releasing Plone 5.2a1.
@rodfersou are you able to create a pull request with a failing test in https://github.com/plone/Products.PortalTransforms ? ... since core tests were passing we might have missed something there.
I'd recommend following best practice for transforming RichTextValue here https://github.com/plone/plone.app.contenttypes/blob/master/plone/app/contenttypes/indexers.py#L51 ...
the convertTo
method returns an object implementing IDataStream
... so in detail, the text/plain
transformer is a subclass of Products.PortalTransforms.libtransforms.retransform.retransform
and this returns always a six.text_type since the last change of @pbauer (https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#diff-667edf09d266049d938e4a0bda2a20b1R29) ... maybe the reason why your transform doesn't work anymore.
@pigeonflight I think this is the change you're fighting with in your thread.
So maybe converge here as the root cause is now known.
Yes!
My quick fix was to pin
Products.PortalTransforms = 3.1.2
Well spotted @petschki!
In this case the fix might be https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#r30973076If somebody has time to open a PR he is welcome.
Otherwise I will do one later/tomorrow
Here it is https://github.com/plone/Products.PortalTransforms/pull/38.
I modified the tests to ensure that the output type is not going to change in the future.
JFTR, Products.PortalTransforms 3.1.5 has been released and it solves the issue:
thanks, @alert, you rock!