Products.PortalTransforms.data.datastream.getData() changes?

Hello everyone, our testing matrix start failing recently for Plone 5.1 builds, can someone help us please?

https://travis-ci.org/collective/collective.cover/jobs/441921182

More information here:

Running collective.cover.testing.collective.cover:Integration tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.340 seconds.
  Set up plone.app.testing.layers.PloneFixture in 9.700 seconds.
  Set up collective.cover.testing.Fixture in 4.652 seconds.
  Set up collective.cover.testing.collective.cover:Integration in 0.000 seconds.
  Running:
    7/8 (87.5%)> /home/rodfersou/.projects/collective.cover.p5/src/collective/cover/tiles/richtext.py(103)SearchableText()
    102 
--> 103         searchable_text = six.text_type(data.getData(), 'utf-8')
    104         return searchable_text.strip()  # remove leading and trailing spaces

ipdb> data
<Products.PortalTransforms.data.datastream object at 0x7ff58c4dfb30>
ipdb> data.getData()
u' My document text... '
ipdb> six.text_type(data.getData(), 'utf-8')
*** TypeError: decoding Unicode is not supported

@hvelarde it is weird, the last change in this module was 7 months ago, all other changes was years ago.

it should be something in the last release 3.1.4 (2018-09-23) https://github.com/plone/Products.PortalTransforms/compare/3.1.3...3.1.4

The quick fix is to change collective.cover's code here:

searchable_text = data.getData()
if isinstance(searchable_text, six.binary_type):
    searchable_text = searchable_text.decode('utf-8')

or

from Products.CMFPlone.utils import safe_unicode
return safe_unicode(data.getData())

Unluckily the output of convertTo has changed in a non retrocompatible way in a minor release and nobody noticed this.

Apparently convertTo should return a str object in both Python 2 and 3.

That is very bad and I am sorry it happened but it is good that now we know.
Not really sure how to fix this easily and in a fast way upstream.

2 Likes

fortunately we have pretty good tests in collective.cover that had lead to discover early a couple of issues on the core in the past :slight_smile:.

IMO, Products.PortalTransforms should be fixed before releasing Plone 5.2a1.

1 Like

@rodfersou are you able to create a pull request with a failing test in https://github.com/plone/Products.PortalTransforms ? ... since core tests were passing we might have missed something there.

I'd recommend following best practice for transforming RichTextValue here https://github.com/plone/plone.app.contenttypes/blob/master/plone/app/contenttypes/indexers.py#L51 ...

@petschki

But this method was not changed in the last release.

the convertTo method returns an object implementing IDataStream ... so in detail, the text/plain transformer is a subclass of Products.PortalTransforms.libtransforms.retransform.retransform and this returns always a six.text_type since the last change of @pbauer (https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#diff-667edf09d266049d938e4a0bda2a20b1R29) ... maybe the reason why your transform doesn't work anymore.

@pigeonflight I think this is the change you're fighting with in your thread.

So maybe converge here as the root cause is now known.

Yes!
My quick fix was to pin
Products.PortalTransforms = 3.1.2

Well spotted @petschki!
In this case the fix might be https://github.com/plone/Products.PortalTransforms/commit/5e297b66e04cd1771d40e40f30cb81a5aa3509ea#r30973076If somebody has time to open a PR he is welcome.
Otherwise I will do one later/tomorrow

1 Like

Here it is https://github.com/plone/Products.PortalTransforms/pull/38.
I modified the tests to ensure that the output type is not going to change in the future.

2 Likes

JFTR, Products.PortalTransforms 3.1.5 has been released and it solves the issue:

thanks, @alert, you rock!

3 Likes