Where does url-quoting of links happen?

pysailor · June 10, 2018, 5:50pm

Hi,
I have markup like this in a PageTemplate:

<a href="solutions-#{1}">XX</a>

(I need this kind of syntax, since it is used by pat-clone (http://patternslib.com/clone).)

But what gets rendered in the browser (-> "view source") is this:

<a href="solutions-#%7B1%7D">XX</a>

That means, the curly brackets got percent-encoded / url-encoded.

This kind of replacement only seems to happen with strings that are part of an href. If I place the same string into an attribute like id or as text into any tag, the syntax is preserved.

I would love to get a pointer on where in the code of Plone this treatment of links is happening, so that I can work on a solution for my problem.

Thanks!
Wolfgang

jaroel · June 10, 2018, 6:48pm

Could it be lxml?
Try poking around in https://github.com/plone/plone.app.theming/blob/master/src/plone/app/theming/transform.py#L138 to see if that's true

mgraf · June 11, 2018, 11:44am

Hi,
which addons are installed?
I cant reproduce your issue on fresh plone5.1

Maybe a chameleon "feature"/bug

espenmn · June 11, 2018, 2:42pm

Not really related to the question (and probably a bad idea), but could it be possible to make a workaround by using tal:attributes if it has something to do with chameleon ?

pysailor · June 12, 2018, 10:33am

Thanks for all the hints so far. I can confirm, on a standard Plone5 (5.1) installation, this is not a problem.

My site is somewhat special since it uses Chameleon and NuPlone, and due to the latter (and the use of Euphorie), I have only a very reduced set of Plone management screens (ZMI is fully functional though).

I saw that the the transform in theming is not used, and neither did I see Chameleon mangle it (in the return value of render() in https://github.com/malthe/chameleon/blob/master/src/chameleon/zpt/template.py#L297 the curly brackets are still intact.)
Completely deactivating JavaScript also makes no difference.

EDIT: and setting the href with tal:attributes also makes no difference.

I will keep digging... and post my findings here.

pysailor · June 12, 2018, 12:28pm

Update: @jaroel was right, it happens via lxml. But in a different spot.

Details: In ZPublisher.Publish.publish(), there's a call to notify(PubBeforeCommit(request)), which leads to plone/transformchain/zpublisher.py(86)applyTransformOnSuccess(). One of the handlers that act upon that event is plone.protect.auto.ProtectTransform.

So in the end, we land here:

github.com

plone/plone.protect/blob/master/plone/protect/auto.py#L125




    if isinstance(result, list) and len(result) == 1:
        # do not parse empty strings to omit warning log message
        if not result[0].strip():
            return None
    try:
        result = getHTMLSerializer(
            result, pretty_print=False, encoding=encoding)
        # We are going to force html output here always as XHTML
        # output does odd character encodings
        result.serializer = html.tostring
        return result
    except (AttributeError, TypeError, etree.ParseError):
        # XXX handle something special?
        logger.warn('error parsing dom, failure to add csrf '
                    'token to response for url %s' % self.request.URL)
        return None


def transformBytes(self, result, encoding):
    result = result.decode(encoding, 'ignore')
    return self.transformIterable([result], encoding)

Up until here, the result contains the href with the curly brackets intact. But when the html.tostring serializer gets applied, the curly brackets get replaced in the href when result is rendered.

The fact that I even land in this part of the code (versus stock Plone 5.1), is because the handler before that, "plone.app.theming.transform" does nothing. Therefore, result is still in the original state that gets passed in to plone.transformchain.transformer, that means a list of strings.
In stock Plone 5.1, "plone.app.theming.transform" does indeed return output, so by the time the transformer is called in plone.protect, result is an instance of XMLSerializer, and the lxml transform (html.tostring) never happens, because of

github.com

plone/plone.protect/blob/master/plone/protect/auto.py#L102


    self.published = published
    self.request = request


def parseTree(self, result, encoding):
    # if it's a redirect, the result shall not be transformed
    request = self.request


    if request.response.status in (301, 302):
        return None


    if isinstance(result, XMLSerializer):
        return result


    # hhmmm, this is kind of taken right out of plone.app.theming
    # maybe this logic(parsing dom) should be someone central?
    contentType = self.request.response.getHeader('Content-Type')
    if contentType is None or not contentType.startswith('text/html'):
        return None


    contentEncoding = self.request.response.getHeader('Content-Encoding')
    if contentEncoding and contentEncoding in ('zip', 'deflate',

So finally, I think I have an idea what to do in my custom site to circumvent that problem.

Edit:
And of course this leads me back to Roel's suggestion about

github.com

plone/plone.app.theming/blob/master/src/plone/app/theming/transform.py#L138


        # this transform for files and images
        pass
    return self.transformIterable([result], encoding)


def transformString(self, result, encoding):
    return self.transformIterable([result], encoding)


def transformUnicode(self, result, encoding):
    return self.transformIterable([result], encoding)


def transformIterable(self, result, encoding):
    """Apply the transform if required
    """
    # Obtain settings. Do nothing if not found
    policy = theming_policy(self.request)
    settings = policy.getSettings()
    if settings is None:
        return None
    if not policy.isThemeEnabled():
        return None
    result = self.parseTree(result)

which in my case is not used, because the theme itself is not enabled...

mgraf · June 12, 2018, 1:46pm

Plone5 always transform your content, except it has no styling

mgraf · June 12, 2018, 2:09pm

Hi,
maybe plone.tile

github.com

plone/plone.tiles/blob/master/plone/tiles/tile.py#L135


    def transformBytes(self, result, encoding):
        return self.transform(result, encoding)


    def transformUnicode(self, result, encoding):
        return self.transform(result, encoding)


    def transformIterable(self, result, encoding):
        return self.transform(result, encoding)




class TileProtectTransform(object):
    """Replacement transform for plone.protect's ProtectTransform, to drop
    X-Tile-Url-header from unauthorized responses and disable the default
    ProtectTransform for authorized responses (to avoid causing issues
    like extra protect.js-injections for tile editors)
    """


    order = 9000


    def __init__(self, published, request):
        self.published = published

or
NuPlone's own cleaning....