Plone 5.2.3 and Email Encoding Problem

Plone Support Add-on Development
1

On a simple testinstallation without any addons the plain testmail is broken, also the email for password reset and registration

Plone 5.2.3 (5210)
CMF 2.4.8
Zope 4.5.3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
PIL 6.2.2 (Pillow)
WSGI: An
Server: waitress 1.4.4

zope.sendmail 5.1
Products.Mailhost 4.10

Hi,

This is a test message sent from the Plone 'Mail settings' control panel. Y=ur receipt of this message (at the address specified in the Site 'From' ad=ress field) indicates that your e-mail server is working!

Have a nice day.

Love,

Plone

Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzer=ame lautet: john.doe1@local.test. Bitte aktivieren Sie d=esen, indem Sie zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda24=6ebe6da184b4f4c311?userid=john.doe1@local.test gehen. =ktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

plone

Have anyone this behavior?

Update:
The same with custom view and plone.api.send_email

class ApplicationMailView(DefaultView):

    def __call__(self):

        TEXT = """
Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername lautet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda24=6ebe6da184b4f4c311?userid=john.doe1@local.test gehen. Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

Plone """

        api.portal.send_email(
            recipient=....,
            sender=....,
            subject="Info Debug",
            body=TEXT,
        )
        return "ok"

The Mail:

Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername la=tet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie zu http:=/127.0.0.1:8080/Plone/passwordreset/495e356abda24=6ebe6da184b4f4c311?use=id=john.doe1@local.test gehen. Aktivieren Sie Ihren Zu=ang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

Plone

Update:
But the following snippet do the job:

from email.charset import Charset
from email.encoders import encode_base64
from email.header import Header
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

class ApplicationMailView(DefaultView):

    def __call__(self):

        TEXT = """
Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername lautet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda2476ebe6da184b4f4c311?userid=john.doe1@local.test gehen. Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

Plone """        

        ch = Charset("utf-8")
        ch.body_encoding = "8bit"
        part = MIMEText("")
        part.set_charset(ch)
        part.set_payload(TEXT.encode("utf-8"))
        part.replace_header("Content-Transfer-Encoding", "8bit")

        msg = MIMEMultipart("mixed")
        msg["To"] = ....

        msg["Subject"] = "Info Debug 2"
        msg.attach(part)

        api.portal.send_email(
            recipient=msg["To"], subject=msg["Subject"], body=msg
        )

The Mail:

Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername lautet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda24=6ebe6da184b4f4c311?userid=john.doe1@local.test gehen. Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

Plone
1 Like
2

Yes, I've seen this as well . There have been some fixes for this in Zope (zope.sendmail and another package I can't remember) and are unicode/bytes/python3 related.

Plone 5.2.3 uses this updated Zope release, if you can test Plone 5.2.3 (always first on a separate instance) these issues might be resolved.

edit/additional: We had this and other mail issues on a few Plone instances of a setup, but not all of them with the same codebase. A different solution was change the relaying mailserver in the mail connector. Especially Office365 has been causing us all kinds of grief in the last 4-5 months as a receiving mailserver. We have switched to only relaying e-mail to a localhost spooler. Problems like missing bodies of e-mails and other mailformed or not arriving e-mails were suddenly fixed.

3

This is a 5.2.3 Installation :wink: I think this is a issue and blocker, because the password reset link is broken by default, this is a core function which should not be broken.

4

I used the following code to verify whether Products.MailHost/zope.sendmail is to blame for this behavior:

from Products.MailHost.MailHost import MailHost
from transaction import commit

mh = MailHost()
body="""
Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername lautet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda246ebe6da184b4f4c311?userid=john.doe1@local.test gehen. Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.

Mit freundlichen Grüßen
--

Plone """
bbody = body.encode("utf-8")
mh.send(bbody, "dieter@localhost", "dieter@localhost", subject="Test", charset="utf-8", immediate=True)
mh.send(bbody, "dieter@localhost", "dieter@localhost", subject="Test", charset="utf-8", immediate=False); commit()

Both messages were correctly delivered.

Could you try a variant of the code above (you need at least modify the email addresses; maybe, you must also configure the mh object differently) in your environment.

Have you set up your mailhost to queue messages? I did not try this (because it is a bit difficult to set up); I will do this if necessary.

5

The strange = is at a place where the "quoted printable" encoding has inserted an artificial line break (called "soft line break"), i.e. =\n. Apparently, soft line breaks are not handled correctly in your environment. This might be a consequence of Python's "quoted printable" using \n rather than \r\n as line endings which does not conform to "RFC 2045 - Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies". This would be a Python bug.

Most email processors handle both \r\n as well as \n as valid line terminators. Those would likely correctly process soft line breaks introduced by Python's quoted printable.

I have performed my test in a Linux environment. There, the soft line breaks have been correctly recognized and processed.

6

I have created "https://github.com/zopefoundation/Products.MailHost/issues/35". Because this would be primarily a Python problem, I am not sure whether Products.MailHost should work around it. In my view, a fix in Python would be more appropriate.

1 Like
7

This is broken:

from plone import api
from transaction import commit
from zope.component.hooks import setSite

setSite(app["Plone"])
portal = api.portal.get()
mh = portal["MailHost"]
body = """
Willkommen John Doe, Ihr Benutzerzugang wurde erstellt. Ihr Benutzername lautet: john.doe1@local.test. Bitte aktivieren Sie diesen, indem Sie  zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda246ebe6da184b4f4c311?userid=john.doe1@local.test gehen. Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.
Mit freundlichen Grüßen
--
Plone """
bbody = body.encode("utf-8")
mh.send(
    bbody,
    "...",
    "...",
    subject="Test",
    charset="utf-8",
    immediate=True,
)
commit()

This is ok:

from plone import api
from transaction import commit
from zope.component.hooks import setSite

setSite(app["Plone"])
portal = api.portal.get()
mh = portal["MailHost"]
body = """
Willkommen John Doe, Ihr Benutzerzugang wurde erstellt.
Ihr Benutzername lautet: john.doe1@local.test.
Bitte aktivieren Sie diesen, indem Sie 
zu http://127.0.0.1:8080/Plone/passwordreset/495e356abda246ebe6da184b4f4c311?userid=john.doe1@local.test gehen. 
Aktivieren Sie Ihren Zugang bitte vor dem 09.12.2020 12:40.
Mit freundlichen Grüßen
--
Plone """
bbody = body.encode("utf-8")
mh.send(
    bbody,
    "...",
    "...",
    subject="Test",
    charset="utf-8",
    immediate=True,
)
commit()

The length of the line is the problem.

8

Only the follwing snippet do the job correctly

LONG_MULTI_LINE = """\
From: {}
To: {}
Content-Type: text/plain
Precedence: bulk

Dear Passengers,

Curabitur aliquet quam id dui posuere blandit. Vivamus magna justo, lacinia eget consectetur sed, convallis at tellus. Vivamus magna justo, lacinia eget consectetur sed, convallis at tellus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. http://127.0.0.1:8080/test/test/portal_setup/blub Curabitur non nulla sit amet nisl tempus convallis quis ac lectus. Mauris blandit aliquet elit, eget tincidunt nibh pulvinar a.

A little Umlaut: "üüß"

Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Donec velit neque, auctor sit amet aliquam vel, ullamcorper sit amet ligula. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec rutrum congue leo eget malesuada.

thanks, your plone
"""

from_address = api.portal.get_registry_record('plone.email_from_address')
from_name = api.portal.get_registry_record('plone.email_from_name')
sender = formataddr((from_name, from_address))
if parseaddr(sender)[1] != from_address:
    sender = from_address

# EmailMessage Object with the SMTP Policy doesn't destroy the Multiline
msg = message_from_string(LONG_MULTI_LINE.format(sender, to), policy=policy.SMTP)
msg['Subject'] = Header(subject, "utf-8").encode()

# Add Attachments from a namedblobfile
for mailattachment in attachments:
    msg.add_attachment(mailattachment.data, maintype='application', subtype='octet-stream', filename=attachment.filename)    

# Send via MailHost
host = api.portal.get_tool('MailHost')
host.send(
    msg,
    "john.doe.recipient@local.test",
    "john.doe.sender@local.test",
    immediate=False,
    charset='utf-8',
    msg_type='text/plain'
)
1 Like
9

Products.MailHost is updated. Use standard conforming \r\n line endings, this prevent the malicious encoding, mostly happen by sending mails via Exchange.

1 Like
10

@1letter thanks for your work on this!