Are you sure this transformation is done by "diazo"?
I have seen a HTML -> XHTML transformation performed in "plone.protect" (in Plone 4). As "plone.protect" is integrated in Plone 5, it might well be responsible for the transformation in your case as well.
I would say yes. When no diazo theme is active, the rendered HTML is correct. As soon as a theme is enabled, it is transformed (tested on a Plone 5.1a2 site).
What happens if you try to access the URL appending ?diazo.off=1 or using 127.0.0.1:PORT?
For diazo.off=1 to work, the instance should be in foreground mode.
This should skip the diazo transform, suggesting that some modification is made even if in your rules.xml you have that notheme rule.
You may be interested in disabling diazo theme also by calling: self.request.response.setHeader('X-Theme-Disabled', '1') .
I suppose that the reason is similar as for "plone.protect": one needs to transform the result page and the transformation is easier when at least the target is always XML even if the input has been HTML.
The fun thing is I can't reproduce this error now on my development machine (Mac OS). I have the same packages installed, all the same versions.
The site where I experienced the problem is running on Ubuntu 16.04, using a customization of the ansible playbook for Plone. I'll try the same buildout I have on my local machine on the remote server in the next days to see if there is a difference. Maybe a system package in a different version causing a different transformation, as @dieter already mentioned.
I'm looking for <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />, did you find any hint it is Diazo? If I do ?diazo.off=1, effectively disappear. Should we add this rule to rules.xml?