I am current involved with a large Plone 5 project where the customer has a large rules.xml and some large templates. Rendering a page with Diazo takes 20-30 seconds unless you set DIAZO_ALWAYS_CACHE_RULES=1.
Is there any tool for profiling a Diazo theme for figuring out the execution time of each individual rule? Something like PTProfiler? Right now trial & error, binary search etc. seems to be the only appropriate approach for digging into performance issues...but this approach obviously sucks...I need clear numbers...anyone else encountered such huge transformation times with Diazo?
Second: I used the diazocompiler to build the XSLT and analysed it by reviewing the output. Its usually far from optimized. A big speed-up (in our case) was to skip using CSS selectors and use as specific as possible XPath rules.
It's a bit offtopic regarding the optimization aspect, but I ought to share this thought anyway:
So that's why I thought of another transformation handler, that acts right before Diazo would kick in, and checks if Diazo can be disabled for this request. You can see this gist as proof of concept.
But what I used was a plugin that added a response header with timings including that of theming. That helped narrow which pages diazo makes slow, but not which rules. But pretty much any xpath with // near the start is going to be slow. The more specific the faster.
XPath expressions in general are very fast because the execution engines (lxml here) are highly optimized (also other XSLT/XPath engines like Saxon). In generall google for "css selectors slower than xpath" and find something like
We have made the same experience in a screen scraping projects where CSS selectors are usually slower than XPath...in general I prefer XPath just because XPath is more powerful and more expressive...but also more complex...well..
A CSS selector (translated by cssselect to a not-that-efficient XPath) usually searches through the whole tree. Even selecting ids seems not to be very fast. Compare to this XPath is highly optimized. Using absolute XPath reduces the sub-tree to search in. The longer the content of the page get, the slower it gets. Compared to this a specific selection speeds up the search for the element.
Unfortunately Diazo duplicates the queries into the tree - dependent on the theme - 100s of times. Here it helps to first assign the result to a xslt variable and use it. This is really (!) a speed-up and I would like to see diazo doing this on its own, because doing the very same query in a loop several times is not very clever.
That's interesting - so upgrading the libxml and libxslt helps to get a performance gain of 800x? Incredible.
This is something we may include in the Plone/Diazo install docs. I don't think it makes sense to put these tests in Plone core as we have no control over system libraries installed on the Plone server.
Well, correlation does not imply causation. Maybe something else besides version was also different... When I was using the fast version, I also told buildout to build my own lxml like this:
# lxml should be first in the parts list
recipe = z3c.recipe.staticlxml
egg = lxml
build-libxslt = true
build-libxml2 = true
static-build = true
libxml2-url = http://xmlsoft.org/sources/libxml2-2.9.8.tar.gz
libxslt-url = http://xmlsoft.org/sources/libxslt-1.1.32.tar.gz
Target systems where debian. I discovered it can be faster on my MacOSX where I was able to build a diazorun using Anaconda/2.7.14 ...
Slow setups did not include an lxml part in their buildout.
That looks to me like not building with Cython in the build-time environment as lxml wants it vs. having it there appropriately. If you look at the build output, it'll tell you if this is (or is not) the case.