Yesterday I stumbled over a failure in the plonesocial suite, which is part of ploneintranet.
the plonesocial suite contains robot tests that test a bunch of plonesocial packages to make sure that they work well together.
The test failure arrived when one of the packages started to define a tile. Plone.tile provides a meta.zcml defining tile zcml configuration.
In the package itself, the layer was changed to load meta.zcml. If I would have used the package on a regular plone site, z3c.autoinclude would have loaded that file automatically. But the plonesocial suite uses plone.app.testing which deactivates z3c.autoinclude. Because of this, the meta.zcml file from plone.tiles is not loaded and the tile configuration fails in the tests of plonesocial.suite.
I see four ways to solve this issue, they all have disadvantages:
- Load the meta.zcml in the layer of the suite.
Problem: The more packages provide custom zcml directives that are not part of plone, the bigger my layer code gets. If you wrote PloneTestCase tests, you might remember how much layer code was necessary.
- Create a layer which uses all subpackages layers as a base
Problem: Is that even possible? I think it is a pretty bad idea and I don't even know where to start with the why...
- Load the meta.zcml in the configure.zcml of the package that wants to use the tile.
Problem: meta.zcml files should all be loaded already when configure.zcml files are parsed. Else we might get problems when ordering of code changes.
- Create a meta.zcml in the package that uses plone.tiles and load the meta.zcml from plone.tiles there.
Problem: That is not explicit, that is repetition. We already declare our dependency on plone.tiles in our setup.py
Did I miss a better solution?
Would it make sense to look into improving z3c.autoinclude so that it could be utilitized in plone.app.testing without loading too much?
Don't know why z3c.autoinclude is disabled in tests?
Doesn't this give us false security if the tests don't run on the same code/runtime as our customers?
Or in other words: what would happen if you enable z3c.autoinclude in your testrunner?
I think there was a problem with test isolation. I believe Jens Klein knows best. I hope he might chime in.
So I'am not Timo "King Testing II" Stollenwerk, who for sure knows the backgrounds better than I do.
But some thoughts about it:
First things first: The Plone dependency graph is the major source of our pain. Because of it we cant just provide an isolated test with only the packages available needed for the test.
Loading all zcml with every test is slow (=z3c.autoinclude enabled).
Loading all zcml with every test may have bad side effects which are good to cover in integration tests, but bad in unit testing scenarios.
So what to do?
In my opinion a configure.zcml which uses a directive from a third party package must load this explicit before the directive is used, at best at the very top of the file. I dont see any problems with ordering here. We may want add a check for this to code-analysis to find any problems early.
In my opinion all Plone core packages must not depend on any autoinclude and must load all dependend zcml on their own. Also this would show dependency indirections in zcml (yes, there are such) better.
How much is a z3c.autoinclude slow compared with no z3c.autoinclude at all? Can we test it, creating more travis jobs, one having autoinclude and another one not having it, so we can profile how slow it is?
Which kind of bad side effects are you having which are bad in unit testing scenarios?
Guys, we believe that explicity is better than implicity: but we don't agree that, for running in production we have the autoinclude magic, but when using testing machinery we need to be explicit. This is really confusing for beginners, since when running
bin/instance it behaves in a certain way that isn't necessarily the same as running
A similar discussion: https://github.com/plone/plone.app.testing/issues/27 - "How to debug which zcml's are loaded when running bin/instance and which ones with bin/test"
A while ago I started to write superfast tests with pytest. IIRC removing z3c.autoinclude saved me 20 seconds from test initalization, it was the biggest time hog.