Best practise for buildout in CI

Development Testing and Automation
1

Hey,
I'm just getting used to Travis & friends, however I've noticed a couple of practises that I'm not sure about, can anyone explain why these are done? My thinking is that CI testing should focus on the usecase where a new user picks up the product with a vanilla version of Plone (4.3.7 or 5.0), that is the most easily reproducible scenario after all!, However in the below scenarios that doesn't seem to be the case. Maybe I'm thinking wrong about this?

  1. checking out multiple source packages ([sources] section of buildout) in travis, rather than released versions. A new user won't be expecting to check out source packages of dependent products, so IMO while this might be acceptable in the short term, to get past a non-released change in another package, it shouldn't be "the norm".
  2. buildout -N coupled with travis caching. For someone building plone for the first time (or in a fresh virtualenv), it DOESN'T matter whether you are using newest mode (-n) or not (-N) in buildout, the results will be the same. However if you want to replicate that scenario when you are using a cache in travis, then surely you should use newest mode? If newly released eggs then cause test failures, then pins should be set in the CI buildout AND documented in the README.

I've seen these practises in a few places (including 2. in bobtemplates.plone) I thought I'd mention it here to get a wide opinion of whether I'm thinking right.... (and perhaps embarrass myself a bit)

2

This is/was because buildout without using caches is dead slow. I believe the -N is an old artifact from the time when we first downloaded the plone unified installer and unpacked it. Then we had a reproducable build.

As for the use cases, there are several use cases so it makes sense to run multiple builds.
Using caches might harm reproducability but in the cases where no errors are masked and real errors are found the time to red is much shorter. Bugs just don't get fixed if you only get the response 30 minutes later.

I have a gitlab runner which tests the current version and then in another runner upgrades all versions to the highest patch versions. This way I find out asap if somebody released a patch version with breaking changes so that I can go out and break some necks.

3

Thanks for the reply. But then with the advent(*) of Travis caches we shouldn't need -N because there should only be 1 or 2 new packages to download since the last time it ran, right?

* seasonal pun unintended but nice

4

this is what I use, with comments:

language: python
sudo: false

sudo: false is important to run on container based infrastructure.

cache:
  pip: true
  directories:
    - $HOME/buildout-cache

pip caching is supported by default, but does not help much with buildout, so we add a directory to the cache to be used by buildout:

python:
  - 2.7
env:
  - PLONE_VERSION=4.2.x QA=true
  - PLONE_VERSION=4.3.x QA=true
  - PLONE_VERSION=4.3.x-LP QA=true
  - PLONE_VERSION=5.0.x QA=false
matrix:
  allow_failures:
    - env: PLONE_VERSION=5.0.x
before_install:
  - mkdir -p $HOME/buildout-cache/{eggs,downloads}
  - mkdir $HOME/.buildout
  - echo "[buildout]" > $HOME/.buildout/default.cfg
  - echo "download-cache = $HOME/buildout-cache/downloads" >> $HOME/.buildout/default.cfg
  - echo "eggs-directory = $HOME/buildout-cache/eggs" >> $HOME/.buildout/default.cfg

here we set up global caching for buildout, so we do not need fight with it in out buildout.cfg's

hint: if you build packages with zc.recipe.cmmi in your buildout, use shared = true option, so the build is stored in the buildouts download cache and can be reused next time.

  - virtualenv .
  - bin/pip install --upgrade pip setuptools zc.buildout coveralls
  - export DISPLAY=:99.0
  - sh -e /etc/init.d/xvfb start
install:
  - sed -ie "s#plone-4.3.x.cfg#plone-$PLONE_VERSION.cfg#" travis.cfg
  - bin/buildout -N -t 3 -c travis.cfg

With -N the pypi is not asked for every packages updates (version-wise). So w/o it buildout sends a request to pypi for every single package, afaik it does not matter if already pinned and available in the requested version local (correct me if I'am wrong).

So this is needed and still makes sense.

script:
  - bin/code-analysis
  - bin/test --all
after_success:
  - bin/createcoverage
  - bin/python -m coverage.pickle2json
  - bin/coveralls

After all, bobtemplates.plone generates all this already.

A good example is https://github.com/collective/pas.plugins.ldap - runs really fast (~1min) and this one builds a complete openldap server too.

5

It makes sense to checkout multiple source packages when you work on a package depending closely on other packages.
In my case for instance, the rapido.plone Travis buildout checks out rapido.souper and rapido.core because when I implement a new feature, it might imply changes in those 3 packages at the same time.

I guess we can find similar cases (I do not know if that's the case for plone.tiles + plone.app.tiles + plone.app.standardtiles but it could be a valid case too)

6

if you need to checkout packages: setting sources-dir = ${buildout:download-cache} in travis.cfg [buildout] section should speed up the git parts a lot.

7

Thanks Jens, that's really useful.
So -N is for speed then. But with Travis's caching the cache will not get updated unless new pins are added, right?

8

Thanks @ebrehault. I know what you mean, but I think that those checkouts should be in "buildout.cfg" not "travis.cfg" (assuming travis.cfg doesn't extend buildout.cfg), so that local testing is upto date with dependencies. But IMO leaving source packages checked out in travis.cfg would make the CI tests fragile. I recently tried to fix a travis failure that was caused by changes in another package - not released changes, but changes on a 2.x branch that travis had checked out. That's one of the things that made me think about this.

9

Correct, with -N and if the cache has a package already and its version is sufficient according to pinning information then no further online checks are executed against pypi.

You may get problems if you rely on automatic updates of new pypi-releases of non-pinned packages. But this is fragile anyway and I would not recommend that.

10

I prefer to have a versions.cfg where all versions and also checkouts are in.

11

yes right, in such a case, checkouts are in buildout.cfg

12

So travis should not be fragile (unless travis.cfg extends buildout.cfg)

13

I would actually use git-clone-depth = 1 in buildout section of travis.cfg, that will only checkout the very last commit from the configured branch. If you rely on fetching something with git, that's the fastest way.

Since mr.developer 1.33 it should work flawlessly.

14

In my opinion travis should test on master only against released packages. On branches testing within a bunch of unreleased stuff is ok, but a branch is usually not ready for merge unless its dependencies are released.

15

Thanks all for the replies - I think I am of a different approach to testing, instead of:

  • fast Travis tests
    • easier for developers when they are working on it
  • cautious on dependency updates
    • fails less often
    • hard to fix when it fails
    • not so close to end user (integrator) experience

I lean towards:

  • accept dependency updates
    • fail fast
    • hopefully easier to fix, because you encounter only one(ish) issue at a time
    • closer to end user experience (integrators)
  • slow Travis tests
    • a bit more annoying when developing

My main point is that the green "Travis tests passing" Status Image should be aimed at end users wanting to have (relative) confidence that a product's current release is safe to use with:

  • released versions of dependencies
  • and versions that they are likely to be using

more than for product developers to feel happy at the leading edge source works with checkouts from here and there, and versions of other dependencies that wont be fetched from a fresh buildout

Though there is probably room for both with a table like the following

|        | latest release | source version |
| master | pass           | fail           | 
| 1.x    | pass           | pass           |

Unfortunately that's probably not possible to automatically reference the latest release with Travis status images

  • maybe Travis status images are more for developers after all!

Oh well, just thinking aloud then :slight_smile:

16

Valid point. On pypi it should refer to the current release, on github to the current branch.

Are release tags tested in travis?

If yes we could enhance i.e. zest.releaser prerelease to change the badge link to the tag (and postrelease to change back)

btw., the same is valid for coveralls and similar badges

cc @mauritsvanrees

17

Yes they are, e.g.

https://travis-ci.org/plone/plone.app.multilingual.svg?branch=2.0.1

18

Looking at the Travis badges for zest.releaser itself, the badge for the master branch is: https://api.travis-ci.org/zestsoftware/zest.releaser.png?branch=master
The Travis badge for the latest release (currently 6.4) does not exist: https://api.travis-ci.org/zestsoftware/zest.releaser.png?branch=6.4
That is also what I expect, because a tag is not a commit (well, it points to a commit). And if you have setup zest.releaser with some extra config in ~/.pypirc you likely have [ci skip] as extra in the commit message, which tells Travis not to run any tests because the prerelease and postrelease commits are uninteresting and are safe to skip.

If you do have a badge for a tag, the danger is that this has a failed test that only failed because of a temporary network error or because there is a flaky test that sometimes fails and sometimes not, which you have not tracked down yet, and which has nothing to do with the changes done for the release in question.

Referencing the master branch badge from PyPI may make you more likely to fix Travis failures, because it looks bad on all releases. :slight_smile:

Also, a badge for a tag only indicates that the tests passed at the moment the release was made. Maybe that was a year ago and this release no longer works with a new minor release of Plone. Said differently: there is value in a badge for a tag, but it is limited.

19

The Travis badge for 3.56 does exist https://travis-ci.org/zestsoftware/zest.releaser.svg?branch=3.56 I don't know why 6.4 doesn't though....

Click on a travis badge to see the branches on offer

20

If (following your argument) the release was a year ago, then a badge for master could be quite different to the results a user would get using the released package (no matter which version of Plone) :slight_smile: