Testing add-ons against Plone 6/latest buildout.coredev?

Is there an easy way to test our own Plone 5 add-ons against the current Plone 6/buildout.coredev codebase?

Usually, I would use GitHub - collective/buildout.plonetest: Testing/development buildouts for Plone or test against an available release on dist.plone.org. However, there is nothing like that at the moment.

How to deal with this?

On jenkins.plone.org there are jobs related to testing an add-on against a given current buildout.coredev branch, 6.0 is missing, but should be trivial to add it.

So far there has been not much interest in using them, maybe due to the jobs being unknown for most? :thinking:

:man_facepalming:t4: oh my, there are indeed, jobs for Plone 6.0 :sweat_smile: I just did not had a closer look at it.

One can test against Plone 6.0 together with Python 3.6 to 3.9 :tada:

But that needs to be done manually, right? Or can we integrate and trigger this from e.g. existing GitHub actions? As far as I’ve seen, it hasn’t been used a lot for add-ons: 2 tests for Plone 5.1, 1 for Plone 5.2, 4 for Plone 5.2 on Python 3.7 and 4 for Plone 6.0 on Python 3.7 (from which I started 2 within the last hour).

Integrating tests for an upcoming release is crucial for add-on developers, but I have the feeling that Jenkins adds too much “complexity” if we’re already using GitHub actions.

Since it uses the coredev buildout as base, tests can fail and pass with every commit. It is not reproducible and can deliver different results within minutes. I know it is some time until the first release candidate or alpha is ready, but can we maybe have fixed pre-build versions we can then use in buildout.plonetest?

I just discovered the QA tests: Code analysis #902 [Jenkins]

All the tests pass, but just the latest test contains a 3.62MB file of code-analysis errors. Really? Why is that one not failing?

Jenkins is mandatory to submit a PR to a package you can't write directly, isn't it? Being in development, it is better to test against the devel, it is how you usually find something has broken, even if it is not your commit/PR.

Yes, so far manually unfortunately. This does not mean that it can not be done. I guess that github actions are extendable and one can write their own, right? so writing one that integrates with these jenkins jobs should be doable.

We have mr.roboto that bridges jenkins and github already, so more code there might be another approach, someone has to write the code, do the configuration dance, etc.

I can try to help, but not lead the effort.

One does not need to actually test against current coredev on every single pull request of your add-on, but more sporadically (IMHO), so the manual step might not be that bad anyway :man_shrugging:t4:

1 Like

Because it is mostly informative, we don't want to stop any pull request to be merged because there is legacy/old code that does not follow our QA rules, which might need improvement anyway as well

Ok I see. But it looks like the QA job is not showing up in PRs, so it would be ok to fail.

What I noticed is that some packages fail (like plone.restapi), but the package tests pass (the code-analysis part). So if a package has different settings than the Jenkins QA job (where can I see the settings for that?), how can we ensure that (core) code follows Plone’s QA rules if the QA test passes (fails silently), and the package tests pass but with different settings?

the configuration for all jobs are in jenkins.plone.org/jobs.yml at master · plone/jenkins.plone.org · GitHub and the scripts folder next to it.

That there is some inconsistencies on it, I'm pretty sure, I did it alone and nobody ever noticed nor cared about them, so they are probably +5 years old :sweat_smile:

As I said, I'm all happy to help push them forward, but I can not lead the process, I'm way to busy right now :confused: