I share your concern regarding testing, and specially with how many parallel jobs are we able to run.
OTOH if we want to use semantic versioning that's just how it is, see setuptools for a, maybe fully strict semantic versioning.
What probably needs to be discussed at the next FWT meeting together with the release manager would be about the support policy. By releasing Plone 5.1 do we drop support for 4.3? and as soon as 5.2 is out do we drop 5.0 already? Or we only support the last major version (4.3) and one or two minor versions (say 5.2 is out and 5.1 and 5.0 are still supported, but as soon as 5.3 is out 5.0 support is dropped).
And back to the testing point, but more from the development prespective, we need to put somewhere crystal clear, @svx was asking for that as well, which Plone version any branch of any given core package is targeting.
So by now only p.a.vocabularies master branch is 5.1 only as that's the only branch merged of PLIP 1342:
So something like a list on each package (on each branch) the README's top saying:
Compatibility
Plone versions supported on this branch:
- 5.1
- 5.0
- 4.3
Compatibility notes:
- To support version X.Y please use profile YZ
- To support version Z.U please import xxx.zcml file
Could be enough? And keeping that list in sync with setup.py's classifiers would be great, so we could extract more exact numbers from pypi and give some guidance to collective/external packages.
I will PLIP this later today.