I just installed Plone 5.0.6 using the Vagrant installer.
After the installation, the first thing I do is to ssh the box, go to the Plone buildout dir and run
bin/test
The output shows 2 failures and 2 errors from plone.reload.tests.test_code.TestTimes and plone.reload.tests.test_code.TestSearch.
Is it expected?
The goal here is to be able to test the whole Plone install if we want to upgrade any third-party add-on.
For example, we might use eea.facetednavigation as a requirement for our add-on. Our add-on was developed using eea.facetednavigation 9.2. Now that 10.0 was released, we would like to pin it to 10.0 (in setup.py or buildout.cfg), but we thought about running the tests for all add-ons to see if the upgrade in that specific one would break anything.
Does that make sense? If so, what is the recommended way to do it?
Unfortunately there are isolation issues between the tests for some of the packages that make up Plone (i.e. some tests fail to tear down things they set up, so it changes the environment for other tests). The "official" Plone test configuration is in buildout.coredev, and uses plone.recipe.alltests to run the tests in specific groupings that are known to work.
Also, each package's tests typically use a fixture that only installs that particular add-on; in other words tests for an add-on are not really designed to test that add-on's interaction with other add-ons. If you want to do that, write your own test fixture that installs the addons you want to test together, and your own tests that test their interaction, and then run the tests just for your package using bin/test -s package.name
Nothing can replace manual testing as part of an update or migration process. You can not and you will not rely on unit testing only. Unit tests are mandatory tool for quality assurance but not a sufficient one.