To achieve reproducible test results on both development and CI environments, we do lock/pin all test dependencies inside test-requirements.txt
. This is used as a pip constraints file inside various tox environments so it will only force pip to directly pick specific versions instead of trying to resolve them at runtime.
The file is automatically updated when you run tox -e deps
, an action that we will do whenever we find a need for a dependency bump. The real source files for dependencies are test-requirements.in
and this is the place where we define compatibility ranges.
Many applications are using the same approach for locking their entire set of dependencies, not only the test ones. Still for our particular use-case, we get most benefits from doing this only for test ones.
There is no tox-deps
zuul job because it does not make sense to run it on CI. With 20+ test dependencies on average we have at least one dependency updated each day. We don't really need to run bleeding edge and be the first line of defense for all these libraries. Still, whenever we want we can update them with a single command, making testing much easier.
These test constraints files are not shared across repositories for good reasons, they are specific to each repo and affect testing of a single repository. If we would attempt to centralize them we will create a case were they are almost impossible to update, due to the huge amount of testing needed. There is nothing wrong using outdated test dependencies, especially when is so easy to control their upgrades.
References: