Details
-
Task
-
Resolution: Unresolved
-
Neutral
-
None
-
None
Description
The self tests right now run always against the latest snapshot of magnolia. Often times, self tests start failing with newer snapshots. In order to disambiguate the root cause of the failures, consider parameterizing test runs against older release and the newer snapshot on Jenkins.
Implementation notes
This would be easy to do using Jenkins' matrix directive: https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/
If the pipeline was able to automatically resolve the Magnolia bundle version before the current SNAPSHOT, it could create a simple matrix and run tests on both bundles. This would apply to all jobs that use the moduleUiTestsPipeline, but that wouldn't be a problem?
this would make builds way slower, find out when to run it (PR builds?) and not (canonical branches).