[MGNLTEST-55] Quarantined tests are executed on jenkins but not locally Created: 03/Jun/20  Updated: 07/Jul/20  Resolved: 07/Jul/20

Status: Closed
Project: Magnolia Test Framework
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Neutral
Reporter: Christoph Meier Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: QA&Testing, fwk-infra-structure
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled
Epic Link: core-TF-features-bugs-improvements

 Description   

NEW description

The old description seems to be wrong.

Jenkins does still what it should. 
Quarantined tests are run - but a failure of a quarantined test will not lead to a failed build.
Hence, the ticket is rather obsolete.
There will be a (kind of) follow-up ticket about what to do with quarantined tests, even more generally how we can phase out JUnit-4-based UI-tests.

 



OLD description

Methods annotated with info.magnolia.integrationtests.rules.Quarantine should not be executed per default, according to javadoc of info.magnolia.integrationtests.rules.QuarantineRule.

But atm "quarantined methods" are executed on Jenkins. (See slack discussion.)
But locally - using the docker-hybrid-setup - the quarantined methods are not executed (as expected).

We figured this out in the context of MGNLCE-232
As a "quick shot", in MGNLCE-232 methods annotated with Quarantine are now annotated with org.junit.Ignore - tho this was done before we realized the unexpected behavior on Jenkins.

 


 Note that this annotation and the corresponding rule work on Junit4-based tests only! 
Hence we prbly should phase it out sooner or later.


Acceptance criteria

Step 1

  • Make sure Quarantine works on Jenkins again as expected
  • Deprecate info.magnolia.integrationtests.rules.Quarantine

 

Step 2

Completely get rid of the Quarantine annotation.
On "quarantined methods", use org.junit.Ignore instead, or fix the method, or get rid of the method

Step-2 could be part of a follow-up

 

 



 Comments   
Comment by Christoph Meier [ 03/Jun/20 ]

tbh, I dislike the Quarantine annotation, seems just like postponing a problem.
I bet that we never had fixed/changed a quarantined test in order to remove the annotation to re-enable it.

Comment by Christoph Meier [ 07/Jul/20 ]

While on jenkins everything works as expected, locally the tests are not run.
However, I do not see this as an issue.

Note that we also are on the way phasing out the JUnit-4 based UI-tests, see MGNLTEST-32.

I close this. No need to investigate here atm.

Generated at Mon Feb 12 07:45:07 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.