Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JsonProviderTest fails if run after other Junits in a test suite #376

Closed
tabishop opened this issue Apr 28, 2022 · 5 comments · Fixed by #380
Closed

JsonProviderTest fails if run after other Junits in a test suite #376

tabishop opened this issue Apr 28, 2022 · 5 comments · Fixed by #380
Milestone

Comments

@tabishop
Copy link

ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.java will fail if run in a Junit test suite if it executes after any other TCK test that calls JsonProvider.provider(). Junits don't guarantee order of execution of tests within a test suite by default.

This is because the JsonProvider loads the provider as a static final the first time it is requested. The JsonProviderTest.java class attempts to set a system property to a dummy provider class, call JsonProvider.provider(), and then tries to validate that the provider returned is the dummy provider class. However, if any other TCK test that calls JsonProvider.provider() is run before this test, it will fail because the returned provider will be of the class initially used, not the dummy provider.

To recreate, create a Junit Test suite with the JsonProviderTest and any other test that calls JsonProvider.provider() and runs before Json ProviderTest.

This causes problems when trying to develop test automation for Open Liberty support of JSONP 2.1

@KyleAure
Copy link

Here is the stack trace of this failure:

org.opentest4j.AssertionFailedError: expected: <ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.DummyJsonProvider> but was: <org.eclipse.parsson.JsonProviderImpl>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124)
	at ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.systemProperty(JsonProviderTest.java:57)

@lukasj
Copy link
Contributor

lukasj commented May 3, 2022

TCK user guide currently does not allow having both test suites on the classpath at the same time or better - this setup is not documented/supported

@KyleAure
Copy link

KyleAure commented May 3, 2022

@lukasj I am confused by this statement. The TCK user guide makes no mention of the JsonProviderTest needed to be run separately from any of the other tests in the jakarta.json-tck-tests-2.1.0.jar test archive.
Can you please point me to the section that specifies this?
Note, that this test exists in the TCK test archive, not TCK pluggability which also has it's own provider.

@lukasj
Copy link
Contributor

lukasj commented May 3, 2022

Right, my bad, I have misread it as you mean running both groups at once (which is with or in the UG)

current runner runs each test (=testclass) in its own VM (as does the platform tck I believe), so the problem is not visible there. Do you think this has to made explicit somewhere? There is no way to avoid test interference without it.

@KyleAure
Copy link

KyleAure commented May 3, 2022

Hey Lukas, I think if this is a requirement then there should be a mention of it in the TCK User Guide.
Especially since Jakarta as a whole has no consistency in requirements for Standalone TCK Requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants