Adds Integration Testsuite for all supported runtimes, upgrades some deps #262

Karm · 2020-01-17T15:43:44Z

See src/it/README.md for details on what the TS does and how.

Some components and dependencies changes to be looked at:

Thorntail upgraded from 2.2.1 to 2.5.0. I needed that for better Arquillian support. I figure we already established with CQs that this runtime dependency is not a problem.
Adds a dependency on RestEasy 4.3.0.Final
Adds dependency on maven-failsafe-plugin
Lifts Java requirement from 1.8 to 11

It works for all runtimes on Linux

It correctly hit the Helidon package problem, so it made me rebase. It also hit an error message from Liberty which I whitelisted. More on whitelisting is in src/it/README.md.

...
INFO: Testing server: THORNTAIL_V2, config: EMPTY
INFO: Testing server: THORNTAIL_V2, config: ALL
INFO: Testing server: THORNTAIL_V2, config: ALL_BUT_JWT_REST
INFO: Testing server: THORNTAIL_V2, config: JWT_REST
INFO: Testing server: PAYARA_MICRO, config: EMPTY
INFO: Testing server: PAYARA_MICRO, config: ALL
INFO: Testing server: PAYARA_MICRO, config: ALL_BUT_JWT_REST
INFO: Testing server: PAYARA_MICRO, config: JWT_REST
INFO: Testing server: LIBERTY, config: EMPTY
INFO: Testing server: LIBERTY, config: ALL
INFO: Testing server: LIBERTY, config: ALL_BUT_JWT_REST
INFO: Testing server: LIBERTY, config: JWT_REST
INFO: Testing server: HELIDON, config: EMPTY
INFO: Testing server: HELIDON, config: ALL
INFO: Testing server: HELIDON, config: ALL_BUT_JWT_REST
INFO: Testing server: HELIDON, config: JWT_REST
INFO: Testing server: KUMULUZEE, config: EMPTY
INFO: Testing server: KUMULUZEE, config: ALL
INFO: Testing server: KUMULUZEE, config: ALL_BUT_JWT_REST
INFO: Testing server: KUMULUZEE, config: JWT_REST
INFO: Testing server: TOMEE, config: EMPTY
INFO: Testing server: TOMEE, config: ALL
INFO: Testing server: TOMEE, config: ALL_BUT_JWT_REST
INFO: Testing server: TOMEE, config: JWT_REST
...


Results :

Tests run: 30, Failures: 0, Errors: 0, Skipped: 0


[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:34 min
[INFO] Finished at: 2020-01-17T16:28:23+01:00

It is 24 integration tests for runtimes, 1 smoke check for the API before it all goes to runtimes and 5 unit tests.

The time it takes is pretty nice, but mind you I have all the artifacts cached in my local .m2 repo and I am running it on a fairly powerful workstation. When run fresh, it obviously downloads the Internet and takes much more time.

Ping @rdebusscher @cealsair @Emily-Jiang for thoughts.

Windows

Test suite fully supports Windows, although Payara does not pass because it doesn't start. See below.

Windows: Tests run: 30, Failures: 4, Errors: 0, Skipped: 0

Runtime	Windows TS status	Comment
Thorntail	✔️
Liberty	✔️
Payara	❌	Payara issue; doesn't start
Kumuluzee	✔️
Tomee	✔️

Payara issue 4445

Payara fails not because of the test suite, but because of Payara.
@rdebusscher , could you follow up with your peers on (no examples selected, simplest case):

C:\tmp\payara\demo
λ mvn clean package && java -jar target/demo-microbundle.jar
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:demo >--------------------------
[INFO] Building demo 1.0-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ demo ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ demo ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ demo ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[INFO] Compiling 2 source files to C:\tmp\payara\demo\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ demo ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\tmp\payara\demo\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ demo ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ demo ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-war-plugin:2.2:war (default-war) @ demo ---
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/C:/Users/Administrator/.m2/repository/com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar) to field java.util.Properties.defaults
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Packaging webapp
[INFO] Assembling webapp [demo] in [C:\tmp\payara\demo\target\demo]
[INFO] Processing war project
[INFO] Copying webapp resources [C:\tmp\payara\demo\src\main\webapp]
[INFO] Webapp assembled in [41 msecs]
[INFO] Building war: C:\tmp\payara\demo\target\demo.war
[INFO]
[INFO] --- payara-micro-maven-plugin:1.0.1:bundle (default) @ demo ---
[INFO] Configured Artifact: fish.payara.extras:payara-micro:5.194:jar
[INFO] Unpacking C:\Users\Administrator\.m2\repository\fish\payara\extras\payara-micro\5.194\payara-micro-5.194.jar to C:\tmp\payara\demo\target\extracted-payaramicro with includes "" and excludes ""
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] Configured Artifact: com.example:demo:1.0-SNAPSHOT:war
[INFO] Copying demo.war to C:\tmp\payara\demo\target\extracted-payaramicro\MICRO-INF\deploy\ROOT.war
[INFO] Building jar: C:\tmp\payara\demo\target\demo-microbundle.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  7.567 s
[INFO] Finished at: 2020-01-20T01:59:54-08:00
[INFO] ------------------------------------------------------------------------
Jan 20, 2020 1:59:55 AM fish.payara.micro.PayaraMicro main
SEVERE: null
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at fish.payara.micro.boot.loader.MainMethodRunner.run(MainMethodRunner.java:50)
        at fish.payara.micro.boot.loader.Launcher.launch(Launcher.java:114)
        at fish.payara.micro.boot.loader.Launcher.launch(Launcher.java:73)
        at fish.payara.micro.boot.PayaraMicroLauncher.create(PayaraMicroLauncher.java:87)
        at fish.payara.micro.boot.PayaraMicroLauncher.main(PayaraMicroLauncher.java:71)
        at fish.payara.micro.PayaraMicro.main(PayaraMicro.java:397)
Caused by: java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
        at java.base/java.util.Properties.loadConvert(Properties.java:654)
        at java.base/java.util.Properties.load0(Properties.java:454)
        at java.base/java.util.Properties.load(Properties.java:404)
        at fish.payara.micro.impl.PayaraMicroImpl.setBootProperties(PayaraMicroImpl.java:2567)
        at fish.payara.micro.impl.PayaraMicroImpl.create(PayaraMicroImpl.java:205)
        at fish.payara.micro.impl.PayaraMicroImpl.main(PayaraMicroImpl.java:200)
        ... 10 more

Please?

System

Windows 2019
OpenJDK Runtime Environment (build 11.0.1-redhat+13-LTS)
Apache Maven 3.6.0

Karm · 2020-01-18T16:29:20Z

< removed comment - integrated into description for brevity >

Karm · 2020-01-20T14:55:06Z

This is ready for review now. I am interested in results on Mac as I tried only Linux and Windows.

Karm · 2020-02-03T08:28:06Z

Hello, @rdebusscher , could you run the TS and review the MR? I would like to have it merged eventually 😃

Most importantly, could you run tests and see whether it works on Mac, please?

pom.xml

src/it/README.md

src/it/java/org/eclipse/microprofile/starter/utils/Logs.java

src/it/java/org/eclipse/microprofile/starter/utils/ReadmeParser.java

src/it/java/org/eclipse/microprofile/starter/utils/WebpageTester.java

src/it/java/org/eclipse/microprofile/starter/utils/Commands.java

src/it/java/org/eclipse/microprofile/starter/utils/MPSpec.java

src/test/resources/readme_examples/li-service-a-readme.md

Emily-Jiang · 2020-02-03T11:35:09Z

A general feedback: Thread.sleep was used in many places. Can it be replaced by Awaitability?

Karm · 2020-02-03T11:41:42Z

A general feedback: Thread.sleep was used in many places. Can it be replaced by Awaitability?

@Emily-Jiang Do you mean https://github.com/awaitility/awaitility ?

Could you elaborate more on what exactly is wrong with sleep in a loop until a condition OR timeout is reached? Given the usual dangers of Interruption and context - which I find kinda irrelevant in this particular code's context.

I would like to have some reasoning for adding yet another library...

See [src/it/README.md](src/it/README.md) for details. Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

Karm · 2020-02-04T11:15:09Z

A general feedback: Thread.sleep was used in many places. Can it be replaced by Awaitability?

@Emily-Jiang Do you mean https://github.com/awaitility/awaitility ?

Could you elaborate more on what exactly is wrong with sleep in a loop until a condition OR timeout is reached? Given the usual dangers of Interruption and context - which I find kinda irrelevant in this particular code's context.

I would like to have some reasoning for adding yet another library...

Resolved during call. The code is actually O.K.

Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

Karm · 2020-02-04T11:52:11Z

@rdebusscher @Emily-Jiang

This is ready for another review. Changes addressed in the last commit. As usual, when it comes to bugs, please, do downvote the PR into oblivion as necessary :)
When it comes to missing features or hypothetically missing features, I would kindly ask to not block the PR on those and to open Issues for future PRs.

Karm · 2020-02-04T12:13:19Z

Works after the update too...

Linux

$ mvn clean verify -Pthorntail -Dskip.integration.tests=false -DSTARTER_TS_WORKSPACE=/dev/shm/

Results :

Tests run: 30, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:58 min
[INFO] Finished at: 2020-02-04T12:40:24+01:00
[INFO] ------------------------------------------------------------------------

Windows

C:\Users\Administrator\source\microprofile-starter (tests -> origin)
λ mvn clean verify -Pthorntail -Dskip.integration.tests=false -DSTARTER_TS_WORKSPACE=C:\tmp\

Tests run: 30, Failures: 4, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20:33 min
[INFO] Finished at: 2020-02-04T04:04:11-08:00
[INFO] ------------------------------------------------------------------------

Those 4 failures on Windows are a known Payara bug: payara/Payara#4445

Emily-Jiang

LGTM!

Karm requested review from rdebusscher and Emily-Jiang January 17, 2020 15:45

Karm self-assigned this Jan 17, 2020

Karm added the enhancement New feature, improvement or request for the code label Jan 17, 2020

Karm force-pushed the tests branch from 4f117f9 to ac6d828 Compare January 17, 2020 18:13

Karm changed the title ~~Adds Integration Testsuite for all supported runtimes~~ [WIP] Adds Integration Testsuite for all supported runtimes Jan 20, 2020

Karm added the DO NOT MERGE label Jan 20, 2020

Karm force-pushed the tests branch from ac6d828 to 31e2bf7 Compare January 20, 2020 12:15

Karm changed the title ~~[WIP] Adds Integration Testsuite for all supported runtimes~~ Adds Integration Testsuite for all supported runtimes, upgrades some deps Jan 20, 2020

Karm removed the DO NOT MERGE label Jan 20, 2020

Karm requested a review from cealsair January 20, 2020 14:54

Karm added this to the 1.x milestone Jan 20, 2020

Karm mentioned this pull request Jan 20, 2020

Changed to use Thorntail 2.5.0.Final. Fixes #196 #197

Closed

rdebusscher suggested changes Feb 3, 2020

View reviewed changes

Emily-Jiang reviewed Feb 3, 2020

View reviewed changes

src/it/java/org/eclipse/microprofile/starter/utils/WebpageTester.java Show resolved Hide resolved

Emily-Jiang reviewed Feb 3, 2020

View reviewed changes

Karm added 2 commits February 3, 2020 16:28

Adds Integration Testsuite for all supported runtimes

dbcefdb

See [src/it/README.md](src/it/README.md) for details. Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

README update: Windows fully supported

5b01936

Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

Addresses PR review, code clarity, return values, Assert.fail, JavaDoc

f76add6

Signed-off-by: Michal Karm Babacek <karm@fedoraproject.org>

Karm force-pushed the tests branch from e94faab to f76add6 Compare February 4, 2020 11:36

Karm requested review from rdebusscher and Emily-Jiang February 4, 2020 11:49

Karm modified the milestones: 1.x, 1.1 Feb 4, 2020

Emily-Jiang approved these changes Feb 4, 2020

View reviewed changes

Karm requested a review from OndroMih February 6, 2020 09:36

Karm mentioned this pull request Feb 6, 2020

Cannot Build MP Starter on Maven 3.6.0+ #196

Closed

rdebusscher approved these changes Feb 6, 2020

View reviewed changes

Karm mentioned this pull request Feb 6, 2020

Add RestAssured tests for REST API #157

Closed

Karm merged commit 6edd815 into microprofile:master Feb 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Integration Testsuite for all supported runtimes, upgrades some deps #262

Adds Integration Testsuite for all supported runtimes, upgrades some deps #262

Karm commented Jan 17, 2020 •

edited

Loading

Karm commented Jan 18, 2020 •

edited

Loading

Karm commented Jan 20, 2020

Karm commented Feb 3, 2020 •

edited

Loading

Emily-Jiang commented Feb 3, 2020

Karm commented Feb 3, 2020

Karm commented Feb 4, 2020

Karm commented Feb 4, 2020

Karm commented Feb 4, 2020

Emily-Jiang left a comment

Adds Integration Testsuite for all supported runtimes, upgrades some deps #262

Adds Integration Testsuite for all supported runtimes, upgrades some deps #262

Conversation

Karm commented Jan 17, 2020 • edited Loading

It works for all runtimes on Linux

Windows

Payara issue 4445

System

Karm commented Jan 18, 2020 • edited Loading

Karm commented Jan 20, 2020

Karm commented Feb 3, 2020 • edited Loading

Emily-Jiang commented Feb 3, 2020

Karm commented Feb 3, 2020

Karm commented Feb 4, 2020

Karm commented Feb 4, 2020

Karm commented Feb 4, 2020

Linux

Windows

Emily-Jiang left a comment

Choose a reason for hiding this comment

Karm commented Jan 17, 2020 •

edited

Loading

Karm commented Jan 18, 2020 •

edited

Loading

Karm commented Feb 3, 2020 •

edited

Loading