Skip to content

Commit

Permalink
Prepararion for version 1.0 (testing, distribution) (#400)
Browse files Browse the repository at this point in the history 
* Use and configure license-maven-plugin (org.honton.chas)

* First setup of distribution verification integration test

* Use Java 17 for compilation, updates of test dependencies, update license validation config

* Update comment on CacioTest annotation

* Cleanup

* Add generating fat jars for WhiteRabbit and RabbitInAHat; lock hsqldb version for Java 1.8

* Enforce Java 1.8 for distributed dependencies

* Update main.yml

Project now requires Java 17 to build. Should still produce java 8 (1.8) compatible artifacts though.

* Bump org.apache.avro:avro from 1.11.2 to 1.11.3 in /rabbit-core

Bumps org.apache.avro:avro from 1.11.2 to 1.11.3.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Use jdk8 classifier for hsqldb 2.7.x

* Exclude older version of hsqldb

* Fix image crop when using stem table

* Update stem table image

* Decrease size of table panel when using stem table.

Without this change, the table panel height is always higher than
needed (when using stem table), because the stem table is counted
as one of the items in the components list. It is however shown
separately at the top, which is already accounted for by the
stem table margin.

* Add snowflake support (#37)

* Refactor RichConnection into separate classes, and add an abstraction for the JDBC connection. Implement a Snowflake connection with this abstraction

* Add unit tests for SnowflakeConnector

* Added Snowflake support for SourceDataScan; added minimal test for it; some refactorings to move database responsibility to rabbit-core/databases

* Move more database details to rabbit-core/databases

* Clearer name for method

* Ignore snowflake.env

* Create PostgreSQL container in the TestContainers way

* Refactored Snowflake tests + a bit of documentation

* Fix Snowflake test for Java 17, and make it into an automated integration test instead of a unit test

* Remove duplicate postgresql test

* Make TestContainers based database tests into automated integration tests

* Suppress some warnings when generating fat jars

* Let autimatic integration tests fail when docker is not available

* Allow explicit skipping of Snowflake integration tests

* Added tests for Snowflake, delimited text files

* Switch to fully verifying the scan results against a reference version (v0.10.7)

* Working integration test for Snowflake, and some refactorings

* Some proper logging, small code improvements and cleanup

* Remove unused interface

* Added tests, some changes to support testing

* Make automated test work reliably (way too many changes, sorry)

* Rudimentary support for Snowflake authenticator parameter (untested)

* review xmlbeans dependencies, remove conflict

* extend integration test for distribution

* Restructuring database configuration. Work in process, but unit and integration tests all OK

* Restructuring database configuration 2/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 3/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 4/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 5/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 6/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 7/x. Still work in process, but unit and integration tests all OK

* Intermezzo: get rid of the package naming error (upper case R in whiteRabbit)

* Intermezzo: code cleanup

* Snowflake is now working from the GUI. And many small refactorings, like logging instead of printing to stout/err

* Refactor DbType into an enum, get rid of DBChoice

* Move DbType and DbSettings classes into configuration subpackage

* Avoid using a manually destructured DbSettings object when creating a RochConnection object

* Code cleanup, remove unneeded Snowflake references

* Refactoring, code cleanup

* More refactoring, code cleanup

* More refactoring, code cleanup and documentation

* Make sure that order of databases in pick list in GUI is the same as before, and enforce completeness of that list in a test

* Add/update copyright headers

* Add line to verify that a tooltip is shown for a DBConnectionInterface implementing class

* Test distribution for Snowflake JDBC issue with Java 17

* cleanup of build files

* Add verification that all JDBC drivers are in the distributed package

* Add/improve error reporting for Snowflake

* Disable screenshottaker in GuiTestExtension, hoping that that is what blocks the build on github. Fingers crossed

* Better(?) naming for database interface and implementing class

* Use our own GUITestExtension class

---------

Co-authored-by: Jan Blom <janblom@thehyve.nl>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jan Blom <janblom@thehyve.nl>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Spayralbe <stefan@thehyve.nl>
  • Loading branch information
@janblom @jan-at-the-hyve
@dependabot @Spayralbe
4 people authored Jan 19, 2024
1 parent f8a40bd commit 654ed24
Show file tree
Hide file tree
Showing 97 changed files with 5,679 additions and 1,703 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
- uses: actions/setup-java@v3
with:
distribution: temurin
java-version: 8
java-version: 17
cache: maven

# Compile the code
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ dist/
.idea/
target/
*.class
*.log

# jenv file(s)
.java-version
Expand All @@ -20,3 +21,6 @@ Try*
/examples/
.DS_Store
data/

# contains authentication data for a Snowflake instance
snowflake.env
40 changes: 30 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Requires Java 1.8 or higher, and read access to the database to be scanned. Java

Dependencies
============
For the distributable packages, the only requirement is Java 8. For building the package, also Maven is needed.
For the distributable packages, the only requirement is Java 8. For building the package, Java 17 and Maven are needed.

Getting Started
===============
Expand Down Expand Up @@ -90,22 +90,42 @@ Development
===========
White Rabbit and Rabbit in a Hat are structured as a Maven package and can be developed in Eclipse. Contributions are welcome.

While the software in the project can be executed with Java 1.8, for development Java 17 is needed.
This has to do with test and verification dependencies that are not available in a version compatible with Java 1.8 .

Please note that when using an IDE for development, source and target release must still be Java 1.8 . This is enforced
in the maven build file (pom.xml),

To generate the files ready for distribution, run `mvn install`.

### Testing

Some newer code has unit and/or integration tests. Tests that depend on external resources being available,
such as a database, should be excluded from executing automatically. You can use the @Tag annotation to combine
such tests in a group with the same tag, and exclude that tag from being run automatically by maven (this
should be done in the configuration of the surefire plugin in pom.xml of the module involved).

An exception to the above are tests that depend on Docker. Currently, these tests are implemented with the
TestContainers library, and are configured to check for Docker being present. If not, these tests will not
be run, but the tests as a whole will still succeed. However, it is recommended that these tests are run since
these tests verify essential functionality for WhiteRabbit, like the database interface.
A limited number of unit and integration tests exist. The integration tests run only in the maven verification phase,
(`mn verify`) and depend on docker being available to the user running the verification. If docker is not available, the
integration tests will fail.

Also, GitHub actions have been configured to run the test suite automatically.

#### Snowflake

There are automated tests for Snowflake, but since it is not (yet?) possible to have a local
Snowflake instance in a Docker container, these test will only run if the following information
is provided through environment variables:

SNOWFLAKE_WR_TEST_ACCOUNT
SNOWFLAKE_WR_TEST_USER
SNOWFLAKE_WR_TEST_PASSWORD
SNOWFLAKE_WR_TEST_WAREHOUSE
SNOWFLAKE_WR_TEST_DATABASE
SNOWFLAKE_WR_TEST_SCHEMA

It is recommended that user, password, database and schema are created for these tests only,
and do not relate in any way to any production environment.
The schema should not contain any tables when the test is started.

It is possible to skip the Snowflake tests without failing the build by passing
`-Dohdsi.org.whiterabbit.skip_snowflake_tests=1` to maven.

### Development status

Production. This program is being used by many people.
Binary file modified docs/images/riah_stem_table.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
206 changes: 202 additions & 4 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,18 @@
<name>Leporidae</name>
<url>https://www.ohdsi.org/analytic-tools/whiterabbit-for-etl-design/</url>

<organization>
<name>OHDSI</name>
<url>https://www.ohdsi.org</url>
</organization>

<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>

<repositories>
<repository>
<id>central</id>
Expand Down Expand Up @@ -78,7 +90,13 @@
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.release>1.8</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<skipTests>false</skipTests>
<skipUnitTests>${skipTests}</skipUnitTests>
<skipIntegrationTests>${skipTests}</skipIntegrationTests>
<org.ohdsi.whiterabbit.maxjdkversion>1.8</org.ohdsi.whiterabbit.maxjdkversion>
</properties>

<build>
Expand Down Expand Up @@ -106,8 +124,42 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<!-- JUnit 5 requires Surefire version 2.22.0 or higher -->
<version>3.0.0-M8</version>
<version>3.1.2</version>
<configuration>
<skipTests>${skipUnitTests}</skipTests>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<systemPropertyVariables>
<java.awt.headless>false</java.awt.headless>
<awt.toolkit>com.github.caciocavallosilano.cacio.ctc.CTCToolkit</awt.toolkit>
<java.awt.graphicsenv>com.github.caciocavallosilano.cacio.ctc.CTCGraphicsEnvironment</java.awt.graphicsenv>
</systemPropertyVariables>
<argLine>
-Doracle.jdbc.timezoneAsRegion=false
<!-- Line(s) below are needed to keep TestSourceDataScan.updateEnv() working in java 17+
This is considered an acceptable hack since it only affects test code -->
--add-exports=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
<!-- suppress exceptions from cacio-tta, see https://github.com/CaciocavalloSilano/caciocavallo -->
--add-opens java.base/java.lang.reflect=ALL-UNNAMED
--add-exports java.base/java.lang.reflect=ALL-UNNAMED
--add-exports=java.desktop/java.awt=ALL-UNNAMED
--add-exports=java.desktop/java.awt.peer=ALL-UNNAMED
--add-exports=java.desktop/sun.awt.image=ALL-UNNAMED
--add-exports=java.desktop/sun.java2d=ALL-UNNAMED
--add-exports=java.desktop/java.awt.dnd.peer=ALL-UNNAMED
--add-exports=java.desktop/sun.awt=ALL-UNNAMED
--add-exports=java.desktop/sun.awt.event=ALL-UNNAMED
--add-exports=java.desktop/sun.awt.datatransfer=ALL-UNNAMED
--add-exports=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.desktop/java.awt=ALL-UNNAMED
--add-opens=java.desktop/sun.java2d=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens java.base/java.lang=ALL-UNNAMED
</argLine>

</configuration>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
Expand All @@ -116,14 +168,41 @@
</dependency>
</dependencies>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
<skipTests>${skipIntegrationTests}</skipTests>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.honton.chas</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>0.0.3</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>3.3.1</version>
</plugin>
</plugins>
</pluginManagement>

<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
<version>3.3.1</version>
<configuration>
<filesets>
<fileset>
Expand All @@ -146,6 +225,125 @@
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
<plugin>
<!--
While the source and target Java version are still 1.8, Java 17 is required to be able to use
recent versions of test and verification tools. This plugin enforces the required Java version,
and gives a cleaner error message than failing on a dependency that is not compatible with an
older version of Java
-->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.4.1</version>
<executions>
<execution>
<id>enforce-java</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireJavaVersion>
<version>17</version>
</requireJavaVersion>
</rules>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.honton.chas</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>0.0.3</version>
<configuration>
<acceptableLicenses>
<license>
<name>Oracle Free Use Terms and Conditions \(FUTC\)</name>
</license>
<license>
<name>The GNU General Public License, v2 with Universal FOSS Exception, v1.0</name>
</license>
<license>
<name>Plexus</name>
<url>https://github.com/dom4j/dom4j/blob/master/LICENSE</url>
</license>
<license>
<name>(The )?Apache( )?(Software )?(License)?(,)? (Version )?(2.0|v2)</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
<license>
<name>BSD-2-Clause</name>
<url>https://jdbc.postgresql.org/about/license.html</url>
</license>
<license>
<name>HSQLDB License, a BSD open source license</name>
<url>https://hsqldb.org/web/hsqlLicense.html</url>
</license>
<license>
<name>MIT License</name>
<url>https://www.opensource.org/licenses/mit-license.php</url>
</license>
<license>
<name>MIT</name>
<url>https://opensource.org/licenses/MIT</url>
</license>
<license>
<name>Eclipse Public License 1.0</name>
<url>https://www.eclipse.org/legal/epl-v10.html</url>
</license>
<license>
<name>Eclipse Public License v2.0</name>
<url>https://www.eclipse.org/legal/epl-v20.html</url>
</license>
<license>
<name>GPL2 with classpath exception</name>
<url>https://openjdk.java.net/legal/gplv2+ce.html</url>
</license>
</acceptableLicenses>
<excludes>
<!-- apparently there is no license information available for sqljdbc4, even though
it is published on Maven repos -->
<exclude>com.microsoft.sqlserver:sqljdbc4</exclude>

<!-- PLEASE NOTE: teradata and googlebigquery are provided within the project, they
cannot be found on a maven repo; no license info available; they should ideally
not be part of the project, and should be removed here -->
<exclude>com.teradata.jdbc:terajdbc4</exclude>
<exclude>com.teradata.tdgss:tdgssconfig</exclude>
<exclude>com.simba.googlebigquery.jdbc:GoogleBigQueryJDBC</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-api-client</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-http-client</exclude>
<exclude>com.simba.googlebigquery.jdbc:gax</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-http-client-jackson2</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-oauth-client</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-auth-library-oauth2-http</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-auth-library-credentials</exclude>
<exclude>com.simba.googlebigquery.jdbc:jackson-core</exclude>
<exclude>com.simba.googlebigquery.jdbc:guava</exclude>
<exclude>com.simba.googlebigquery.jdbc:google-api-service-bigquery</exclude>
<exclude>com.simba.googlebigquery.jdbc:opencensus-api</exclude>
<exclude>com.simba.googlebigquery.jdbc:opencensus-contrib-http-util</exclude>
<exclude>com.simba.googlebigquery.jdbc:grpc-context</exclude>
<exclude>com.simba.googlebigquery.jdbc:joda-time</exclude>
</excludes>
<!-- acceptableLicenseResources>osi-widely-used</acceptableLicenseResources -->
</configuration>
<executions>
<execution>
<goals>
<goal>compliance</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.github.ferstl</groupId>
<artifactId>depgraph-maven-plugin</artifactId>
<version>4.0.2</version>
<!-- configuration>
...
</configuration -->
</plugin>
</plugins>

<resources>
Expand Down
Loading

0 comments on commit 654ed24

Please sign in to comment.