From 654ed24fff86c7d6280acc9dedd74e087c8522f9 Mon Sep 17 00:00:00 2001 From: janblom Date: Fri, 19 Jan 2024 18:35:51 +0100 Subject: [PATCH] Prepararion for version 1.0 (testing, distribution) (#400) * Use and configure license-maven-plugin (org.honton.chas) * First setup of distribution verification integration test * Use Java 17 for compilation, updates of test dependencies, update license validation config * Update comment on CacioTest annotation * Cleanup * Add generating fat jars for WhiteRabbit and RabbitInAHat; lock hsqldb version for Java 1.8 * Enforce Java 1.8 for distributed dependencies * Update main.yml Project now requires Java 17 to build. Should still produce java 8 (1.8) compatible artifacts though. * Bump org.apache.avro:avro from 1.11.2 to 1.11.3 in /rabbit-core Bumps org.apache.avro:avro from 1.11.2 to 1.11.3. --- updated-dependencies: - dependency-name: org.apache.avro:avro dependency-type: direct:production ... Signed-off-by: dependabot[bot] * Use jdk8 classifier for hsqldb 2.7.x * Exclude older version of hsqldb * Fix image crop when using stem table * Update stem table image * Decrease size of table panel when using stem table. Without this change, the table panel height is always higher than needed (when using stem table), because the stem table is counted as one of the items in the components list. It is however shown separately at the top, which is already accounted for by the stem table margin. * Add snowflake support (#37) * Refactor RichConnection into separate classes, and add an abstraction for the JDBC connection. Implement a Snowflake connection with this abstraction * Add unit tests for SnowflakeConnector * Added Snowflake support for SourceDataScan; added minimal test for it; some refactorings to move database responsibility to rabbit-core/databases * Move more database details to rabbit-core/databases * Clearer name for method * Ignore snowflake.env * Create PostgreSQL container in the TestContainers way * Refactored Snowflake tests + a bit of documentation * Fix Snowflake test for Java 17, and make it into an automated integration test instead of a unit test * Remove duplicate postgresql test * Make TestContainers based database tests into automated integration tests * Suppress some warnings when generating fat jars * Let autimatic integration tests fail when docker is not available * Allow explicit skipping of Snowflake integration tests * Added tests for Snowflake, delimited text files * Switch to fully verifying the scan results against a reference version (v0.10.7) * Working integration test for Snowflake, and some refactorings * Some proper logging, small code improvements and cleanup * Remove unused interface * Added tests, some changes to support testing * Make automated test work reliably (way too many changes, sorry) * Rudimentary support for Snowflake authenticator parameter (untested) * review xmlbeans dependencies, remove conflict * extend integration test for distribution * Restructuring database configuration. Work in process, but unit and integration tests all OK * Restructuring database configuration 2/x. Still work in process, but unit and integration tests all OK * Restructuring database configuration 3/x. Still work in process, but unit and integration tests all OK * Restructuring database configuration 4/x. Still work in process, but unit and integration tests all OK * Restructuring database configuration 5/x. Still work in process, but unit and integration tests all OK * Restructuring database configuration 6/x. Still work in process, but unit and integration tests all OK * Restructuring database configuration 7/x. Still work in process, but unit and integration tests all OK * Intermezzo: get rid of the package naming error (upper case R in whiteRabbit) * Intermezzo: code cleanup * Snowflake is now working from the GUI. And many small refactorings, like logging instead of printing to stout/err * Refactor DbType into an enum, get rid of DBChoice * Move DbType and DbSettings classes into configuration subpackage * Avoid using a manually destructured DbSettings object when creating a RochConnection object * Code cleanup, remove unneeded Snowflake references * Refactoring, code cleanup * More refactoring, code cleanup * More refactoring, code cleanup and documentation * Make sure that order of databases in pick list in GUI is the same as before, and enforce completeness of that list in a test * Add/update copyright headers * Add line to verify that a tooltip is shown for a DBConnectionInterface implementing class * Test distribution for Snowflake JDBC issue with Java 17 * cleanup of build files * Add verification that all JDBC drivers are in the distributed package * Add/improve error reporting for Snowflake * Disable screenshottaker in GuiTestExtension, hoping that that is what blocks the build on github. Fingers crossed * Better(?) naming for database interface and implementing class * Use our own GUITestExtension class --------- Co-authored-by: Jan Blom --------- Signed-off-by: dependabot[bot] Co-authored-by: Jan Blom Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Spayralbe --- .github/workflows/main.yml | 2 +- .gitignore | 4 + README.md | 40 +- docs/images/riah_stem_table.png | Bin 39528 -> 42860 bytes pom.xml | 206 +++++- rabbit-core/pom.xml | 115 ++- .../org/ohdsi/databases/DBConnection.java | 374 ++++++++++ .../java/org/ohdsi/databases/DBConnector.java | 90 ++- .../org/ohdsi/databases/DBRowIterator.java | 123 ++++ .../java/org/ohdsi/databases/DataType.java | 22 + .../main/java/org/ohdsi/databases/DbType.java | 52 -- .../java/org/ohdsi/databases/FieldInfo.java | 255 +++++++ .../java/org/ohdsi/databases/QueryResult.java | 53 ++ .../org/ohdsi/databases/RichConnection.java | 328 ++------- .../org/ohdsi/databases/ScanParameters.java | 38 + .../org/ohdsi/databases/SnowflakeHandler.java | 289 ++++++++ .../org/ohdsi/databases/StorageHandler.java | 233 ++++++ .../databases}/UniformSamplingReservoir.java | 46 +- .../configuration/ConfigurationField.java | 137 ++++ .../configuration/ConfigurationFields.java | 63 ++ .../configuration/ConfigurationValidator.java | 23 + .../configuration/DBConfiguration.java | 168 +++++ .../DBConfigurationException.java | 24 + .../databases/configuration}/DbSettings.java | 11 +- .../ohdsi/databases/configuration/DbType.java | 119 +++ .../configuration/FieldValidator.java | 23 + .../configuration/ValidationFeedback.java | 77 ++ .../ohdsi/ooxml/ReadXlsxFileWithHeader.java | 2 +- .../dataModel/TableCellLongTextRenderer.java | 17 + .../org/ohdsi/utilities/ScanFieldName.java | 17 + .../org/ohdsi/utilities/ScanSheetName.java | 17 + .../org/ohdsi/utilities/SimpleCounter.java | 25 +- .../org/ohdsi/utilities/StringUtilities.java | 16 +- .../java/org/ohdsi/utilities/Version.java | 17 + .../utilities/collections/CountingSet.java | 9 +- .../org/ohdsi/utilities/files/IniFile.java | 22 + .../files/QuickAndDirtyXlsxReader.java | 8 +- .../files/WriteCSVFileWithHeader.java | 17 + .../ohdsi/databases/DBConfigurationTest.java | 59 ++ .../org/ohdsi/databases/DBConnectorTest.java | 46 ++ .../ohdsi/databases/SnowflakeTestUtils.java | 91 +++ .../databases/TestConfigurationField.java | 142 ++++ .../ohdsi/databases/TestSnowflakeHandler.java | 100 +++ .../databases/configuration/DbTypeTest.java | 37 + rabbit-core/src/test/resources/snowflake.ini | 16 + rabbitinahat/pom.xml | 96 ++- .../rabbitInAHat/DescriptionTextArea.java | 17 + .../rabbitInAHat/FetchCDMModelFromServer.java | 10 +- .../org/ohdsi/rabbitInAHat/FilterDialog.java | 17 + .../org/ohdsi/rabbitInAHat/MappingPanel.java | 18 +- .../ohdsi/rabbitInAHat/RabbitInAHatMain.java | 1 - .../org/ohdsi/rabbitInAHat/SQLGenerator.java | 17 + .../dataModel/StemTableFactory.java | 17 + .../rabbitInAHat/TestRabbitInAHatMain.java | 35 +- whiterabbit/pom.xml | 181 ++++- .../org/ohdsi/whiteRabbit/scan/DataType.java | 5 - .../{whiteRabbit => whiterabbit}/Console.java | 15 +- .../ErrorReport.java | 2 +- .../ObjectExchange.java | 2 +- .../org/ohdsi/whiterabbit/PanelsManager.java | 32 + .../WhiteRabbitMain.java | 691 ++++++++---------- .../fakeDataGenerator/FakeDataGenerator.java | 15 +- .../ohdsi/whiterabbit/gui/LocationsPanel.java | 354 +++++++++ .../ohdsi/whiterabbit/gui/SourcePanel.java | 39 + .../scan/SourceDataScan.java | 425 ++--------- .../WhiteRabbit.ico | Bin .../WhiteRabbit128.png | Bin .../WhiteRabbit16.png | Bin .../WhiteRabbit256.png | Bin .../WhiteRabbit32.png | Bin .../WhiteRabbit48.png | Bin .../WhiteRabbit64.png | Bin .../whiterabbit/scan/GUITestExtension.java | 66 ++ .../ohdsi/whiterabbit/scan/ScanTestUtils.java | 238 +++--- ...racle.java => SourceDataScanOracleIT.java} | 55 +- .../scan/SourceDataScanPostgreSQLGuiIT.java | 121 +++ .../scan/SourceDataScanPostgreSQLIT.java | 107 +++ .../scan/SourceDataScanSnowflakeGuiIT.java | 142 ++++ .../scan/SourceDataScanSnowflakeIT.java | 159 ++++ .../whiterabbit/scan/TestSourceDataScan.java | 200 ----- .../scan/TestSourceDataScanCsvGui.java | 86 +++ .../scan/TestSourceDataScanCsvIniFile.java | 74 ++ .../scan/TestSourceDataScanPostgreSQL.java | 83 --- .../scan/VerifyDistributionIT.java | 223 ++++++ .../src/test/resources/scan_data/README.md | 6 + .../ScanReport-reference-v0.10.7-csv.xlsx | Bin 0 -> 9379 bytes .../ScanReport-reference-v0.10.7-sql.xlsx | Bin 0 -> 9384 bytes .../test/resources/scan_data/cost-header.csv | 35 + .../resources/scan_data/cost-no-header.csv | 34 + .../src/test/resources/scan_data/cost.csv | 34 - .../scan_data/create_data_postgresql.sql | 4 +- .../scan_data/create_data_snowflake.sql | 32 + .../resources/scan_data/person-header.csv | 31 + .../resources/scan_data/person-no-header.csv | 30 + .../src/test/resources/scan_data/person.csv | 30 - .../scan_data/snowflake.ini.template | 16 + .../test/resources/scan_data/tsv.ini.template | 14 + 97 files changed, 5679 insertions(+), 1703 deletions(-) create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/DBConnection.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/DBRowIterator.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/DataType.java delete mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/DbType.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/FieldInfo.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/QueryResult.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/ScanParameters.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/SnowflakeHandler.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/StorageHandler.java rename {whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan => rabbit-core/src/main/java/org/ohdsi/databases}/UniformSamplingReservoir.java (79%) create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationField.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationFields.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationValidator.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfiguration.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfigurationException.java rename {whiterabbit/src/main/java/org/ohdsi/whiteRabbit => rabbit-core/src/main/java/org/ohdsi/databases/configuration}/DbSettings.java (76%) create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbType.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/FieldValidator.java create mode 100644 rabbit-core/src/main/java/org/ohdsi/databases/configuration/ValidationFeedback.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/DBConfigurationTest.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/DBConnectorTest.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/SnowflakeTestUtils.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/TestConfigurationField.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/TestSnowflakeHandler.java create mode 100644 rabbit-core/src/test/java/org/ohdsi/databases/configuration/DbTypeTest.java create mode 100644 rabbit-core/src/test/resources/snowflake.ini delete mode 100644 whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/DataType.java rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/Console.java (85%) rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/ErrorReport.java (96%) rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/ObjectExchange.java (94%) create mode 100644 whiterabbit/src/main/java/org/ohdsi/whiterabbit/PanelsManager.java rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbitMain.java (60%) rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/fakeDataGenerator/FakeDataGenerator.java (91%) create mode 100644 whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/LocationsPanel.java create mode 100644 whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/SourcePanel.java rename whiterabbit/src/main/java/org/ohdsi/{whiteRabbit => whiterabbit}/scan/SourceDataScan.java (59%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit.ico (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit128.png (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit16.png (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit256.png (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit32.png (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit48.png (100%) rename whiterabbit/src/main/resources/org/ohdsi/{whiteRabbit => whiterabbit}/WhiteRabbit64.png (100%) create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/GUITestExtension.java rename whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/{TestSourceDataScanOracle.java => SourceDataScanOracleIT.java} (72%) create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLGuiIT.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLIT.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeGuiIT.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeIT.java delete mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScan.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvGui.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvIniFile.java delete mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanPostgreSQL.java create mode 100644 whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/VerifyDistributionIT.java create mode 100644 whiterabbit/src/test/resources/scan_data/README.md create mode 100644 whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-csv.xlsx create mode 100644 whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-sql.xlsx create mode 100644 whiterabbit/src/test/resources/scan_data/cost-header.csv create mode 100644 whiterabbit/src/test/resources/scan_data/cost-no-header.csv delete mode 100644 whiterabbit/src/test/resources/scan_data/cost.csv create mode 100644 whiterabbit/src/test/resources/scan_data/create_data_snowflake.sql create mode 100644 whiterabbit/src/test/resources/scan_data/person-header.csv create mode 100644 whiterabbit/src/test/resources/scan_data/person-no-header.csv delete mode 100644 whiterabbit/src/test/resources/scan_data/person.csv create mode 100644 whiterabbit/src/test/resources/scan_data/snowflake.ini.template create mode 100644 whiterabbit/src/test/resources/scan_data/tsv.ini.template diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 65a5c4fb..f1c4cd36 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -21,7 +21,7 @@ jobs: - uses: actions/setup-java@v3 with: distribution: temurin - java-version: 8 + java-version: 17 cache: maven # Compile the code diff --git a/.gitignore b/.gitignore index 30e7e2f5..75067728 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ dist/ .idea/ target/ *.class +*.log # jenv file(s) .java-version @@ -20,3 +21,6 @@ Try* /examples/ .DS_Store data/ + +# contains authentication data for a Snowflake instance +snowflake.env diff --git a/README.md b/README.md index e2e907aa..853e66f4 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Requires Java 1.8 or higher, and read access to the database to be scanned. Java Dependencies ============ -For the distributable packages, the only requirement is Java 8. For building the package, also Maven is needed. +For the distributable packages, the only requirement is Java 8. For building the package, Java 17 and Maven are needed. Getting Started =============== @@ -90,22 +90,42 @@ Development =========== White Rabbit and Rabbit in a Hat are structured as a Maven package and can be developed in Eclipse. Contributions are welcome. +While the software in the project can be executed with Java 1.8, for development Java 17 is needed. +This has to do with test and verification dependencies that are not available in a version compatible with Java 1.8 . + +Please note that when using an IDE for development, source and target release must still be Java 1.8 . This is enforced +in the maven build file (pom.xml), + To generate the files ready for distribution, run `mvn install`. ### Testing -Some newer code has unit and/or integration tests. Tests that depend on external resources being available, -such as a database, should be excluded from executing automatically. You can use the @Tag annotation to combine -such tests in a group with the same tag, and exclude that tag from being run automatically by maven (this -should be done in the configuration of the surefire plugin in pom.xml of the module involved). - -An exception to the above are tests that depend on Docker. Currently, these tests are implemented with the -TestContainers library, and are configured to check for Docker being present. If not, these tests will not -be run, but the tests as a whole will still succeed. However, it is recommended that these tests are run since -these tests verify essential functionality for WhiteRabbit, like the database interface. +A limited number of unit and integration tests exist. The integration tests run only in the maven verification phase, +(`mn verify`) and depend on docker being available to the user running the verification. If docker is not available, the +integration tests will fail. Also, GitHub actions have been configured to run the test suite automatically. +#### Snowflake + +There are automated tests for Snowflake, but since it is not (yet?) possible to have a local +Snowflake instance in a Docker container, these test will only run if the following information +is provided through environment variables: + + SNOWFLAKE_WR_TEST_ACCOUNT + SNOWFLAKE_WR_TEST_USER + SNOWFLAKE_WR_TEST_PASSWORD + SNOWFLAKE_WR_TEST_WAREHOUSE + SNOWFLAKE_WR_TEST_DATABASE + SNOWFLAKE_WR_TEST_SCHEMA + +It is recommended that user, password, database and schema are created for these tests only, +and do not relate in any way to any production environment. +The schema should not contain any tables when the test is started. + +It is possible to skip the Snowflake tests without failing the build by passing +`-Dohdsi.org.whiterabbit.skip_snowflake_tests=1` to maven. + ### Development status Production. This program is being used by many people. diff --git a/docs/images/riah_stem_table.png b/docs/images/riah_stem_table.png index 73104a4ae79c9ab26fc3a8b2bbf4d46d5c619bd2..e6de0bdd8e353ced3c35f65192df920f92632686 100644 GIT binary patch literal 42860 zcmY&g1z42Nw}xerk`|C&x`YL!mPWdzL8PP`=?(!wkxog2kZzFfE~%xYS-QKqAHRRx zd*yqc{dRWh%$#%P%=^yx>9vY1HYOP+5)u-&yquIe5)ujp5)z02{RxmFn-{q-B@ zKfj_-d_(!qJBZ*hqXX6RDxeZQYfW8OU1cRfQ%8Fa6Enwm<{X~(PLCOoggpg;S9^0; z6Nsn1or8;@rwHvoIRt_C$7Coi_8577cU1_6Hj&r7rK8{ z@}GL7%w0^Kt({!09UUN#^_skMbaNG^<=HwhJ%Br24%&(ch3H`e)xAqHkN)7#dJaP?N z3bvn4eW6#P7YEVfA;8^c&j}I4W5NV-0U-O*YUKbrD`boqBn(}2vzoUjabUSX@Q+mS z(I?VBXGO_mL0b|_t?ejVqc7&@kqIEkvS4n(xF2|r1^({?77xUcMf#%~`dIvbpH@<2 zL5g7kSX{m5(GNNpX6%1Ui5KCuIxi|Fb98;Kt9prv;e;(4!;hFU3P{v3^3zJ*{axgo5_Q!Uv4`rGiPyXBFwRt%Yh1w}j29DFG}Zv&wTGYd z*V@gjOzuaE-3u6pt=?TMCQAfAVR^YdRqDvgv|IwqVGY;EQ6;14{)kQ9e}8vlK3SmE zzubBh$-jX9>SbZv|4#85#_9H?;)36W0m81zanUe}N`zIv)*%rwkc^ccEOvh>DSEw0 z=d~1sW*0fv3PLfzE05^?&FTe{XyLjE-4X=! za7ype&3*qbOs~#4ryw|n^wmef$s#?C1>X~8KWm>(4HRK%bjw!)2d7)(RA~Zk+X+3~ z27!3=r=0)Q+D`tU#efHcLrvonuDf`O!lW=!j_B^*!8uQ zbG8?n8tkUaB=(yw3>k1OpX_-m#4{?#VUciTv)tdFj3g48vG=C&msVMgzA}%d5}AEd zRKj%w9#()C>E&n@YFnEb+ctThjoq3J?>J^$$o7_) z^~DwNwR@_JA1-%HLXAt_)`4A!9P1Foygj%mJ}w`I`xDtB=F`P)%YPI(E;g62rt#UU zdtD;Bp*}rPlsQtL2%Jxb_#4ceK8Q^N`ha;aezMj_2k2Fj&16A76UiGqobEZtmSQ_e z*gAA2`s&Yj$u}XgfFJnh2@;??92PqG6~x_qu?IH;&l(8O=&}VNKL9hFW7OhrtqJIc zVj`V-P&E;p$9-3qw+SmMh?j;SIAm+r=X|%Ec!0#n_j@|gmwb>xe5_1<*i)sQSI>R{ zZegt6%@$YvQ?haxU8_$Ns&cf*c}hVfEL6njTobFHF~0-EbPFmmXwcrC$fp}h<&EzM z!yO2t_Gb?NOaXK5)Vk~ZL}(LByiW3lP}uFo(rklAX(4(6Juk83jPNqL<(LYg7u~G#k;3|sOQXzsFokQoje}~JDn! z{lx~2bxps;^e~1*=<~a`!DvnH7!eKSmlR?K4emrml1KdZ3ypgBN6`UtQ4}z+sl@J| zF4qhs%BYOGqFlJ1+<4Btww!(Oh0VL)khAcxafdS57vOP*+g+T;I8m z{ats8%d8hq!V_Pk#oKV_YSZ%1 zL~N`~UCPq~sJo&5rx>=zRi=m>g?6IRW%T1DQ+Bj=k8omR&gc*^0=uA(8I-PLSlMN& zRlHlmFX&~T(WHQ*$@Kg6;-g%bqP2ubv`OEZz9A-Xw+Sq|Z{zcnb1F!YgFFLox`uq9 zb$eWK#jVhdF%zk%jXz2W5V)7VYEQ*ENpPGerV8A)*74SUvd|R7qWU~Kg*tm%@H+e= z*myvnzEKPXW0kdNQG5(HY_jX&K3EvuwK7ko`w&!3QoG==Gw!_Z1=;b-Wqx z`{<5*`h+4qsUQVS{oG+cevQxhjAf$C?aFc&x7=muROp77(=7D%&7W)^h9vTRwM!PKoUobwPz zcFen&d^TlnY`lP&{Y^HN^Y)ger*&3o_D>2zCKXK;6sn|mx4~%bx}G!WdsY3O*zE^ehZ0lON|^R zF#=fuSDyVvY9;!itwzm7_;LZ>8q-!xSv#%qn1WBTEPZ0Zm)MrwJ1*lzUxjnz z{s0TRFF{o?1G}fdEWva?j2T9*H&3X;6h80 zYbpe}Hl|Ordjx-lm`7hay0%#iH+v9g)@g`yYJ09K(}4Aj-8K_ zTW+QIGEptj15Pg`+eAlo>2aFS_201x6iK&H^Q|k19{1AOV8KF2xX_$|pw%Xp1fItk zD=vWEy7kA1c=?ZNM!AIX3jEXYRr3fU>aB-d*4(ux?<}=m>1SOz5dAmwCDbllvD>oQ z&h<$hqr_M`*@E|2xx6OpPb|!h_4RkxLXyGy8mrxrlPW3U1ja)j{aa9J$#utpAQq}y zm|fR>2_7#wjCp2CWjA!d!)x_NyjZ$T@(n#@b$MYl7X@F31bNDJ)D-dNeEFeh1qc=M zx0a;Q={xsbDf0_wi(1S^C;V?6%deIfG47oX3CM4cmYaw!NApcOKK6V48Bd9cehndp zwqdl*@wxpXpyF8w3KDaw?1`Qy=iKa}Wx4%e8iUKU*79&y`uW=nqvpTtjlU!#(WSi= zrnPYbIu94hnRxxnxG+)#waJzh>fz3ZiU$=Hfr+8B^ox@6-s08$Z*A0 zfl;?JwdA()wtBQL@q_ZwE~le!Kf0`~wh?u6rVt?Z>=!*#KIgArj$+ta9~ttvJh*N@3z^IetGdDr!g&oOvN8@}>U8?6+SqA(~-+syXJ>+Q{Sz*nmRVXUZS-Bj3x;Du=RQQBIe1fKVTJ_JIvr z1)Wz|0*jAd!RIqa=3Yi!5o8SG#z1r=L-j9QC{I?W`xG2Q2zTT-Bfz6%MbV+A!R0vJ z?^WOqo!*Od_F{At5(GW_s@`bKc?E}xk*~AkL*y`N$-gTdRuxB#i)`U$2V}Z(iIrmr zxbg5X$O!Ua#EPlh(zK-WtWM)n%vcqfjM6QGm zzT{lNcuJMMf(W=Y!Gr>l_%cFq5Aa=2ke~X3)@V9n77A3=4x-;Qm- zMb&8>QAwMus|JNRHJ0bR{R^w^@{0un8oKAyxpm~-{10~}u14w~dZVy})>f(`)OlRg zy!Mke*5I$EYBSTTU6Q^miWu5#6tY)i!sA9fd^s98_Tz599uIUdP$eZH^!q35HO#x4 zNu8IDky+9-wcjM$vwM)J`o$*2-)s!P_n2x2swl_JcD@!LZ*++vDvnBD=e`<4baSYe z>Wa)US{$UZYg{NLHYJ}c3_pb-&KqkLN47;6ejbhJJGXc0-|0d-hDE1|WFPjpX7P7C z63*kr?FbL=KT*#4{X7Gt-%>&frW9Gim`E_*L7R2R-u64|(c!i!E%%uZ%~0p^k!DG& zE_z$Nm#&rH{FOS;c^zb|~c zAULhH*!KylAF%LH=^L=?Q|G%^p*gIlcWgFDcT>8Xe8JbkG;QCX>YhGFLv8;c2P3b0 zxt!wwG2*9q z=m;dV`olFbs&S~SzuLTwT2ZL5pW-b4mgxI&!ZK(BMehE%Gu#D_b~~PN(*hp1AO)_5SirKT62Opw)bfMGu#vvV1 zm?uEU17gH0X;Ug6%|{sz#EuyI9Gw)PNldN(BCM1uDH+~EkBIMA%p(_)A8P0Id*gfC z{+{dYQ-+J}Cl_*0xK#~Bzskst7D>;;#Wt()%$?sZ&Z8&dA<$*txOhe?&BL_rg&*?L zo4wzt7>FRsNu`X7@}1O8V9xj9w_fL}wz2g%A)ktW2=sTVNX~BkX`oy=NQjLRsPEgw z#E^mku0zc~*oq3HlOV_mM*jSHq;*z3yTgkkr+2nT6kUjM@2vh-Hx%=}K8A|D6+SQf zRomlI%!jz{95NrR_lQ{{9j|y_<+mHxV2$!0ERolK2};78Y8B5P`sov6$-HcP<4hIT zdMD}p)4qymxX%fecX!qhn)GIm<3IVODG$V2DM*w!J-?WN1$Ze^Tp$D6aj#oa?|g7( z(v?xT?w}#J)ULivK6gKJdwiKWrTClK%X^ysqbc%xR{o6^n9X}*waYb>c!tU&$Jav= zAGWmQc1TJu29^cfS=_d@w$k%8MiU6e1Ijx)&D^C_xtc!5MupQ_n$pJ{g4+pSs(KcI z9}zrE&?inr$E^eQ(^`7U-0w#q~<+0EmYhq-OeLzl@5eeuO} zs&>h2-C_b1IWBpESWd5o-zxjn_I_2gUgUBGsKyuaz;>h^e_jr3iL&d%aga|Dn(Ae%pq@*w z=VTRB&rrM8C{1~^XwY>KN1^ZBdsNtybMbjWoxzdlY~g9q`%91N0L45+hS^Ez(Ad_7 ziaJvOWsc)`zOqf({rje918>^*D}8YN<3-x5m+Q%R5v0L0iZklnYbUdx+@q&d_66y{BOE)?t#87A-j|e`D8t za4h=O+p0iGh)4Ne;33T0D9j?emIxlvX>iq@77}|XW8yM)e7`TVI|R=g5H3!Snoo8w zwD*Z%Ad0VLYbpLg<&+{if15u8uA`fxj1w_+|LSj6zde3syOCoZlzSXq@Zi*EEi`oJ z+~1i)AEgo(ACKxtdXa#KFs20{nIJjBUSr?O!*Qmui5b93FpvY%vFFW=+=OA|rrg;a zWKbZc&?eH6ZOq1n7In;iY?SvBIk#UOjf@l1_iuaWV%uKxJH=g-S)ZnM&$8G#w*K5~ zr`wb~M{`&`j)^6j8!^$PQE@V%lK-}>Vpd}3FT%x!iB$kccM-&KGqRuDkIGso7bbQt zOHLSIcOiQY#3c%id(}1D(*TOvCM2*M=gxtT5FgutIqh#9;F|vnl4b_XIw;Omgh0*- z)J%c4R8HRLS!l^r%GJ}*@%@H0vS&P5W#Dk};|))-OU|%ip5T?Hcu*jQE+r20k8ylX z6>R{VLFYm)Sw`o2T|gHk1z8%h{&RP+E^I}5GHT@IVvre@jI?0SwJ}YR62 zszNNGCn8a&QIIn8McX3(1C_|Ai72qShJZJnz^@|RJicZSJhcEoKqWrtYW1RG=UovQaQzKn7X> zLWHav1Yj1?U_8Vj!uVUo)jY-T^Ro*aQpQIl!|7EZP8gv=0u3_3A?)&S%Ii=kvlal} zA)KH~{c$06Yi#deg*DSRM1+sNJtQ80T%5*o9Dt(wAXh%Sa?W^gyFx1k;?IURc9=to z8jkV@0MrY5O#tAE3aJy)p!W)`Qxlj06zBnFtIwDkq(S&hf!11r{m7P*vI4XY=;Xkd z%gKRU1qhC7D#z*Z1VOHlS3Pm5a*(&+(?x5@@W)a#oaKtI-tvLql`x?B%ev%U42+P_ zz}mKtRGlF?Caes=tg#@UkawLWu%3|N7&v5T6RFev`T|HngYjosu6J%&C0_gxylDp> zi6N7NjDeSott?UO)25tSNB^jVO>#KmQWaG+GCpFFY=91F(>}gC;UDt!eyd!5)k#PY zE)QCOJGE{ZC(toEvPCim{bCRgqqA|u#-C18poXeP{vG<2`^+g#=usB&jn5AV;cUB9Gcf1T_Dh zR$hcY1_}IpUM?_gkb{u|dCyhgJ@+MqcG+KK+Jpcd1IR?vCdWXJX9V7TLgdP~vO`tL z|4F`8xLASk&v?NpZ}j7a9?u5!m6AoNZ%OXUEBcr};J8u-*9*GID-}<~;?Y=-4l_4;_ECVs__V*d6BS7z;g1$H$P61Pp6=*12 z;VSPoedd{?31Vh6=C>*%e=GH##EJ~C zb|f%f-yf_VwRwsePFZ2l>!E+3&?Q>&u#V6VOplWRJTU^`Q%TgFAMf|9QZjH~n|FV z=BWbIJ`MDa{`}7JfG|C99@6H(8M~A+?MEOd0P=p5c=od0JHyEs&>l!oAW$bdIyu|F z*$DAn!d8?glz=6Cj5nMSNeO4Lq))PQa~5Nk1dXZyY+ThM&8#fQgx2^X)RsRSUE@Sk z5g7h1a1|klaP`ckZT}_bXGaWWx}X$Bz$kJZsUw*lEeUAp4?d?5*_%Q)fd~UV4GCkbjuRim4-adT9F)jwqsNm5S5+>LOt2PSl?wz$OQ#k8^-}(S9e}h# zr#Z!Q6bQm!f-HJE#!)a-ugX7u$of?7-ecuh1i%RRwjN*3Ea+BUS@zN{75pP5Fv@BgMt1p^ zQ!IruAS&g=?P3ri2qlgxxAAyUmJ|6&85oY|2M@9F8)h?(KIT(xgeG>O1@aI9kb7|G z5>Ex+qnGUD08q7Zp93qe(?>Vh?eo&F&mw=$FXU-T8U2^y*Sr`4Os!D$jUr}w&?h-@ zS$@~54i=OKG?MaLFT#Nup(e`&x*$1%UW&f%mGWZchUnfU>wz4bI@&Qr+uYxJ-Fi7H zYr_pjt#xg@6)Z=-kl!o?C*rLsc93^Uu<&y;N-1NfHYnVx#kMLM+^&k0g-rrAe0E|P zaz#PBIf9w72rQvfv$zR6zUe#JJDWSYg6-}?vaCth;`;KCH|U)0_-h%{b^;XC%6c4P zCG&|g8mp-YGnt1L#SJvu92P9|kNo7XW^mm8r9PcZ$ilvI3JS-wFrl#aw-U0lgI>+s zDds`}m&oMAyA(C47^BGG; z+@j3}Vf@HD8)!yqN#pqZ?K(J8bxyzBUQUic&8Iabr!?%^tSo zn5T8Ck9@r`el5b$g|$xL;sdQJDC|F6{Jv_52yI*J`@k!-*7+67S+$#33C+s?O>d9hHDHMj>qjbX@_z_W-&2PeFW?qA`6DOg}Y#+C=J!e@q2jRoi^ zhrhGy^C`+i`)iQ#^L0N*6!ZdI0%Qk%M<0rrn82uf&!H-EmTK1PbfIl1xI|($ zl#1%SK8TU8dxYb4J|iJBUr-`5Bp5*gWr}CSPQWHl>b*Rg%Q0>v;C5W_ygKU1KOTTq zGMIKNsb|YsZ_RlWh->FnTB`ORE;pH-ZC@R4WCc8rt*cOPx)vF$I}wUwQk7t5T(7a2 zw0V)`cb-)A7N+Cy`}+^!TPOaygJwPsz06Bl$ajD^?iuMlUcBg^w4E~RhzhnjjFhGQlr?pblvJu3v$y)dBLDTJd`k5=A678IJExcVH2)9#xPy0XnT+1 z3gFq+>?dO-b2Qi;yE^Yqc;!zb2xrP4en0C^lYBE#O1}X$Osby4LB9@KQw*qExiUZg zv1s-;z4~}N{>(tRZS>k{j60{+d8WO@Hxn>>;K`;&79fX*gb}Wz5wF{>-FSO0OLfm$ zR2htKIl0S5QHwFRJT$)r$VOHnce~d(NKfv8&jhufafr3PpJP9J#B+8#KXr^%@7OyZ zE;)3J{wk&b3xGwu|5`sit(CjKl}=2Ddm;K6W}5wi&4$yZ`xEl%zjB)=U7avJAMPSl zIM4KX(?3(0XSM_91;Aj`t9?b^Zp|lTKJrh_e7s5>eWH!h-#N(pn8R zYE=86x&9io)R#uG#gGOnre-9~ba?8&5wejZ)%Lew7Qv>IbGi7WH%s=5(R!0?j`P>{ z<7|AQzIeJq;fWi?x`T5}?h+lW`H&vdta7Zx%AoTbH@l)W@8}dJ$AJ5vPg5I}7Pb>o z^N1iIlswB@@@R!+U)Wk*v_(G4TccqE$LYn^JnTjG#RAQy@w!h{j@puiZmzo`&Lz%# ztk0u{e@N^ZyckUM+X}e8-#!i?BpyhavHeRC1>GpoQ2U89WLy73!J>}OAtYV6|^gUjq@l)Mm$ zExl7lxQR@eo_g4aJyCl<)3JNqcBfTmxZX)QUfrJ5njf!8jOCB(JJi_ERLq8RyCyr# z->X(w43+dJK_65o@$;?VMNMgq&Cf*y^TMem*w`@PcBe(Xm50qTJnB75bNoSUTynE68n7F z4q{^BF_iYXuWLYOEm_$|YXMvOxE{OgrDgxGT+#1%u`%M~AztK~lkmub=V~9>w)0FP zD_C(yNo)%YzyC7+Oq`CL?VYGtxSAv)J25&Hpmk3o(%OEGEkoav#nTn81j(s#TPWt~K}0au=uSs%SqoxkH7mf#C{rgV^n_(%0!x|V@OeU&fmK3^1$oaiq+PgjoB-2r zE6>#RnwL_No!@>|1%@9k1DVVpf9p`6pZm7iSB=m9S{gBw#xFrh&TCUlOAxLw@oV%o zf=1z;@?;@@%tj{pnls;P!x}px20r^)V#2fM4_d!o8(g2I@mcqP>|)>4?Kkrl627P7qcT<=O-e`oh}X!L;f*XNWl_PKM)O6`*!vhi?EEzErHo`>B0hPib=O6-E<#J{i-=-SJyJT>F^ zqXc}D=e)L&MSmMr3*4U5;=&vBuHCI(IGHsvOnWcs78HAEDZ4rEu4QaG-iTud?|H~1 ztm?F#R%q#;=Ti%K>#DXtGXx!-NK+fCOaNgVghsCce?$$ZEhbqQ=7`4I*XWEudN!93IGmE*ekT6ufY=a0ZSQ7^5@ zUU#`Enx;sVqe_ZQBAbpV%rdhvx3on8c2H zxyAB`iSwpKj)fPM$qzJ(O>^DVn{y@j_H6xSQJosjKOPP)wBt!0gnD!j$__S{BA_8pMIoUqBqItDKlX+-<@v$z}&Oawb*}*s6@3O)^7j}A^VS*3K)ix;xr$-yfiqfSRhVqO|=l8{T zm|*whV_a!!XAS%E&dMiuAs)yJ&QLh?yGyu9DTTk3rHKU+)BX=^t}{ zdct<{KAx5_6q)ybBR;y>9C~86QK)hMcP86%XTmEyg)x`V{rj~m+~zEpveDH$+h5x4 zGiBqT9~>bdIEq_WqpZ=9HZPj;y#~}RNLCbK^7#d_u$$Qrv`O3}+anuv=_i(5luh_L;o;LP}@gCGeHYG{mO%O?AV#(xfxtnzS4` zy;%r~5ZgT5JjJ+;n>eb4-@lyf)^twW&c#O~AM~MXc&COk!2%kmqX zvpP?r)laOpgS)N8lC!WHggP^iHXI{Vn2@B6T_~rGJ05fGVNII?Kb`y#>AbXQ)wXg3 zkhv^j8-rI7HbK8zJm6{A)7=-Jzf;@t<9&KA>X)PbAijrLVzFfB=6gZz7P+MwBk5hZ z+O&N~sq6CIXUb&zjzM=Wh&{Oc=Gg-KijE#Nw%i0ZpKv7Go}Z*=E-ca~)1O$>K{j0F`t#M(^xw`vjXy)Y1zJoLwt;&EuM zODbpIFSTKMVOKoYbc2-qLbY8MLPHsqD%%yEmh&nD!8AUrY`aC-QT?atJVR~(J)RU^P!l*fd;gvqo)3{jkr#CE-;8U&L4_4 z%(ZFATU3}j8?xs~Qml;4MN7RSBnWO&4It%uJ!v? z-)K8s6c7JodjiXSWpdXu?x?Eep8G}bSAp?o(p)JT*eEWnv3Bi~Pa)ZkoPVqgZ$C!O ztT`q%a-LD{a%EuZBMKFX^knhgzh-(eC`$KvZ_kIw(Q6_1i+T3QwmTY9ZhGF87+zbP z4&*Ip@tYjO4hN+|GCsS+)F+|#E>k52+4aG<^=@em^|?I^q>&^f`s0!O@EDj&HyL!} zMuNyBqZ&=cglx z^o!TS)W#prIMu7g-a3g2rcWuoJsk3T*Q)UwZ{V@GS@s2nhP4ly)@SDqs^dO*ldTN` zr~9u{-c$`tD`Ps|!W|i=G9LV5u6Bz96Js=%hWe1o`CZ<#{QDVf%u56*%e7q?_+4_~ zyk*!vwy30Q-M#SWRLrQ?8$;?LZo!fGA`aY((P2iHw9MV{h;DKoPQ{bAgef9EU%1K~ znb`G3EgQLRC5hAoyj@$oHTZvEWD?YL&NunQr#gq$hTWu<5l#ZjT4e8AQd8OdDj_p# zzvf>q7wiHJQW!02892AE6r6rHJ_=fTuD72a zSPE?0Q-N(eeT)@Y7`v*N;&kJSp5ca``4Bg}!m2Tf*XL*K_ZkdazW&6*f(6WyliesF z&?b_i0KMn?hTEW(2sIiitVD*rfXE8D@`=+`uzaVC>iuD1OCRG8zC=y+-<{#c=oCc0 zyFw0om4!CPq&h5L<{ey2U2u){~HWbaH}E z>jeqycDo!}0tCwC*aYEZpj(2-RDB6aU}K6dFq6+ZJsiUQj-ZJBHvMUmvCUnDc_F%g zid?G#nj1KDCVUDg(rR!#6OKn5Sb{HJfNCJk5a- z68wl9zs^x12qysD(niLSOT3hWbfaH-~+rJ$yMvPMH)dzWInyQ7YQd z2D#G3EEvsh!Qb;K0hp6BXkFsB=#;E4eDg${3@6tCvgcJus9fV&8E;OHhYP?;ogD^d z)N-vKky@1OzX4|y5tVF~@ZH(Lf9yR@xqRxlI8bGC>&)0M0$ZtdxVmF%AcK z;0KUBrl3t)JO!rq>#Ju9^P|#{a9~Sk1>lD~7*z5Q86cOSj3dPmV~_x(btFsR_orPD zhEyOR7rM9wAw8Zv2%Z-fYnNsZguI=&$aaxCEHi$?DJQpXQn!g0W+W$o@A&Zupl>!p zt3HD%@RyRS=#-uTDh-B*SSNXHxmiZ%)nBUtwT^U?e({4d_0*hq1~HMVyqd2>1$;=Ab58Eb2= zt-j+B$O(hKIMFYNSVo(s1YWWH*ng76XhR?joFJD=WzP9W7B?^eTaN>9eQdb{P_=>c$))qjRi>fO0E8YA zgoF-0t~0&3A`%}j0{V*g^HJ?KBY)2P2j#EHKn3|vK$rG1MNlHo9K;T^9kFYkyoEs7x}hTL#52>^h1 z7m>t_oX8DO*Av1HuuMk@Fl17JfmbKp`zLZ6z_CI~APhE z>GI*=M+W+s67X*3Gr_ALDmp)B1Rjfr#0Ji9ynCMZ(*_uHyhn7GHbzqt^qsyHI{A7m zn*&HS#@Gy_ynbFJ%5+9IlSJqPAlCGF0sxvGRB5VA5YCM3eF z{xe&+e>Am$lyE#CA3kWvLavns+5f8`gES1NQ4YV%pJRD;TS~W7WX56=A&+yV;^sZx z#tr;-`Z$Gm2gvLYWD;amqt4qgH<}edQt<@*!_ztg6E|z#N8O2o2u< z2V%zq58?`snSOJ0So|A#TGsVfbOg|ebVA_dTRIVgJfsLL?C0fuU4dQ5r3q{T#X%By zWjDfDJ(gb$gd|di3V;+7fJ~rMGR{u|-_o=~l>@BQk^t)$!3b%8e|Yh7L_h6*TS3*` zE)7hO2pG9k)5)&K1m5Af`PQBypN*inB*2jnd;mr?mUqOPl7(|YRF=*hW$uypQvxVJ zOTHJt>mz6nj1^6gF$IB<+ueJ?+gNOqi~P@Ef+V2MkfyU;PadGooQ<6UUb5XhY@l18 z6kBhv%tfc9c8^_2qni7NDz5)dZTt`uyo;UOI|0ZYyyuVe@!pi306jIZ7Aj<9R3O2x z@{f8iF(Er<>G>CYCOh)n)tG3_p`kZ^q)foUdGvTJpxd~JXVw5sMUQ6OIJF>Xec@$U zVym;SoHfoJaCXghFoDw#%t|)k4B8gKk6OTFWCh$vK>ylh@ehr}4B9%XLu4!at_xoM zoy@q9)t*`gNQ-}8K)PagyXx11dES0W7xrm8s4oI2xZAt2h|+~heqc0?q+0vL#nkWI zY$C*>Lc}U!w=KWzy{&WBd&K@MM>3ykqVB|s_dYIfqzWF=Vx^1h()zbq!ZVJa3zvHP zTq<^Jn<{peFaQ#kFaXiC^-k{^?{H&aU@O`3`ZT2Y@k&gf2jD9k-u+&m+VV?)2I6yFb-*L(4Q+Zm*@M6pI@GslfFhN zl6q9=_bNzPKUFfo0c)366dNr9yM~>QR0G!pZKQ2dJn7v7M+S8f!p5y~*~yp3yzch5 zdor79)^y(vHx*zh|IEx7I2{L0XHPPV73P9Bbf=8lZ5Iy)G2rpz^yF8PAQmMzI&`i*IzlnJ!N&Q_Ikt>>;#}`jJcMQD>BIM!PAzhi{%5%``*p z#i}URe-hsRizzq@82MfmG3OnbQrHi76ti;R<{0icLM0_Jp2h&i{1=n){VjKvZO7iD zZLJ@_#@-zg9I{7sp<=un};0cb6~W7DL}n`%~up|p*pOcC;d^SvDdE!7M`#)sPtk^ZDawg4x@ z>KYJ!1Ne1D{}Th_{hiN5E^CP7G?pe+F$1;ks69Gh-!u7RJG|?R+JCy^vZ;vnVlhh@ z!fz~20`oc!Pov;iA0mgcutfI-2^+kCA%4%KX7d++#cByH@kcX6!)LcQRHzPd%ZP+=JI+Wf1iW zU=o5B`pvN#j#roIDU7}xwTAiuo;X|ubU8Q{>_|bl_3}mPjn7J_iP!qqS(`rwP2MOh zddD()P2QQewuXIG77Di$-poS$~2a4)sD|zY`CWz{_76 zKV>9!_>Svc^UVcVZ?e^M1l7N{7gspUj~rj`uqPWZQyrw7O;g#+NleBzf`N34L&CiSni<-u=aPGhXvjU7%(g1yuMMTuME zVFPgtk)Zr?SB~)~ttZWbr#uX!$_#Vf;cT`5~=L zw|mfVb=}Dq=8=)!&82LFkQ|30vCG*{UXxW0-OXxTEJ5Ol9e5he+_tYvy+eO+7cZ{%&xl(z_2+2KB#BIQ*siYM798JC zUaPou2Ixih%I(N#y`B0ljG-e|a0~SgMBOEKr#K!kW4iU*D%EZdpJ>caF~+|ei}ZQB zM+ZWc{V7?QTGFU1+D1Z?5vki8T=jSaD{F+)sABzi0@=^%GWG1E%uhJ?n=GlJ_rvM$ z`;*|s4~PmD#iXLKUuw2f)o0+?M~>oIOeL}b7Kw^FD>=XaUtR%OFtX|`iu}pC^6N?C zm+1_*o8g#0*w#-=1N?sc;I^IOhy!jNzgc?vj75ExsF-tLg(3<){kfD&%=Uye2midL z28U5Ia~j{>kK^@`tY1+86Xkq{M*7GE_sa(0%S*88^SSu~e2Os3SMIhgmE+#?BrUesB(j?De_OTGd?e5`4IvDUnDHrG%cE#5&R(}c3sxzN|G z4Q=#xRuzjyr>mo^q$%d9lsjXDEz`ff3(Qv~#sAoEjycdjbd&uijD~gl!x>w19O1pn zS*;(0w}RWky@0URWLcqYsS*A_k3teU?{a-YLWj0uDRqQbZUSA8foQi&lbBNOKGs_7eE zwUB#J30&0}yDI{^Q2LDa*hFQf0>J}URf@~bwgdPu66d28nAcX0Dud;4qXpNLR}_U8 za7Z^BAW$Z@?(c*qBk;MTpW?(1PdTSlJQ5;xPRm6Uf(h50r}sBm+!UGp6<#L_TYtW% zp6=hpaNV)G{r$0n>Rh!z7*Gcuv!LHGi!w7^w%X~R+==inTW{wu`0XieZrPtQ%3lB7 zyxbqOwSv%uWgP5;cda{lRx+Ba4}XDP84q+!;!cNFle2}n#WtIg2)a1i9B4i=FdrzyWr2qV10O#nAoKwIuF!3v>8D}j# zUssOw%gocgxeNK9AQXRFL67|}sqLY@0#v%S1akeEBFW2T*(FVZ9mqXHsjg;Ve0=(% z`R9CgIpv3qrCb@dWiJeGi(eRt>H{~i8)5{E>+i&WZ`O}`Dk<`j6`#J-2>Lr?ppmIf zs=Bd905^hsMh<(oII7`5t4k3Xy67a>Ppo*OxW{N$*9xIjBa{+6U2$dqA?bKMJ2m>r z_xJU8`wrUkT0zo=SH4$`RSvq+N7tSOKFk^e+d#bZ2A=GHTB^5~Hqq``G=I4g=B8dG z&f!&-d6LIhUu*>%n+Z=6k`Myo#CBy!s@Q?CKIb{A27mfruFEcQ(pVfyw?BI@QXMP3 zlcTYmiJQ6Jmm=6S0SUE-vjK>;g0Z(uLlrnYRCZ`!jYWQ|<@gQgw$i zgL`~?)LS`^zej=iegc*PABrcRvG7YkNzib1HFI>+GoeN%L|uWB?@xc-2oPYkw-Kqx zmcKJMAX{EDHt0(>a!3X)($8fIo|)>@i5_*S56n>BA>QTDr5?XqSVkI@-PX}mikD@N z8Q=K(@S11W$2L(TkIu3PuD2bYw;8Vf*P!^zqT1)duiOd@rep*JHNPjVfs;ozL3PdN z?k{4P+{%gFL!9H zrrje^@5k9OO}YD_3Sf*|VQ}>_BZqRh(JXMVD<%avQ@8@R`jAw+^FP8Cq3JEgl*8bs-k?vm~X>3?px z@p<0&{lB$XtOe(sIdk^xnc4f=*WQC5-p_I0eX9D?njK_s=VfxWuXUgC>IpF}ffU97 z&0LVcj@g{LRW)-qp&BgLH72})U!gNyIFs}$emVhL!*c4o;vA>0C{6m?ifyGf0+vD+ zp;i#$iFh@fOR(XI-iCn;vJZarsuif8C`KDvTon)1YzAFfmE@5~IWzHPpuCt}cPf&+ z+;e1`CE1yAs!+$#*d8HZGj<%qQ512{Gsn``33>J0o-Zet#2k6F# zb~1^yt2;f;dwbj&Tik=`qr25UNT`{Q+(D>fv3r}YTeV1>np*97)!Xc8_c^rcGI0|V z10nq4-6;lO{nKlI(;4%S62`p^{Jg*No_L^PsKeyg=BQAW&~ixC{`BSi4v6qPjq}d# z01V8pzcar0*el23vcdG&j)Ln%a1)Zm`VC}9(0F0 zdu`egjx#2-Eq*7=BM|IYZoUL2aXva37ae%Pt|7tnp)BZ63IWsWybChgXA zd`)b$?CQ0yS4%nWeWw7ABB~eiqVeflXe4_Z55|e$evVZDIC@qIX2E#|bM)oUK--`Q@wHmOK@gNNIKatr(L|c8&GvBC%DBZq0pQr!l=gBbl zbF09tJtqRvCm=80Mt#_|zV$Fgw?#R}(M)IZt&mhmh<25Qtvv?fY76E!rOB~|{mw@! zjp7m6I(0)k3eF3GjUBE#_VPOjOfsG|=i5=fxYlC6Xm0|A0If{$2o)nhy4Z8|w*e?# z;>u*Xq26#&(E8_S-ew_?tvs9S8;eo?ywzp$e)0B#H{!(O9c<^(XA77TmM1@v@m12~ zJi?k|7K|jH?}y4?;XgJF*%W+n8BjtAr{LL%*OfP8t!vc3H@Grf%hHM0L*;CHw(@Cm zO^|rR@1kL&ID$h)rhFp^YChV?)s#O zQ4o02@XWSUd8N5ud>>|m{tG_i&f9m`{!dl$f=om?reymc4QV?vC#?Pml2R24G&kOp z4&=WJMC~gU>wM-{%;s{MbiBUWN;l|QSsXXwQw&Ar6k5ekeVByfF#jW>JXN^MY3dZk zzNG9XVpTz8fQCrBC9N8Q?E7pzULpH?jJ@xfo9_C7$D2xPd>&lPiU-&Oeg&p`(ukJG zsO@{Wogb+V{G;H*X=dgc42l>v=T{LEh_Wz(6WtsBBIeB&N-f_aj6zZO?k_G2aQD7K zA0RhX+b#I@o{!l`8iywolRcvlHD^-^{r=>^av;8>?hl7TCsU48+L+EyAVvNJA@j9V z(nX5OtLGP!vsY^%m})wNDa@DLm4Id66kPOCvt-kuwG z^8|DIktY;q$3ywjVofYCTmGszX_6(8HwQTe}N2F{xhY`*p+DC!QI2VlQ6GMB!+Dp~&~K0ED^<9ZCnd(GT_bntZcy8XuC(RnocH z_sOfpvC5FR12kA5hV&Z%Wg;|BVTu5*=Jc!OHMC7q7#a8%M;!|L%F8*@=?80OLUEjv z5{e*tgLBkdHVaw5Axk)5J@kIe5~%BSd^;Mi;bmB>G7cszRJeR9)R(~P@D{?zi87F zcmIs+H>*gpxX3?A006t^C^JqD?J>Z`%ffXmVlY0$-tT)uY7M|@u_@QSq&wNGB01Uc z+wl^G7VBim1#kgY$*~w#$TeKjf=TJgy#GQj^X|~P0yh7RlyslnqiW(g z>X?XOI9Q}}>a1FyX9M!s#BJYH!GD%|kqw|s6VTXwRj)I>3G@1PIJo9r|L@JGASw!3h705={kXF-V306h3ZuxZ?t^T!~21yuPUl7Qia zpGZMcIU(!!9X^i0GsP$Znw1Z~Iiu2oSN{L5;LB@tmvi!9{Z^y5>~r0&NS7Q(213M}NsU)&#z6w^)#TJ?dz z`bbV7UyQ1Sezm*_ur68Ljp8|gk2VQZBQWnWf?mW9nqd_ z$(htve=Y-pkdW1b8&eB<3j`w9!8f;3Wea>nH#8o-ddbvq_?PBnN)RGSc#y{> zN;;(Kcefgol7f^3cbOnTess=!eIvh?Ts#o;+6D1v3_fqUn~WsLAd=F1A6h-u`IP05 z$cZPdKX4rJ5_I(nEXVVVOr6jb;U=QOMPzMDJ0AKRlH>8k5V!uQ18=V8P4urFN(%q?c) zwIC6UVvw)Zlmxox%FzAqWBtMEiXr%1(u-yOXNnaWE{f}bU|S?@^wQtG4>re}9f!tA z=9G|tAMzDy)Arp0mX7?C&?y0e@ovITc+~K^z4(5!7gWN(`=F#!u7yu6{}uw}EOjc` z$9{J`p*&qpThTu|)Gg$3*P|-x0!5)LK}$|)Eg;MY9p^T9AEHe}b#21OG_Ury?S}Hl zRnJboJ;mf5Q%eh8peE@6NL|hT;pfCUhJ{f0I!GOjK#Z>2xe0#17AIBgXr7lNqDyFJUm-BwF(g88u=K6AcaV?9s z(+fru-t<|w$#|*SK!G~Z>3SaKVlbaaWJJWs{%=8q4AJ1)#q%9LmVn%Y|5q3yFhPwG z1r(&C7W6$-w2_GJRWwhXCKo-<;mND(WW7RF;VbvB6wNXh^ec1ybJ@@X;u6-U^L~BP z-JmR<3C;CJk>B_Io;q3JB%&`hoR>c4xyG|JkhQw_`_y@(fPElWiP~g_r|NKXT(#0t zduJdJw+w@-5nuXjCtZ_+{E3#n!*fxle-gz(^1pX~E7XDYw^nkmPeu6Egy;GY6DXMy z^>}dw@C?^SEuR8BD%yN2 zhv>7*lclg{5@BBnM`d~wUOs&$9ubXB_#f&#t=g(51gwARbhM71uHQk3PGxih){j`^ zXyzqX*(}xM(os{#>vu&dB!cRJ!Zu5tdA%%#lJ2|(Tfg_8gK&ak3-8rImU&rzxa*%c z%OF8xU2D!*+G@Y4R3y;1(5z>pGg890tXQC_TryTXj9S4E-?f%`s-;lVl4X1T4!W0q znF}%qsK2>a3h%El=1%-ROT5tHTx%S3&?Gus_=w?0QvCeia;qcLG%HZe=&ZeTto)G? z|Jc9WcBrX6v>xE=sX_<@tj3}9oDpIQmWQkQ&j!;w!Uyt3RYA!^ds#lKIX-xq4nK5P zVJ&z2uj=*o8y+~4rY;MCd!Uk`Zp8@f;J4-8OZkaPDdkdJDh%0Pk%MKTrC<1>*>#{w zTl-w|t4J-5Pf@62VdSF3Yt!dXt1pkw(P~@OC5?N^j9!kmC zY)xj3aS=XG;PXrkGa1Z`?i%NBRgkIA+{b4~pa2mrg_W~Lln1$llM~Vkz4!NNR0^}c z9B+=N3WH1yMXD704bSTRQvD8^z{eF1ntuBLQ?=4d<1^>3rjpiA7&C|zNMEa7xu0Rp zPHLpWZaD*=0egbhOXgv>2rq|cYP0V;pKf_oXjx4QF~sNv(PIkb^)n4V3WvmcZ+-fk8AuK1Lu;g(hvg z*j~Um-5dJZL|v7{%lOUOK#oGb#t&Ui0GTJmtjkQkv|ZkFmPwf6pmG@RXnNXIUnw!d zJm#GvFGj?zETe!BifOuk1nHW0`Aye|Ff8pNe|0~Sz661@hq1+z!3 zIn75=__^su+-f(bs!_A4WyZFQ4wjYVi?k+=2vt2((jJKhZ`;N@d`kRS?1X=V+HJ+{ z#`yjEX{Cp?=7mA$HqycMN@B&8iz^gUg^y=kzxO1|{DpU2mFu6yhS_i~#X{N2@}`QekpOsnk12$Ga695*CPB597&WmCdR-?iN*;4H3H%(2O0 zH5==U`4T={`TqNv?qtlfL8uNm&Q8C!Zob?aT(KUnvWzbw+GeD|DRJCU4*2A)nW`Ot z?@5dL(CwLMFtIG^RgqRb2)-~x9<20b2f|$pZ>C$$8>>vd@e>T6yGdKVF@E&n^LsxY zap|LfdLx(Id%wH+FcDxUos*V)=5-~mps&5sBz|v}!L4Y1gE8&TulsGMRoyKLFQm{X2mUKA@Uqj1aXPAK%J8c-PZ4n=Q`cbta%A*`sX z$ksVIlh=fHeamG~8*1Z!+i2XYGzWs&m%G0e<5Nx`RaOH|YY&bH=akhh@ROmd-4Of>uM@U*O#686 z^@Bw-_*~4%u_!>RHMOTB#neA}YAMcQB+WcgKOFDh$BdSF`6gu*F%u1far7BpA6I$S z%zVMng?1;sFgj;*uvZ7+MI`;$xp76a$}B0#y2F~4>JIFG+?)%_~(Id*aqOw1Ed3Nk>*m^IKKL!E6@^?x44JY%)5amNkA5G?~ zpE!a;7k)!p2ysepYA`{O3OlS~jP-)ie%)sU;C6p_V2Fn%ce6u>$_t(cXO~^`OVAfl zGdfc57(losH?(+w4z=#RSE=TDoRLP(06lbseFQk{|A&dT`@u_-)koTg%>__GC_Moa_U7#H%-$Djm^-d0e*7IYx}ZD8je(0&x~b?$)*+5Rs9LG|Yp zeOfbgX9NsW0n(Q*U%fSmr@%Ug6nVOIZ705S_9bzV^vw~|o-nz0nYj4?yFOHhBVqbZ zZdN$Ow#50p-bZgI7pS$PC)Y*{Zm6!1f@uA&mc|5ky2s;f#f9APBqGMI4I_j<0gOI2 z6-U@K)Wz^tCB>3PqXi8v+9TtnKCf=knb+@E;)~j%CaaNO({OF4k+X-A^fK zFqenL02x^b74l9SDq{do-RQ(mY#1(BYdezyX*fl{kY0cZO{xoVO=Hx1g6>??L)_m%U-&>QPxPesTTKlHq}<0m zIZ$CYvz*wwm9j~wX%k5`5kCBp46r@^do+B zbyFYZ&Q5W|LUI&p(Z00fa+`YbFdP(H1zmh9nG2mFpg;;$ND`c@w)r`Y;|_8W5IQUb zb&|Eo7roOKELNCWfyN-^u}EiA*-@6RzXS7mQ~3dqLH33adLMC&{A+Djps5#TSMC|z zuZ9E^?Im5NFXC@Vy&+$o9N$y;51tysO&-+k`&11nUj-_(3WRd~8|@29tRN(Rszm80 zC{)ZIj?5S|kd#sg^Ad!wK7W`gMqLvCgX1D=6#q}lu&L&5fCcN8!RqE}+s258^j{#E z@B`yv?Hr!_i`b`IRR$m#yRe;uSd@NX-ajqAqwsE;mg8D+C69)pWbH3|Y}73CxNi%6 z!zCyPZu?7Ftlwd`Hd9|H8S{d)f&20D45-n{qLce?Du_Ls>%sPC7qR0?-3t2Hmjpce z4pEX-zA)QXk)Sh~BSG@uT$l2sLmjLAmf{Po!Fe@z zB?>?}aw9rDJ_`3U$*3QnB&zNH=>PQBDz#bC0CbK};1XH@V1QA7p*7<@QR^pEqYSrG z^+0NQI0%I;ccwd*d?|qOEbkyg!brTzBIoq-Oe)+5As}GfSGhn-*5+>?HMi~M+lt3s zM&If)Hf!g7zwGa}OfNQ^Pkgs)J?J=^4RYFkcK*+A31NVLp(;}(CXoH)(_txJ0>|c1 z&bxtE(D81io#gG)D?Wy_vWNJ|UMSJVEGT^dVbtG_Ft3rr&R`+>Tq`N>S7#8O% zh_en#s#|sF8s*P=YBS7w@=43f6NgL1D~rW4N20Z5f9(X3w*-lYR8|_4QfIP3#M6uh zUCPS6-Pv-M?Y{z6BI$p9i3~s5@QfPOwxk|&My3AWFoOrPp(GKRDdF+XY>NKwH@;1~ zOA^lnGRC^!st@Vo^sE?+Ii&-ZfbS6Y%9XwLu=N0$D9EXyc9Rggcz_CemeSq~sD@2N z_V58u9})VPb3Zu(Ht$2U)ObJ)^hHyn>Eq-3soJ(1zBg1~Ys2n~?)prs?9p!3`(Xu8 zPCWbTlj@O2FFe!q8?Icky9Cr4F1RE8o_9zvy3<%ToJQ^`dIrpWZjgT0cFg49K0y!3 zzE2U1?noBk>kOAUW=_4Ir4JJ9-*($i0U+RFi7^pvWi?UH zU?lsSSEwI}$X;9JQ_9RxEwq|PMbUp#%GI?cqX2lVRrL5>;d(J0*c2NsA(1xpU!Vp< zZUaCcy3D!oGDV-#YjC~+g&s0{k36N~D_K&>F5yG(QP5>JtYya0ye&2vWTYtq;ih%} z_6|Ud9I8HQbUvWW&KC)Mo7aw*O2YP@RnghsSXBXFM~eU!cr%het|WK5Kde}*x*518 z&-IiGPBaY?V$TEq$yXD(XLcnJuBX)W>(-8NQO(n!>k8+Q#01 zKD??hihYFb4GOVkb=F>AW$BOP%RyLEY;1X2^@vLFFmM(WlCD2k$tfh)N-2eS^O5a! z^IpE%vZ9dG=#OoMm>&jL*6UWum=&fR3aH;p zFkoxt9z2(o#F7(uJ-cgx&iR^L^y>YlF`IA!Uvux&}*v}*f@Ef=oQYTK;P40*7d3|WJ!B-dSg9!MV(9Rv5YJK3J>c?ORQ z{_}k9A^ilok$8ZC5)IaIq#0Xp|JELk(J;QNze-y;p9HvPlys%WymKX@Cx=qag!o&7 zJ!M{^$?%@$nwqMay$mo>@Mi@%$tF@_6{dW1@_v;}!|$7YioPOs!Q3Wh)2tw zgnwkmVtQ%6p3THY_hF(#e3VvcopK?Oa9Dq`(tdxxs|U?5yYekk@TX-lr|bI zMup1klFO)9bz#YYFPknOyBT(;P;+*etn&>Z<3-kit-Uqi$nde{U#;}cIi8_{$*E_E zRIqOvaa)vj8^gO}!xv{$!ug-p`QmU0D#bexb~0jzD{0@m+jfE9dWH8`Q?vZ$4>H<#la6E?i*EW^B~h&!83l zv=b>5i#klLkb~8)BPJYvU*Y+%Wj)_Y;WU?#jA^hDdeM>7kd|vf-`|T(zT$vg)FC+m(H5s#6bE3m&xzs*ox{J2SqA7ldxVgaH7sAPn@!m&|%xrVHMlvGsz5= zHHBh_%`rWXbjFdvg3ZI4d37TqJGH0-?8@Uyv%2@EZ!0hLpKD5ITn-;c&#*RHOpDW! z=EmPbJPh5)3e{P(=+P*(B(>TB;$D@F%0hYjO0 zj)_I}^_|$mRri|xL}4%2u3;5#ES_xqkkTv=@UX27ot-tf7h^RnQ>fWDbI?HItz)4N zdVwIxv?(ItnHEPFhv^6x&Dd|(#~&%q`uU_5c*4eC4*H%Zc#(If$;@Kk2x!-vzrn}L zZFm9BCeEK<83Bp2wDXNF5~8W>$Hw2^;L@;i1$ecAe)I71vtr)L zniOf3Ia3l2A?h{4d^kVJkLUbN*sXF|eYfv%;wK%v_Y+Pt85y+t+RLj20q4!FLh+N< z!j>4)>S+e3&^C+n5EPQny#jCD2}GuUeLIYh_CXm;_$S^3;#@wr()bP^vxGG2G^l=! zf@SsBantxEL`e5wz465G#L`PQX+k*dP+&+<6c2UdO_*=5A(6@&!U_u;D6>5vpOb(w z-An5`X8o2Qh*+FHeh^>{!!x4Y5aZ*eSRa~iYy_KVm$+b6#jt#oqxOW_gKh0 z2$?1bDPdrPUIenc3*U|nKWw)Ryzz@Q@npz*Ue(!Q+`i`;8B6T1Ww$XC>}UADARm!y z>UJ=}Q&;4&sz?a1NEEpKN3Lw$Rul>gywDyjDw2j@;`fe+NOpFZAW}c^4ncjo_efU1x(m;b ze--rY^))s)hwkQcJW)A6JT!W-u%p%Q0`Ef(J~W{uo$6L&;6<7Ugy1t3?BKmaz!-nD zn_FYQ8%H4jQ!LWCc0XqO+j-^1FO87ELPnR@NzL|&Jf!p^*@%YYj)7SEzY zoC{WaBDk<^Moj=6TMZO_?&5X_XNFnTf}Kzp1`GUP~Rl=QK$BmO;m!r!-Mh4 zEy8dS4r6c>7}nb84(HlLMK1Nb%`t)f%achmgE6XFhfC6hwpj`|#L94s>D;bQ_{20H z5WkV_H1hRH>zSX;(4>hRMQYW@#jv9jO$aw!>@T#tpX;{;^)T!GG}zDLV~av%;1sfG zynH=gZc^aRz0c)(j5jz6PRC=mk&-EAeE-69L@;!t<7~asJQNl;L4T4y*1YmbsqP}( z0yuUq;>V8}o2%0!{q_)+#fs4+e|d=!j-HU7lAK4wunbVuy9Y~*^E1m!{vGuVI%Qx!4be5A{zJuG$DqS_yOhoD?sJbkwB_{}6K(P3xi)d>Fe1NjoY zXC@|3){}rz>3`N=y>!IO8lw?BSF%lsDQQu^_2l zd;D}DrA4^Z^@MT%7k<}L?M?%m$)Wx3T)073)DhqYxDHaHi}<3Q*7NNBe4L|RB|tIF zHdQ8yQZ+uyuQ{}i#yKktb2slP?X|GMRQ-;r-Dv+=$;sNap`*qN(%6e84n{sTs!78($ zKxOf!(7Bc_`nzdR#GlG|@_GB&djodZs6Sp=+Ntlm>y`4NK=x);N2JQb7=FF})aa!h zPrlK!ylbcG%~IV^9Rb`TwSxw>aH~W%+TH7PP?3-F8V++x$0v~Qsw8;IWkV6rBAu8z z-A^vHS!`eE{+L~gQqPPj52u_q9ALL_`}mcxs)?qyq^D35qZb2Hah!YX z&ifs;z}K<1IT}vuuE_dEZ0Zw8{OW#t4~Gov=<(*M60w_=@n-^5qw$=CwvD{_%IX}W z{Y9Vrn(7k!!NLI(y_aJqdeeh_f;1u|?uVJ>rlZ2!-xsKJ6jo&WFZy`Ty+EOn$$8tb@@7q=D^_*9W*=Z9UmW^9cN_jK1 zq1r|`7TD8_!rA1uEzA2{4{H3fg5Mr(W2R%s^+R_weYi^=88E3?}F@huX= ztN!F52jNAMi4Qt`mHoz`)E*&wen}oShr2|@jM%)jbj-XzT#LbfhLqiq>`w(Jx4}t( zirb>g(Ru~h%JG}Q%(j^EuTt?1rHrvEmp0iFMVo>rq^qSA^_k8Iehg{3a%n*qe;1_3 zYwSNfv6H0_R~~%gc|p*d9%`$poF-v@_~=El)MAZ$s^`m5!D!vJE9^umlZkSt@xVk% z^`eY(Fi|bm`#!}7eIVX5n=YqSjvK(}WzE>hc$Y39akS1x{)?;cC^afJ-*Rn;Fy}=t zGK}bxM+{KoioY118JUaKQdifIcsO@hOtooyx-stjvDKJFP6ENE8g3#z2(4N!Miov5 z+mD$5z5vzbIcBhUD$G~n8UjrtDL`tFC%gBhGvpU4M2MXl)h5$7-56vYXDyx7i_Y0u zIO>rO-?V)fy|x_b)e?wfaT>3^#;DI?MP04R!r=4C=b7*3>Z#1&01|5nGt<$6qF7GOE^7BADe~IQ zKg_r%^!P`mel5f>T6eo!8SHWN)t{cpQ`1b%qRGb`W~%1LrekYImsH>COdEb~42FX+ zQQ{$D_+myhWu(~IfCO$2-A`mHyRY4^?N+T)-lWMipPjCdNTgTIg1^0ACUWkLw9jCk zCtc>sv<+nGukG!3A+8ToDX(QN42W(`_{B-|if%OQ{%G0nX89#EPm{0G`g0{i z_CGmU+f3Sr=ZDq8xyrAe=gcPyyjz@h)DG(i-(f972Pa^L9jQ)X_TXl)zWBafQxeL2 zAMMdIRD52eElf2-4_84WCcDc^M3U|*kH!U*4Mz@(@(E^<8IN;>SU%4hP-Si040g84 z@*A>`$LSJPS+go?-QhsI!Qm?HQmCi}8w+JkzF|k$C`i^>Wp~_k3Y~JCwa?t1Z()oK zBy21u_B&6C6*xWH=p);baQIVc-qG2AKJH-qE}q^`=&~y_R5Lif!Jt+P7DN4;%H1@} z^YTa;B?z!<7+a}HS*y%>*OsK5T&zUwVw`QF7fCbR6Fsk0|7CVVSd*|w^mM-;RqF7B zu{Bs5o!<0V$p~i=pP^qjuE&ACV=N3Tq2P0Q00hg8EmyOes*TC5-_Y%Wj@FiP6Yi@vq!&Z z$w%xt+`E>?R>vaCMwTD&snd^(*^of3L#*~v{xvVb*$wOsRL@jb%J8o1O`AW@f|aT= zD>woj*YOe+Jndb&V>|}RR?uxOk51ypKW|NPbjSm9}_Av8`=Fi020aZ8Lq&2E!(|gK%M(O8 zPN;nkO!l9s?D7WvjUL<_+F5bhgB$m4&fmy)>~Ac{s8{kl)n&faL{FrCW2cy}`qJbv zx98qW@W}DxSFLvyQqgWbG1i^~HjE8_`<_MR89>Ytb|M^m_9(g3LyPTa8}U)0fiB3L zQ(y1@TUQui&pz?ZpnY_g*)gJhVdX%BZ@|^H!ovnL9h}L4O2KX+ageu~oNMotzUcV+%^Gk5SBgJ4)Nx0?_|^$a5#t9nVtf;q;p6U z{sSbpbMiGs6K@a}I=kjk0qC2hUy$#AyYB7~Cps9LUg*X(AdP_S*7|5Owq|z^;vRBl z+acdI&Us>u6$9^HzV>Yvf#$3puKYjxpT#dH$7`C#0Dr*{IhSy7g zHtQIKL5gJqT;~50Yea?Def$hK3=VT~j&Jd$?=q;L``bsFB9VqjdJzQazn;2vA@9NT zp{C;&J2fb-5bTsWfKJ9H=5y8Nl zd8;<6w4nJFw9rDk9TCf)=`9k=mhc6epETzl%ab>s#csQK57vM>q)RQp?rb0tWNKTRfRe1;Z+7`OaLv9C=e;Cmak?@AwVZ+jcSusO8=X`x3ki3xu zlxcS=EutrXdule4?^v(y61Bm zIFT0T2?ki{>3L$j@TFjOhHA=)|3eob{qO@na32()1G1asWjjJPJn}1U$={xS*yauJ z-?o7|PN|3DKtIThbdHk$>FRDX|G>{Ch)x(j2d9kQj$1kz|L_0biAszxfR2T;Bw@!G zteHjw4@(dH_|^Fg+q(ro(kq!f7V8AN9f*jWh0o^B2#Z1}#l9CY(-xZFt$$KBCHzGY zAYAZIQXX4y-717v6@oY-%D9<%X)@XPrCH@OLW8a8^Z3oFYGH@1GhV`9a`$0h0y(x5 z&?DS6S_xnJz@Gw@CTzM-*|Suxw~eIn8NTKazevz#c;KJZ&c9dZ=KE_|0f236FL(0Y zi=PI(GRHXd;hhA?Lo7o6WF?bFf7t8gX6&>dO{5&m4D2yriy(ppPy-ekvnF&-KZiX< zxxcRz9Nm`TahYiYO!su*fQ9-g$%$n+F#Ac+aHMYUyytxgKajZed*TPN9mGXw{p$Rt zm)lau`yH@rKo@#cipLI%9uF*Fow`qMJepzf~NgpUtj2XIXx=Yjicl*v#^*9}h%9s+k&E0+yAN4wf-2R27|6lJh?V?KGS#7glH1uSSnkN(F(#=@}vz8&oZNcn%zTnP4(DWNmht*>R;A<1gB?|vUtVJLXZnm=qlA_hMaV+n zh<2%W`$wbSI^G38X#aqT5tvTDZAlpW@<1R*d5bhJArBmTD!2&o_45k3p2|lX#&B51 zA)|n^NyU13%CZvduym&oOu2L9J<|`4hWnm2f3c+j>MLA zdzPn1qngVS_SH5ejbzescJavkk2!c>gXsM4vmaOKXe7LXNlto!3HHj23JLM-oEuVh z+m-p(XGP7FtG^l7TKu$W&X2buSCY<2)G9a8O&?}{ySO^JexMZS;k`3#n$2^4;gHAJ z1&r1t4im$k$0W!Ulrju2E{DCAQ~YKs3<-$C5Q&`z$4qrt01QCB#GrMN`;vsys{CTh zGPpC*QJRCnk7R}>j50Bt&2%Q(wy%By#ig?I=VubJM6RPANy$=c>8wJHrO%fpol52r zcl#26^TZAj`1)Vr3mb*#+uU~^?0erQ1Zrc9iam{_Q3~x%bQPVMq_SO+X327qxh^fd zo~V4)r7_r;ywP8}glsb;x21o1rnKKdwG(H0_U&p|3odYby6B4rt$V3XbJ?=|_0=c9 z#aH)rwe=Jk$FU8KY4QmU~^%jKw~Fw308931|J#su@K(^k73_;6%g$x1rm8 znzV?a@v>4~xxl4QBc5Z&D|BwW#y7RfH5>}bXUL6KFTwk>iIew8jLM?T_n8&;py*et2?`eQb zgmGHS_`Ix~N+mzOWcouql4epk(y3zbyNTfEuS_%P3L}6=q#vbVVhyb4$QBH4W|58# zMsVP~pu-NsjBASa@jH5^G9wW?5uxB3dqUMR8i|4>Y4Yx*U{C@_;2~oYz2od(~=Z_c;FF)Kl#R4+b+PUbOLCU>$>-dM5VXze6@ts}l^p{pi14k0kn zh`2O#q>uJ0GvIKz11~2ECL@;*zg^dj(0v-{8GfssD)=oV!;iOT4IJU}HI_A!{U=?M zHB)5iA?g*!teeHFLdWtW)j72hkFVkRRDAU+e{vLpQVm*shYcHj-KAzXk{+V3zpXe+ ztaclC$djpaEtF%nC>>pJ2SszYHfJKzJujj3TLGP?5ATc zT&kPyln%}r+6vaxiB)tnR9A|T!U5U1Q1b%IpIoO;c9l3-0}-y8flnGgjX~RGt%up= zFcO?!T@@FLpebLb-(kAURFHv-`IxmeuEMGLtn18fW3Mv4r*+@`{5Y0z;LLv{l-x+9 zbk)rFs^Y?rq%z(>^^782AI!%z{rMj&-5!{iTj+^z`d=Y{(p9q)Emoh7hO!6?Ug2#| zhUD{B8PxB0rWs}~4=L2fQ^~A_6@vWliOrSYQd3k(WBsMy(}nEM6sf0oQsNiwGjG5Y z0ikcaRWX;OajF}hC)3gSuY?`r2m;fBdEf9}UkD!jWJK>RthF#p@AkY%%TnvjD6C14 z;F#a&Z@7FJ%1XqorT@xglj~hJD2eoa4;2@0ZGh38u4uBQAYgLeCNap1eDp`eU09<5 zcUQbP#p+C0%QM|_fAMvhkZ9wRD$Cg(o;W4Xa_L5mNx%D8xRi7(UCs~bUDOUgOS!Bb z4N7n5Qn3`(bgyE1e(&DoFrKU|TTQb_^wYFB-A8=*bCsEVIxAVmd!RH+l6q!LzVT|_ z@J-aROa^-InOp6Pa0)DI~jWdrj{ZAbM>lMlJb_u-pSCHT-=~8 zSS7ZZ$w7gl}U&8sYgzKM7wtq#_*3gWR|;Q zX^w+2*BdaW6Zo8VryX!uTvH_q58LpL9J${Q=NhdK5rhJhNxkeDV*b0+NrY}HfDNq<4cGXM@ zdghTw9$Iu&3@R0A4PIpWUGk2Y=CiL36cVs(2S?WZtO#Jdveh>UpYc~x{)}eQ(C^h< zO~9wcRO3B$nqK6zW|!7@C1HoR(_@nO{PTuuwT<=kJ$z(jP7Tsfc<4N%A!6XEo6b@XaDMHaRPU zOPUbugEh^fuJ;!N8Hu4%8<$D#6efyL0|CXfbi|!II9DGQp74}pR|5gp!eyZD)H?N= zbDzpeht}}DCS-dtdl1pCSj>2wMIOoWjD)s#id?Lo#?XqlR(AtQ9fBAOV%M?nmj}{Is5sQ zxgqDDNvm573kKeeqN&xI1pT2DdL$UeY~T1^sI(#OK!`R_$_p0-LTFRz)FGWSUlj*3 z$Dfow_41Mc&TO$~^-{NTdhCpW4~>gB}9;h!CS~%IMp?FM8X692Ln)_W0`xGP*E7cT73AR+Lep5 zBxToUB*Y48X9Goade3Q$1{Fv|#ytU1wRHtYS%?I zu7EEf%JqJ+Ls94QYJ}^0X2jD6 zIc!&WpMCli-R`_H9D2}KM5UrVt;}w5!5n6Byh)uISk!o>zZjQPce*TT>pqh2NA~|U zcIDwvcYhm?A|)y%OCnk*ONi_$>tmS*Aq!+Yw ziit44j({h79;D26j$AxNXX{_RHGD3@9h7n3j{=+%ic+m{b200jMA}&b*~hpivjjNl z>(c=bQ!TUlt7~!woH!i8t|nd*6b{uXj#G(im0DQiEGYU0M)O1 z;=9X#4HeA|$f3Ytb0ubJHNs(+LgS6KYC0j>fK>f*ZGtXKTzj_>;B0`o_# zVk4-oSLFk;(mE3s7M1LQQ>GWRV!x?TZFz9X)xs6*YB0^wt;l_t_bhxNYb!@4WEiX4 z)Y%bGtFwx58736|YtDS1QEURG4Ky7h(VDYkh%HPU7jc3D(!%H@4z59}J+EP-VE=Lh zLW8GUu|H3T-FzVbMQ~GZPQ6lLSJ|qi22PsSYV`g>&z#1&8=#E9^$K{z+j%XAHElb@ zit@Dl1B~jPXQ%>igGuHTs1gOG>r`*6+leD5O?O5nn4(2i6Rk*M>#3~I%SX01$KP4i zDdNyXVI)9+FJ~HpA1Xx+_e8#vDwIQC{WuQZ2x z?vwc5+`iSeC6_tM4>5cpA56qK*w&i`qSxMPa+e(Vc={xI;9RiWJGf(_kq_2K_PcFWEvvB2v61~5#Hv7S z-|S^UfUo#Iq2&;<(Hw&Wz z+mdUUgJ?Tqs#ZPvFm{<=F{06>2-Ps`R|Gt+wB+6TIb_r(`9BhbrAeE)?@eP5%BHIv z=;s2iEcP*J3b6{ah4rDqaZ6GFgqRlvYbE*un=u8T6d`>RVSb?2ucZ#lLz6 zIYy7!;Ja7z6#EI7Pa=ju4^1n@+>8msbrzXf|3`{1-tTlxTVgI?$+uo(@1X)dT; z2}~S#r=|JqCzr>@+x6R3CFZoN=14u)DZhZ(IK)a2#=&kn^P5$RP2@FNhrV2g!W`}7 z>GM&(bfIs8+4c#ZyLS}N?+pIaaL>2#wJ{8AoQ?(jJjjX(60LXORPNo;R&G|^m>Xt8 zYoz8>O|rZekDS)GM^jS~Ql9KORQpON#%}4=dZ=|ZVHVG_ja8-aU8_%%KcH?bE)|=p(10XNL(+^ z=G~R@iB>JdW}w9ZtRjL54c2k*dypL!LRHL@oWv4j43$6chjrJE8Pqyiv2kl~D_hU? zm{;nW);%{iS?EcX=GLiw_VE*2-%=&4UOc}7%d|64YZY@~zWGOAvcs~?svCabc0uQL*yV_&IoX+* zqReKWBFVNLvl&MGGEKs1AYsGNH~FcZmg>5m%U0$OWz_WSDXfD#TRt%NteW4w?7g&{ zH$LFHr50;1)EC3Kl$u18Au(NM*mMExV2_I~jVXG8-hqsN{*-Ui)@076BSB}g)5P+a zaV>W5-=s(GungYd;J^Y+&*X_e>+o{`dwyk*TGiY6vy7aOV%esXC=mK@v)ls~oT%~w zhUSDj=J4tGnlA44<{HtZ0>7pf2KCoLC#{B8QZH)C+p1`0N(9g6PPxc6Wd0CkRwF34 z&iY;k^}B{rC->PwXLZ=5_2C~CE=}wca_fmI4M_%|7k_(yCuk>iSAlmr|3_xo4lp^R zYlZF*q}&Un%aSZZ*S<@s4;i!lT`vZdwEwLIdBKj63%Cg*`A3`pPOf3#rT=ZaCL)Og z#%UuCISz@f0GH{{NShNl2#Z07 zhS3L`Q5D840C#~p6Uz8Tm5s2>Ltmi*&3~8251(KC>758UKZ9d@@QwW9JzieE9r)Sy z_wz8&4?z^u5aQjTHkff1kIMPWdjo*D?lRWvN1wuWCi^u;yzEbljKC2cDwuf1OHviU z`B!Hnj<73Y((dzs=D>y4emkDMB(q5=DDL;cg@YvfNncN)>8~Cme4=%t(5CtwCWqD= z0}|LABIkWHN;AOr(oe_=93eNOh69ekwyflZ%qMCjTuUyOj9R+ z{t*1m^uJ*E*ufnpA8kZU_fsXvx8We!tE_Une|{D9_d=rD`l+eUDQ5WQfjznI{%Wyb z4D+Nqp4Vu~200SKA0}CzkxK{-CO3iujrv`rDl?&*wE(MO{ux@s5}pLq@IALSWJB51 zY5b?#ie4cz;tV|T3=|OUIK_v;3vN&czOevrXGQ6L<*vHqRHXKvT0u8pF2*=JpcXRVpp93B-k18{^;=I}ReIn~WfAb=YI2<}QEXG$tXImO<{U19CDl^I(ZUY^g{{FS$r& z26jU3Z^!wCyT8N#OpD}t+dY1b;?~)*lQlGAQq0hZttiScB5;y8qI&_J$Cr0k%bK&2 zxn6PV^zUSKu~@Pb%gT`@SrO_A7Q{ZGgIs!5xFaI#rgo1hj2-lYDswf#%0WslTzyEF ztq;!ffA*BhtZUkNRZsxK$u>#8L*}*eyESTiC~>1(v2W`09uB1gtQizcxBe>8Y5da% zGoq5p3&4x5Bz+$rAbA1^5ItG}lYcyQwV!rZ7;^SvfXlU|F+^WRoZ|oEU{3Fg1azP! zTg5#?v*6mbvCb4Y*uo~k6(KXR{NGhp(a3(S0@RSpw@BD{G(a)`w2PIMnoal|<5T?L zcQQGe!Fu@?I9e@SWTpI(_`x=(h%2T~i2>!{ey|e|qhdY7XN0C(8#aeYwMQ;8ULzKJvfoCiYaN8QKLt)2Dx zkZH8db*_ZsJO>)n?4)hU&P_?rXf-Sdirksi0hgJsl|1d^b1?h78DL{SRf0ch0iycW2zdJ@ZOy^J$k~ za@FF1BHI?+Y#d7&Nyk?caXopjjY1~Axzgcg-lVB?`i(NnVm@$VbpqtQ%i|T)_HKH* zFtq;6ZYf0Y!1DTu{hP$H^AaRP_{IFRx1#UqB~8&f;@NrQb!=0d6GRfG6)ITeF~z_w z%xmyT4%|rcsv?u2GE6uZ1&;Bv(kfXLv$I?oHN%4AVaH8kq!54qRw1>{?mjA<)Uq3{ z^-KY-@xhv99N5HH*}_?-7y-<{uSN5;_uH$JoW_~WKFIEj|@OIC=u`sqejg-?|!2- z`Y}CZ`*W;7yERI<>Ua(y0m~z01L*^Ceh2#jva@+WW2`XA>JhAzxRM+IclscJEmFFd z9puGIo+fvhFi#>#;<&%GX!WeAB7u1Ll{a01t3rO|<4^iHi9<7&m5M`~^GJ*Fy=b#E zt1H0AZeTv7;?C#WnbDeIJNQ_((ZbWF^1;`hcR=zq`I?(vlg?rCnTxB^GA$@oZkqZo z&j6+zO(@@j@|4`5Z-~C<*!i7{y}w8T)6?;$XTje9TKwukDdJl*AltL3mbolO14(37 zutev#n-AaaJ9cLe0DA=gzy`af#NnP{@4l`ca->nWl*QDTul`K&nG6a1WD41EHD86YB8or-sX zjFR3k#`tTp1iaLG;#Q@L+eD4~HJ+|4xdp7b*uVbT!$-gTwNmz;Ux2?hHXmM4nB?z} zCLbGgI9hs})`6@F(O5}XH)V*FCX4CL%GlJMJDl?ulv@DKct>v)U!&B2BOn ziP#RnHDB|qWztx`$pO3^(n7DHR*E_wc$(LXKY`n2+5igDuBPjcf)!L)rlFd&YO!HX z@KwLf8we%Dm*ftLw-o#MQn>dg3bqk`Choz)U$)do@#U9#vShvuzLy3jZWFPH986EP zNk`HZAP^OKcuxNoTBz17Cu-z3+9?cT8*&5eJXkW&FrTAJq-gp25oRC>K9p-qbt zrA>@>fu|Vn3a_!{ZYW$q{AY1e_XM>?`IEqeY?t*JsY4s63VT)khfz79T;xs5JV?Gr z-E)r$;FF^+9tzsJqV~C%{AQG?!i5P-UIgajubOS(Vioxy~0qTPxBRLL2S(8zc{<9X7KZm_}vR$bd zMu5YB2m}UpcPJ>6)i(SWXd+wSJE!W>kN$`RzxA+KgQ<=wG=aS*8ZCxZZX zdNa9>#CadYdFBIgR@IRJFlqoGCJ*-#%0f__Xb|Upfk}JXBPnwO$PS~HPi)QD@*0yH}Lr%kSF_h literal 39528 zcmce7cUY58({3nI6ahs=KzdVP7jJkX^KOooT??ZvJP5l6za90oijVJF zsg=4&-tBWnQ&80W*`Mw$bExO7^HIO9sa|>bklLDcy5#VL_B~(5Jmq;xd`O6k*Ve^W zN1-kOnXcXyBCJrj%dxu8K4dN+LARDyE&Q`6AP*=q47zl+-1DE0--zYc$RGG5euMUB zW8iuj9HRkaj)KL<7e~AWR;$*S;Zt;Era6&AGz5}LND}rg?nJVO_!MT(snX@ ztU_nBN!m`TIl#fym5tn`lXEU54&~>-VU5t`FdMzKk7hTS@OH@di0|n#ZuUUnJR8aV zLa!W{bar=lmv@N>3JQ)rK1a`qF~xTId6?cJ*g1bGhWD)&ck9fVxJ~w&uAm8(3Ht2x zf^N=PWwSxXr4TH>-rND+XGrYMHYP3xIivPq8F;f&*^qYsm0P&awYu+33ADg0IY*{6 zsVGXFYEuf0xm0iwy!fHGR2hTFm=`Q9%6`vXRJKUE zl7g&?LP$#7nmwPASY3c)d5U!Pj85Wpfv;LG<2~u6z1Y&;{iE;i3{uC?etg3r57XDy;(vyWcR0qaY}c{ z;@Pikaa|{Why;;R;>Z%Ke7a;$Zum(ZG6SZu;IoFw8kgAG zpR*z%GQaM260E*G%MEAZu3RrSWmG=0_vydCQnFszq>sXlS&vPlx&r-elM|QP_6NX- zdfs(EHLQYVaC7C~J(ke|Z5B@P!zm9i4v!((66aYlUz1CQ zN{sk%10iSL*+ku?pDYHZ9vEVsteJX$JT>nG<7yav-*zxwfzwW3`PdOR8?e0=a=poK zdkpeNKU2PRw9KMx9f6s(Vh8rCq&Cx=cdYa`m=t$HC+plYDnAO`WZ<}~8+0(Ji9HxL zu;30p%Ef&`byYjZSz-I+=1gar{46-++)Z7wgX(HW&;pXZnF@Z{TwG~cavoXG+2%&s z;%;?T+XCz2uTF|5tNF+zd!gL<3=j0+c7;jtC)iqH%0Y1VS(j&*nOyx`aFd-f=Y$2l zgeI4N!5M%3jqRwbq9eHxT=0RxmiHGb1bKyvPr;<)*gJHF6;`zZ-6B#y0PEl{U9h)VKbOo6EYJzg zvc}oco28c^Liyeu5`;5bPkNUE3AP;Imb6z{st(`C*VTzRB;x0+z47Lk^t&_eJkLu> zTwq}WpMK5ay6;txb=MqWSFs`o9qCM8c=0Po6*eK9^|~cJQS$kafD#E?hNxQ(;yV^<|`Hqq52Jl=k>Dxclhhu`>+t2c(X1 zB}Xl(@WC11&eYtQImG1_LffMB)d!)eTGtfh@p1;vDZCoemoBT*2m$=f_v(9;I|7Ln z@#&T7SlxA$n|+;+K6tcs%TouN?215{$4+glRwWV)-c1KMxWJ_Km2jprlU>T)Nz#xI zBF-))b)u{AX@;D~SV~;&^I?sF3~&hAop7+0Z>_df9G|`oxKI7^J;KpKtj|=Xt-=hu zmu&3OIOeR-+BEKp8k6phZj>_dU|4k6Up`@36rFRT)TK6FX^UGAPs_gTk4{StimQX+ zut_0oaTuG^0n(|#lC*Oi=HBO|q&25#nf`kY*-MUEvl4(VO;Jy6#u>U$JnpHm)(>kl zM}L_@p7m)yIgM>L`Ftfh+w@?woI z$-UpyHi1P*_g4?6@9!r|2-IPEmarS(x41z68Oe3sWW_n%dXKr_><~p@@#=#&0b`7Z zsh@djjaT$GZB+M^7k2j)>tz(J5I0=Fq9NNT{DGq7*@%h3=zgI!3kbw>)ms~t4SwBC zaD2(P{&{?Tq|||Y6l{0$$KFN&yU}S=u;NtJf-+pI_G0M=$ILzD(v=PL>7I@fgoa); z{IhxQq}N5NFBK6>O^UIV>W$Po zS#?kFfcvidZv3u-J9K=4*-R3)nhavW$BT(b+(NRoyHNypaD!~pPCbUGSC60ejzxFr ziC;`~`v^F;dHp;hgjnXD{d%5$6mBM}B;I2|&7tIe$N}hMseb*J+jWBh>GjV#GnuD= z)+SeT)#tuRENUD_{Yl*0lTw4`L5Nt3lb|9-=>o2VW3T-)3fdd?eP+;={!C;NEkkDv z{;F&p`U3=uCOh&B;JzogSG|p1^9K!Uc>=e>xl(tu=d-~f!N-x8_z#?{ys(>c&(jvg zOXu{9oX|wX1!=dSR{`2t9z}`cw|WnyiRwQ$%7(wcr30=x784IG&c&os*)I)iVNw)nFpPnr|&PYT6tgYb%J^c>I)rw zZ_s4Y{WMZyq?YMB`5dXKo+T!JwoN)SsBaPQ(a870 zJ^8^P(91mxiGV@s`B5PcZg<8Ut&bz>mm3^{d4|^hf?8}wpSn+{Xx_NcXavse&s12S zv-8J|T8Mg7l7Q1Zzt|~w&*NvO+PR1I?lZ4kzrC0+t|!am%-JQIopu-Tb}o$mec+G9 zHHbc@PQrkixeDa_4yK4oK8>w$XDM$Sk#$1l{wWkm_)>{-%UHt6>2nSm&7P% zoZeKCaU*pATU&2tk_9>KvCu{@{3XYlkbiC|#ZmAzr&lNM*cW(6oPO+0YQSWC_=L!S zZInfYWsOGcl(rZ@xQ{28xDHt#yVywSINjh0n|{&y;dHeyj+>2jdAB>wCUCb6PCi$_ zR<|fLif9>cloW63+9m}K^mZqS`ecYZvK0y6YAmt_?t1rO+?}hLd*`RASjsWPO^*{R zb+$z#m=`fW&Q*3v`MUpfcJpBb_*8X(b@`TK1Q+btl1b-c{99?UebKYO>;5f@@NK|b z2G>7x+rp0`@>ja-1q`NUJUci~Wc{0S6?1jD%3cKjTI&`?d`pH;O8B!+zBzZ{K+&JF z12+yqtXQ0Cst!4c7krg_@}s}m{ddT=It{(P#$I(-R`AJl;t5mm4%Lhh35&@*eKJu#pxhZl zGB%h<8&60B8w)e-(Ylxpx_kn$F#11ue_9vyj?!QLy`rF6Foy=rBmvX;;KBQY-&%Xg zOH14z-zAwETuPkfAjAr|O*d5S>9`9R2q)q@JO92aY8L_u&o_}3%7&+z_BMW3pejJdm*Jatfu2U8->mQ5Nl z>}&pOS&x&WHtI*LtD~ulOmP{Ria6bt@Z*ZKG6WhKI92C1R%Z64PBd&|{-eYgO1+wK z!lQJwz*#K`x(9Z0a)K8W6wE79k|W<&o-Ba(gF=7vd<@6LSCF31R9@g^tmI zGz-0Axj-PZ_UI}N&^-~cS`FhI+J6Pjj&oGQf_jRSQKL`S2V4(;0TNQVQpIgAi-n4O zWEHl8*^`I7LjCU{|Bo5|KRoI(1l9&c$R;Zs{}uuyiKC|wny(Caohx6yJu_zdpw`{t zfH~Q8>YZO~$;VM(pBmo>dhrB|c8dZReQ83%W!GdK2e{y*c;Sa@xss}dfkvbL`mz#Q zPY{@!x4H7EqM?Ir>3`?iSS`wFL57m;k3-&WBoImJpmQ#}^AD}lix>)Ng~A0MGmA$> zpjd@Rfzb_h=_Dy+QvBSGvDTw*i;$5sC3kZ$*J+N`*$;Ww4PMN<&~*6JzB$UK z2JKg7l(;xpJk-nZtqJaPZw;4)e><4&cC!f~PC$t#u@%6(ca_gx7tXE> z>z7r)aL}Cg7!Ej^c*x;;%kd+KC1FngOR*iywe}=4gGWvv87M%3K>(t1Rt_w1%H8Phs_C5-b!@B&tECQ46rC(6 zv?1f(Zauc5-p;tmHwdA8x&wh8ta1gMqou!>0-04s9SkwlI&{>-;&%VG8a!fSFVbbJ zSmE|})KMb-d5bX_E}kn@CY{QlE?VNVNr7+E<#DkmK2=UVFjNc>uOq!UbW-}wEo^p& zgWe=uzq{cA%f^s`BPP*ASKLNT;z%nK=?CoU1mE{}i>OhK(Mh)ak#Y#bhw?l7E{e)VDqO_C6u6S5e{Y828fsDH(w;6e$&wbL!rh-Yt>hb<| zvxkRRlX|P^qvb#4|E>eUgXA!2Z6^#HPER6%?=ZHbUx6iqZMk#6&FoX48(;_NE(6 zd_=Qe&->Q;tZR(cZFa#poFR_04zV*(F|(yqhsjdm?9+#0HVxJn*~dnXIo)nrJ+Q{?lWQ`JIhq1Z`^4_m{V3h42Q#P*9= z?BQ5Pa*0cFV2-|9$hTRNkBsKpairXXiy$7jMs#XH`knhWvM*kLWxxC4OIH2UHpQ7J z3rJ@E)L=fh@kRDK=C#n#ipTEFS@o};&&xKQ5fSDfARqV$$={?jZxG3GVm1Ky6XA5FD}1_%aAi%vEXFxjtTyA^YNg|N$pHPx6k@m$XHP2 zln|=Je0Ig_mnErq1!bH3xY)Mr8m+a%RY+(QGyh`s*H>E{lP0omj}7!7N0r!sR+AI1 zR8u@I=Owc<2xRmNZl>vadoX}cned|4y$$K}0nCiwnkm@h`drg5uQebls$=uysf#$L zOgJM7!54SO>;kL4h1~Qni#e*q!0f&vy=?4Z!@}gc`RC8yMmJZo5#trMAFoxEOPR4H zFYeu;ehXr`%)+n%HATqJv z$}d9q7ROEZmpia81@IHMtq=WN=?f1xJkQiT;6$~Cdkehr!a;1JJ(m}}8dIfQyJ(R~ zDlZ4F7MDSMT`QoYukf)_6MyyCSIxIV;(L%At^McB5>EhUEJS7LS?gCGNq zg$9<0oX5Xg8Nz5l%y<3sQjE-#uY1g%-kUk!*sR-$y5+T&67={hk9VllCotMUJ<_sL zymKyi<`_xXR3&-XB zjy2&%#7XQ5XYA-cQ@Mq8{qOO%J(FZsh|T>gA}tIc9$glpXhZ9I_aOrUJ2*HsYrJ$= z;+3ep-&Vy`z&2-Iim1d~zH@JbCV61#{sMz+wS(%OlmRo`x|}hm!kNtg!eU!xCW?{s zGG4|Y{mvr~wQ%i34XpoRkgEHR&MAcqCUa=(x0>ko^Ggx3l?Zcv&ODE5)|5|~`9;br zr_>T_N|$KAcxFr<&fw+m1S$)AvZPLgTWUNoo(eM=X|D(%ZRzx}6H3+bcIEQ*_a7Jt zV+ORcvY{D!abczgcbX1Q`>~&T^lOU@pI6~HX__;{Z9nh0ObDK6roc z@V75EKNb!xYL8(uF)6MrZ$7qr)Av2h4af3h13FS-c?o*0^sd`wrrXl%iCJt4$1%yK z2BHe2Q`b?al$qR)%sO6tzv96!jwFf@xlNFAw5{FbTeJ!yY(}I#cF^hbQGTdR%d-IbHvt9tb{>gjp4@Y8thC! zYom0L7bg z1Tebq#4S|AC8+?KjsH8SlMUts#JhDsr#-$6Nwr9zlO(_bFWNp<4W9+*;qy!sG%Ako zMn(U5|0VGLL*vJQte~IPwR`9P6*d1%UmE&%RJ=4sSAF%>5uN-Eico8Z_Q(978%fDl z{I`BlQSn^Ujny^$@wNvrM5HiDNfH$2qI!-Q<^Vi0Q8I-7=I;c=RRknC|A_esSuK@U zfFxH6Uc6=C1(bTm_Z@We@j*M#={30%7a&Dwi)tu^;T<}_BtN+UL5=A7e$u@GV)f&{ z;O~G(sW_ir4sjghpyGiX2?q1X1b^TLC1@RW-VN2FI}g_cIR1vjAST_EJG*o@%|BLv z5rb4L7i`RFcv5tMR`CzlRFc|0vWSso)nG5EV#!HHHGj zg9Jp1qeMx8Ko40cXh1`Be4W2I02N*W?SClAqCVsK4hp4JJqHZ&2QVrqFh>q(Jtemq zkhO{npuP_3G_L!X-KT-YQBff)_AVBfgkqr@_}HQrN)gKZ&$#!1ahdtZ#-qCi4E8|7 z1JL>hFYwYQW*|@0$ptieMU!p46ALo=j-EufvQsh30u!k;)OnD)+0}AO&Fyl_%XO1J ze|}tJ!)_ur=zB}=TW2YBrhT<8&}jZt*Z_jxc_6#BHa;RZKvXsOxah>3*XA(756`>j zMh@{Ibs-g+2TmLhk3NBRrG<7wI^a5oB-Z#^_nFJM={)r%q}UZEa9}3vWX%)XxPq=) zzOq#%Pk-{-+vn_>g#Nd}=iO#7`|~4iu@Xj8Xu@pzgD+{AZweP9!GK;ok7L2X87Es! zZev%@UsRHmN-C9mKbx$00q`^HQy=Bv>Wln=%PQZ16+=H=ck4O_vwymQGb5SVYAEI% zZhO|qA8ja%uHG6T!-VIk^smmXDcI?uV$+$ddhFkg_2ROB0_k)f3bpVgZV$s5jWhL8 z<;`oOIpdqq)rLkwxk2*%=SSH%>FUjkkPQOsxf*2|cY)ZcV)Zz}$xOd3Hl6lbiF45Q z*h;bE_t!rA-%=_p;w6?5`}$OQ!c(A@mGdeuet#{>qf@l_2QQ36Lh7tAXAbM5sG!|8 z8ri20S$!s-*N*P|6)rU^v#Yn${@u8*fcs80UV;wFDCJT*w4kE-T=IO_z0R(p+J7Gz z73yoZbl1sd%uV?HR|ZiFaU++k3B!wArw&R%U4m}qf``6?>SDO{ZFG~V0cNke1~{!v z4o;jPj@d#6T!m=jUC+kUTi?|88iK4E?rSawafdV;$K06E%g(n$v6px*gqyn$U#LMN z(35%N!W}#-f8H`BTkDwadMigTZZ?kxD%8)>-#0(z>a6O<^Z92SYcNIA8vo9|w;Bv! z2^5t^cA^#B#r>p^3xZkVK=lF_Qbcso#r4u8lZX;E_R-;CA*~Ta6G?fU!&jeHHuyIi z%!aa^K1JUM;$DX2#RN3vkF>~d3V2}_6&n$NQ0h)9~4k3K@zq^l_ z#zz`5>E1ok2|jaT(KtE{eb%kc)SE1}KANK{46m@31E@bB^n@4b+m^D+%0D#7pv3=V zX@b|!!n9*QB7N4&0IpO=*46i0Ru%wiPO=glfxHij{hs6{#{wSidtcq!>PTh9G0T0i zW=zUQ;c$}`kdj;7M70ww?r#LixH$XF)wTq4c5)gOcVOT%?YW6te_)p~k1klGMu8=o zcN&nmu?yuE__4Ba9$DCd8sW-jQOEaQxnaA3_%Df@Q<$-Z_WLUgCavKy^+nTC>+6FH zl(qIh9AGvA+50d0S7gUeyYRb@IvWp$SVrpHAlT=Fm9~Dk<%|osdw?T&k0&Cql)>L` zk@C?Ye`B#8^_hs30m~DQnW(Ak$9Ysgy41 z<~!~5U71TeV(J}f^uS)*b=WSIqVf{WU~X}txq=YRYOrO6yG>=4ld+C--(-neR4{{%I;q~(i!+GLQO3!6PX4jHW2J>8GR9&XeiepRgD z8n=)gBQ5WnaA@7M>IVY>yKNZkn4NvGXdNmm-{I5q;;g_u7SxRDTleAp<$>%XYp^*Y z*q|@9$`|3S?7bW`RyL72U^>mUZDKG}(W(Q$4|DN)*YVkq@BMO=(X{&(pV#UR=EdO} zIO0eeSV^B1+ho@aFpQ+fhw<*)jeD;~8&o-cUbZ5!vr(Q>gd}*5gD%^s4(H&vut|iK ztj}iX!<4y0PBXbVFbVcFV<8+@ZZj987J&7iI1htx$d!~$S#w+GWm)cpG1__skM-Rs zT*;RNg5+~v?pl#ucRN8h#s3=!L4V~`L4WW3iP{aoHCx!FRaiG&;upeVzV~>wh6}>1 zk|jd)>mQ8q7W}fUUSzYKesnMOzISGpgk6x&C-*|uhsg-DGIyu#pF@Qo#y3vz7%Q`~ z)}QU8nmE<}1;V&B)Fqkc zwjZ6gUR@5pLh#!Bq%^|bqyLK_^5c;>I0L<7JBkl4K2VbE0hgRGah$jC%E`57)V!-2 zK0tj_JmYQ<+?ufTSWwW-hj6^I?&InL(LYO&=#_tW<*QdIkdz!Ay(o7uivKP!S5xIa z6Y^@R$&V%IIDx?>(uVa)(g++IHa( zO#Ia+P$cYNFbKi!$sg(cImUGAYxe%DQ=6frzTM^dz}J^5P#+pvGw%G_Eau1znZT{K zvacONxXlG8lAJyW%iS2d=~l7Wn=u1-SA%DGMj?RAN`>Y5!sy8&BedgbbOy2~m0pw}yC zA3a^eYw%5a(d#BJUa!;1?(oTvH@)Zl7TWs`3LP&;jg|XD#B7c%Msw8+d^&l>$^}1L zRXZlT3u@VVwkg4_>mhJ}x>@tL^HJ`_!m6db(CF2?C3; zn8oP1z%Nm%v~(4cBWJNL{Hlv*IL{@*;gte(D6keF8I+>Sud`xSO>u><*30|BBbz(B z18W_^JtyHYWk?~}#6?Z%sdFeq#6hD@_Q%E^uyG7_+#J09x;!hy9yN&+yUW_+-C52j zlaXt9>DimVc!d^!+VdG#&Ylhx7@bs$vI*(2m#+9=4=&71#&Cmm!MI0UDWdqK$WNRh zmDYpB1H31fi-4fu3#jVByWo>XhF*^zHTe zkJjtvTkBd&F>;p;ekQb>v~vs$FO(Jp^j3{~U-?GQqOt1MeLrPiVq@y!eBflG(rka` zz|iK=ZYvJm<+admhI^w<5o>g4KPsFt8Y>>464d07A}g#ao^rA?_>)$YH-Gc{ty|oB zr|oKQNB`Tl2^UB*y<7+4W- zzoTO(v>KE&_de}lV|E~hbv$d%a>jF_6u9{xf4L_Izq9WYF(ER-_bsl~`tfNn>%DOm z_oaEMZsXV8{I8d|E-N&~F6BndmYlAmue3ghS}N2jb6?aC{#N-e-P8ar{~u7*GGC)F ziZ3u8I$VUsYri`XDB#17cLrV8ys@hmG>o44U6le~$tr{GSxVRR0|{;wK*o0!_q~;ZaW@3FJ&G1yKAjcDC#BAc4>s&RIr-8Pv(GoRzlMJ;__fnx?{`W~2)LOGVWFsT z-(D>CMV?(VVH~d3(`tqz!b@Sbu2_Qda!dt`tMSygySgg{8)sDJ?4xwBJ*y#>dGD(< z^J`uMnnZl}LX4AbrWJQCSA)ntNp*X@kE*GR7Zp;jmiQ%GjY(c5 zBE?#K(_Jh~GBu06n72}wi2_BvbxiEN-=Xq1{*RRZp0w@jZb5VDSKG4?y)O;lsts!s zg>Ad|eG2a-`?kCj46rp9z~F|O@XVZai|Eg98RbnQQgkFxZB~3$eS~dD?sf_gv3|0% z_%GmmyKW|KlWzZ+QxmV+)==!_oT8?8ow7& zZL>#c^a>8!Sps17;8Oic(PJPb3_zYIkw&4~!i;~z-%PRb(nJbT>U#2jF92ZLUZD!5 zu80w%?3LNlOOBz>c)wraVtFJT|LZ}hc5Hw>h+02ta*)gB#&S)OFu93ZJ0gC#pVIc;spw4ZX>!oSLW%d*?2H? zlA$uogP<3uwl;Cv$kD#JiS@BPYvqiCb#yOrAnQHNnKf+_+sz?Agx{iubz7=$#R?g z5Jg7L#^P_HZyxX@>P7Mbkh?_%fY85m4xe$}RuMThvN9q88z2kM&dEV~VKgyDMj0P1%v;r9cw*a&SD0VeTI~Gup zjtrx#<338Vbj;6g`;#S=Cc|40@O=!xOsM=KBVb9G(;*r9@@Sn=F@PQE9ov~Kuc5xS z>e@sFK436>&s|L|y^wMg7RdYk`vZjTo9um+NYz|moV52G_Q(QD1;g@d)}-x+p{nPB zB~bHVh{~jwv_%S7fy~GVb&aw&*!%*h^U|%KeatlUbd9UKyFJ^Un#ZXiP$LR(9xlKuIUoL1eD3&EHUok)t%d9uopx9a|FZk1iK5)ZGM#C0t{rk4U4 z+K|=oENWnq?tOT_^XSs(0tJYg4Ed#u8B~Qb90Y#;i>XmzxT+Jk`xdaw`xx@K&Tn)L zp9n#LF3bQm6Wa0+n1<4*1bANHBfmnPCXAlH^C2%C^HUJ$4e-u6krHNyhrFFf7hR0# zpMXL)F4==}LXX9chCe*|*U-Cw(}eOdJI8fd+}sJ!J)U*}sT=|V8a`~HljfUKZD#@& z^(T2{q6~A-v0*v@!YK5JJk=%9pF9g@=)Iqxf?}f(F1zv;8=d{unUnN-DluyKRzzv6;F43 z;2v|W<+r{ATU%u2@aS^wX47`XEl^u|+?#0bch`At{0~d;$KZcsq^^tV`2mDRq|)%P zF*K_{sUU z4|R9+?pqyDIQNB1jG6)2M&QmK8ZIU_Je=`fTEnh7#~gfEtDfJrF-P(nRvemv7dl@*%z#E(>#TSs&j}{WD=v1tfU z(xZ;8?%CRTb-ShSZvJ<@&|6rY&M}W;4WEWnUfe+scZrAJ#!%}rw`HF3uaiO|0pBm<8Hf^EQv zOLj>9R_1Ox`~Bx=0GMW9%z%^Avvw=vQ+{ym0d(cr&d$fNvRVI`V0pU%%OZ#4nYR)p z3dI;WUh`0=KbJZktZ(ZmM!^Fuh+{&grr2N7kbIQf7rgS66l$Ze7f95cRJ`=~A0#%J^vbg4T97YqOyXWWR1U)hug zrqilrM%qq0^avi7GliL0FU^v*; zJ2{IBf8dJ!h`>y(6Ung+q^8N=itmx~zf-ucJldmfVFvflPUaPmwXXkWFL2fWlbjVw z^qGi|Ani7F2i*ez5z?n7D9z<}N20%yOWLQU|IXWP_G9)bm6Uw8GLYw(ZG}3i*_^#D zIaW`s*qjsNNjW)SgHIjrlk%+9VCFw3Tp+nqj^4}Dh^bPOxk1z=az-|Pm7I~y=$%dc zTZ~`&AnXq0hTKj6lkw13tSZaG|8}Ui(z%6OO!QlOK>BP#Q^IO@WH;kGHFt1YsI#(a z-eB2YMrJdrD*ws-I`(O6Wl7|E{8y?t)?v2U4Xzn`VPvwFd+lYy29W7B+t?O{{%x57 zVE@!X^YMQ1nt8+hYlx|6WFDg=>2Y!c^&{~R=yc8}a?01oqpcy8*p2aV_46U=Lov_~ zTZ&Tn+Mk4ui|dxhXS=NfzxFrG^*G=k_km=vxr`kQa*JGq zFle5)Qng*;0mJzZHjiHI{OB9~C~;<%l(Idz50vkn;5gQLT8y6qT!y=#;%&>lXz1LT zDW|S;zfJK9t=mL($f~(GTcVrXi`OsPj!>{!QG}^2Ma@%nJ}uio&r&#$z*JVAK(Io} z;r`RN7tBa!7G==UL$Qj)70d~g?{PF2F;4lDRV^0&q_utBzmyUv{xdNhF4A1}VYvRX zBCFn^!*}MqT*tgmhLObx#qzKo1Zy4@l<%fJFaI5v9a3Sf;Juc*`*%_dskED_<|8`E zX)Wun_4R1biK&E9jmwzmkzOj158KZlIB;#k1MG05xfo+zyNEK6sASiY*;;3zt!*&D z1!cF!>A!~l=+nS#SD*c9+&g*g@&B1jh6C(z?djW>XRWk6fltn*eWg^Z;|=F|`@nNK zEoDt7wwHJbfjMj5n)o-{@H42a>z_>W!~f1C*XK2e8*vAIorcN2CNmyW3!hvO)<(ud z=JnYjXuJBB+vCm0o=Cm6K508ah`h7F0Hl5kjNF`Q`hXr#53lKeWFu>0UblT!PR#0C zb-Sq(bAnDA!+)E{*&uy>W_Ea9V1}?6Ztz;4HhjVQjuAhB_)rh?*9Nz($ud)`8b{C& z_!9Wl8AzWUM?infu$s>bNW7U2u3NcniK(m@3AfB2bxovQ)0s67sWY3iIKVq`Tj_uz z^K;%t(}_wtHX*3luDr2B(UcMo%PN=qbOKWf@#;kb_gzx6^b7xkz7UC8aTs;Ut}xHn z#FsFrs!r8(9LF{ua(#N$!{(l>-F9I3-f6rtb1XF=IJRx4rY}$w3-UgA@}bzdOYf+? zyZWb|eZ(C&4Ei?Ffzd!<+;qD>qjr&M=23j`|D?0ab61>nvVw)=ej#QKl8`x?*0>EZ z;^>(y+!`T7-AH^fke2wVgS#@>;4CIMDr0&h#C^?VOcXSK+g(+NfdE0`_IHOtfza7}DC)soQKAzWhIE=XXIjLf#f~ak!mwuyiDu#na ziKVo`QiJl#Z&VZ3xO}v1uD<Be%Ij_L4|?=+mra7J*{O8oNHOOpQ8By_^X}U4CPHA!I;MLR;`Y} zc5nu29I4-?3wl1(##%pCG2IlepRF9RsuRB$-|mIb?f3*9-(QkKrycq4B~gn=Js-LW z)UyaFbvlHp7wcnwRm25&0(FqnB>}p#f!;<-P9{aPzz+4&eF{)tuq?%v^1XgMzSaz+nh zh(Ym**cD!U<~OusLhxlv5$9u_j(#JPZ0}%rvSt5sg|7Y0Vdv=&w-;dg*f#l}R7Xy}if&Mw6(}{^2llwqj@5c{f!L&45qUC(x&bZH;$~2t64e#ZhswI1I ze^gAyNAd@k6zN%$FYKolosiA822V8;jd)ezGT?W>M1~ z3_C7bVPy5@ z+RMw1Oj>n>>x*bm<4}`dUc?VYtNWQ@R32kJI@jZSDj4Ng4LR+^Q3uytKLptZA3rv$ zajrm#_ju`uvt%gxr)}xQ`E>q51%wb6;)H-C{qwV}CWxt+weK&2S(C3)5T^fk)qc6) zw3y86oyS2;aR)cM>VS)+6c4ucx`Cg=Jn%pOw5t5WZ|*bNGd|;#5Y?8xDryvP0GvSP znbK{7%}6WtWkR%LW*J#}bGRr<4FH2{UwEfs`*Z1I)g9C!^`Ci0iC_ zW6U&3z81O)L8d^qG~)rPqVv&6fa7Y6>fP0_(x+Q1odU$(NCY9ZHa86fBkjC8Z+Si@Kdg^ZgT<2RZl-3cG+@WfI{5(%FdBFp4h# zLjk&ZT@~L=J<6ZB8GNpm8)SGJ0BfO-KX}D;(Eu2J8bD^Cr(M)(IW4-(szOsldJ$a; z6<#2<40IllYZYqi}G&e7L^ypE!rFcxKa}4OEsg-*SC=&UVDl{YR z@`&=gtG&$8QV!*pK%XAzRK#DQlT=9s{x%l#g!xUr^K@RhOUV#tc@IFNp_Jw~Rl}L7 za(ExOFr+&Z;Fb1gM^G6>3FD*2xG$8Wh>*N6*0JIvCD<(jWybyB7oW}qV2GOZh zzKQ-y%k+jJ^eF)CK|J=yt4RRuaPt~q4$y;yb-T2jI?Xr=W+yaNF9c3(W2gGFIKG6b zYX@CvvN`OgZWL?L-6_cp?Hw95q`uDc5O}S0vD!uGCUx2$Lp6%D_w4riHbfH+QoDC_`fgys^gq8 zMOop>OP_t%LthmE2779(^+f1uap;0G#^n8VXbH+l?Z+ojB0meOzNta;;X4?P31Q`( z$Da}8Dp!>0K6g9Fq+Wko+(LJedk#A^dX>~t&UO_jN^*+vcQ}icS#_U`sm`F_g9=Zc*EJ@ z=VhJex&(y0pMt`HANQFv?uG2E(WsrLLr{UJRnJ17a!)6KfBlZQw{WRP z<>hKvm;Fj*CUh^a3%e~!szgtp70MrcR~xky&MFM{-t#zE7rM`JNR8TEkHJ>@`<14E z11$}*1E;&Nli83m_Hozp;`jcfc~{o5xq(yE$qF_EE)L$v+q#Emtlj$PgPlwphsWLx zESTgPG$+nAl8V!B{CX&{t$_@({sxUtM?Xcv2*dTGTS`r%VLnWzF5m8}U0l2SaC8P! zY1iINdOs*p;_7UZKWj^@Ka9YcID4N-dOz488bC)f3QxY?AS17RD3 zkzKx>kr!&m2~yoX(PSZTKbwCDds>oPlFI-s3G$@31Z{osD~3HiW&wW0dEDWrUN3qB z^9K@4D8$MYW=|gAgp}NF`2UK>BHoga1FWZ?7YvLff_L(`So00IE3#n3Tez;3?8n}p zv9r}pNJ5IAwMp9j&8ca>W%znlqkgRL`hLDLHXJgY)6qu=Iep8$PP%y=cZeu1f+*X{ zdW_f@2$L4P0|2SHDTJ?xldVCAo)ff{qla#{+4| zOex3kGcP@~eRWpwVV{eSEBa-N@>xn@$RI@dn09|iR-)7?~j%0$X9u#tBoa1ya;)#to^-PI~T^;TZcC&aXJD9>fFJ1t33VKT1Lv4xN6Jl(wa}*9J_-XdrrU(41XgU6Cc>~x83+a^ zGA!?o2H-z$8*O?(5zq|_HPl(hIIekdX__m5_hYx9qg?@Vx_8HGA}#*oQ(m({N>FJ5 zAz$lmNaQX&bt^14e3F0~G28-v7YKrV!{*xEAObbd&(hInI~Z$HS-38TvA;tS_uTf` zEPGEZDPW6q0B+7tzfmZD)OGFzEHuEV_i6JoV)h6fwSO_qyik{lL5M%TtS%cGsNz6m+V0-6u+q7So8rQl z5iI5)q||s`h^~KmmJP|WOhb&=rtmkCWrWB&lRYc}fd_H;Pihghl+z>#g0bv3raa1V zSoEUy^}SWFrTJuOif+YeM9L%6PfFqmq&yeyPl|h;-CSRQpVzPb`6HzGPRs-my)z}_ zu!!^%vh*_0L*~$fO0VoYIlN(yVi&m$#HzeILsSdNsL;FWD5V?kW=s-A?xndQFPcpp zc1H%{dj%46=yUEBu7_!}+WpUc{6>;8+SbcM1@O)mI@z?AD5Zje^&on_-pG=3>~HO< zt|%9S5=&R!v!4f|)?0?lurA$ZyY&x$-f`L|6k;`SFZyCg+hjOkzD`<((F9k*8Rd9v6~Q(pzE-ke)4rI`OQOSK@_3`BzcK=dAE@0dA;6~Bw>Rz-Cgct??)5BWXQ<(ldm=2dT1YYfK8uy79*BM~ zNrnv8v*Hh5#y|3klL~9=t;aa8TrnMT6XnGGZRK+AWv=IOV15?5%GV5P>=1NRq0954 zw-h;~yw{j&Fs}A=+f)0hoxm3X%$%NmyghF-^@V*dftOuu%o6_!J+DyE(5Zf=ijF@` z2}C?;FQsjNca1xn0|0j+D`Csk;IsCq` zXZnQKHZJ~U!YOj;|Zx_4EGs>%niM>2JHgiKb^$ z#lZy-!mG>2IqN!^?Rc*#j`NNu^`m>`Xc|K&-<(Yi?8>N zr}F>*$4??0#SzCSS(QD)OF}Y2%HI3PZgA{%%;ZFe%rdfN3!#i-l@US`LS|<6-lO01 zI`w+JU+?$r^Zor-&hwn>x}Mi{J)e(pA6D|^B!0tkXC9quWfr9!4}+bYx}8qU(C83Z zLQ1!R$wpXS!zRd`5q5WiixQuEFZqD$J>!W}tub;GSJ-tmmpEE4k;eLvFtRPcMVc67 z>9>f_rM-b}sWS-YukmKWE)G9oh?@H<nM~QK-WNP z6d-iC`7x7%S1n0SalPCbsaQ&F!=|1FZu2>KE8lb2$+SvwWanI64Ia?l4WGgrX2>QY*gU%71fw9tnNc45p86kGmG8X~o3<2e#=^N)Pi>&*2_V$1 zPv6Ley%C(f+@$p3ocyB;Sk{XZ8Uf%Wh)$0KH4~XjnY^fEjP_v=!;j0Y#Le~}&bw2y zWcXrK~S!$^s%RKd7e33V~yR#}LNeo~LmJ zV9m>3_aiO_a%I%gRhT1}odioKY3~uIpF)Fc0}mlADz@mF+x%^E$tE&xxyL+7N3?Iv zX_v5C1}*$w$mGsMgG`7pY?iYr#I<CpgE`8B#v-Vn zHO55V#+cUD`1x6@(wn)mH~t*DBqkJb(Q42qm_au3DW%*QXDsX_NN!*uSzyMgT_ef! zS5KG1au~w>)2MyIyFrQwUV2!Q-c>fm-7}o%ES}Tk!q`Ft(X(-pGvPj*he_&xj{R+d z!9t$x6r=2q>IymXzfPqVd(23VZw>!_T>34wLe$-7`JmK)6!jT^ z$Qq>Tv$alp z2=vU3zZE2AfR+`R4|nvhHC=8Bv7_it3w`$nK50VtqM7G)S5ds@Q3_ssHt(CE-3gn6 z?P8q~{F&F>tWw~G42&)EB;#!&E5=%6XK)QQh=e&J^n{o`)-@O!q+YDKeLPC96)^X$ z@Fkj_pQqn`eY26uEGNYelZ08Wa@c&UggW&G++QwFCvu@cR&6Y6jC9b;!#Do3q1^8? z;k?8QnV|TgK`V@nPa_H=lWYp%0&lJ24`uLl1gh?K?9@h{R1t1~*ErMM-7YhNg%r9f zya}T3p`ju6j^Z5JG76~XK3m2=-^e3>#N@zHT1@-7`|b_1oD+i->(zA+`W7BEg$RHi z*HPb~Ez4_&C0ch>J}=B)V&3=?$G&RPS7vJ}^)H6(rMs=GYc0Hetx+Sr_jCI6YB8c3 zlONBF*!q@s0Ob6fwbhWBPss;N7S1r0Q7pP;sfYciuUmVkJ2r76`F?^<3xD)%;@k_~ zxLlDgyWA&~OG)l^12tZz3Ri9vuV1sezPBN?;^A^U%v79c{{#yFIV2ZNoe(zaVXH>* zmHcF7+@1{r5{+ccw4c_HA9vC<3Jqg+Qx29MMG<*_Nu%l8UNSRwIjbg-7XeV8wwt>E zg}x;WnuEPPij7ZH`+DYvt`fB_npES zrh)Pa&*K6!v6o`fdQ!XN=kwK)Pj0>)+L)a&7)JL{0l*UN3)kl2p%jMt4-F50@UM7i zuUcz=sUCS^R_b6wc4y!qY;~ZXD(}>Q_09$V^|oqPC$CT3CwnObeTvc5OAeUhuURgx z1U{;1e1*Mt=}Rf^vcDOxn18WSbzkq$ymS1qjGNfHD&=r|cpr}hpzjJw!KI;8qpCX7 zYImdJY#YJem{9lXw4FZwiXk_bx)Hv~KFwS2@(lXbE@&-A&#I^Tu1=`e`~^f%8jfF)@6i~G%mHU(`5c~(8oF_GXtf>uzyuj78%zZso1otG z;QUm&^oS9Vuz$Tc?7sD@5X{l%m(^lr9Ve*{wJ&&>cuTEF|MF5knh{%q+Jkzuc@HGe%Acpi#C)IC%r8YJO74YG)|Lc|*Z>pqh zh4q@>m$ce&DZ9}lGQn%S-{^W&A=l;6)de--7i+=uf z8KqXCv4<87C+MgSr$~d2n|q=k;7Y-651S}*V0EshH$3cKDtC<8+UNi2qRV7c7LnKL zmv{o(>En^~M2@WUoUog}qZB}e7sr?1_pzrMTk0R5t1wAEM z-o^F{-gc6JAu794V2rLbaCRHB?x|2ORUf|gYzE8C|}qu&Zmv!TbQ1}$|!LGAw;rpn~H_ox+dNuq=F>)vB`F8M5OQO370qR)5jMv9UeSA2y5A;u@ouq*Fv^) zu|f(D%L9$n1-3E|Szy{K`qx)m>-)!TlP;ghn ziZRO$YqC}w>10oFb*>vTlC!pC+K~$mU#A>L(8KY&%3eId7u?^Xr-cPyy|&9jcXrO9 z{(4xxvJcv9V+2(AeC*H?UVIE43-R~xBemXf=;ryp!PJ{&_EkyB0yoR3P=WLV#w5RuTIfXc$ezCs+Dvg_0c34HrT~_gG$onbk%iA+N&EQjG5mtk73@M z<T*xE@h&pEgWH{USrdvqL)S~IR+Zs=k6WCZ||R5~}V)Y(KwSjw(B zAB=@t4IW2MbEdx5iZk852odwYB@ufCExn5H%LO?>t0j8+sge77$T-w{S{MAO1eI3LML)csh_v*Vda@Dk^D`BH5C9X#0%fUa>oN&dw5;K!P zjv2nTQd6)=GaA$*{#VkBafSyQO2pO^X0ouLeI#ROFQ1X`^}FJ`?c%w=7w?C@6YEKFuy^$x$jL| zZmn;(XGa70HM(k~C~;%SCgH>5miaN`Jiy1TEd;k=ej|ooJRHKdR_GV#UAVwLwNbTD zL{b+RG_;~0IkbrbJA(q7wk9w3NuqVo5Yb?s=bF*o1Ma{>x;1m8Ny6^PnCrS59~bjn zUBz6T|KRp}7#4zezOmx5$wD}lg*h5CNFv2h6c44FYy0gP{K}EUgJ$&yi{NU_S{_f( zZF(@iAB9#moj|U8s!kK`rgJz(!Gp;Pu66H@J-&tpum!y!Gs<7dOI%@;y7EJ4$Hgkz zhlMQTF*u4+bY7*SwoiOUv%PFm5yOs_PCMq1b{kc0K5J_%8|AxQJ*IY;BK^KFt-fsa z^Kg6geVH1Nxu*TJ;<3mgbJG6#(xTa3GpVR@&B)+@`}rZ=iYE7`!{z#zTqgR7_T%Ku zbzjiOas>S{SY~A2m&lAJWuXc8L06K>ev9wwLV`o9_zP@9>t(fRdn5jwn&JXYW|3pc zummVD%Tgf;rY%Ru3Jud;nSCm6jP`ZIhhH8DUl*Xq!6Hku_TH@!Ys2Y|#+AgtpUS7DHF| zm)IrG!B|sT3ZMFx3Hfj#Tnjm@IRthRhJ=RKVEAz(0;zddwo6CsLhb6Y{$QXuU46^s zCBncbU?81XGdkNkI@Gv=#5Fr(K$$JY{t^GsdZ|=37v0VvQv{8;nh5Cpgo2?02^ON- zLP;jQc>WX-WFrc`#tMH}ui&o}iV>Xcq7Wa=Wfp=wD;+~}amtfxU1J!a{`OF0Xx zG6&(sHRb-3P{)ik0dAxniRbo-3GHWnd<%?jcPL>AW*=QC@uCji2v|kjh4Nq6xOTS1 zH`)wzTp5M1CvlZP#Ub6k6Zhy5b1J=)I6Vvj2p6!{_x9JEs*_A3KU{2e^ZJqVCBpc} z@+?CwZb73OnnX@Tsk9&2};PrvHn)HMx`Zqq>kz2d_0Zvt{**?FacjZ9K$@p%t zChGzB@}x=bfjCNf!Y3R+jKOKGvTBz+z^T-mHxWh|1Zy%J@Xt@0WFAP_fKe$hR%2)s zmudO~K|T@WTQ!B;M7aU`6AR_Jlm)EY+P(C0PRp~EM_fTJBrp_sKi&tslH}LscIdQ9 zAk)zByA{m^aTXF`fddXgW@OD(W0pIw5TEH`m4N3AW2qsg5p4fn#qBpjSBQeoU4fg| zV}00=(|%o#^s%m=;mAJ-Xd)T1ugJ_c1*`0dzPz8V+M9TOgId}z=Ex6zxLrBE)}tg+ zY53YO67;_!K^g*@n?;Cs^bv!^QFfz=9m+qL^wt(CJO!I9hF7Xq^ATO9NeQr?EaeOm z4Qrs_qYN;@h%8>YC9LID*-ME#92;7H1?%qa^AA&0A$GjgYIvqx80bNUlaL52OYI|q z$;GPv_&HOHrnoD3K&_|Ae=t7l(|SCW|D{XRMeQ(&UJ$;=1V~HJUDv_~id1QI=p<$H z@;&CWt9{MeVbQ-Lgv6W>n7!QEC5aRdwt)p)gVqT$CA{2~Duzy-za)^wXq>7bwzz*n z^mT?rBD=(*oHS3V_$?G|`(6hpVXXj5*HpZAR6;&ihfbsfG8zpx6jaFVZ@AV$)b9|5 zvQwxS+nCZVN$xj+GQdip<;2e38rY+%IcM#S3hliTYFkfIwy0f%+vkehcpmC#{`p}^ zvKg!!S`&%+Ow;2`Ty1#ZJ;*#X{7$h#Q5?JIBGGq;4@sd7H*kk3g7vtq>!g6s@E)w) zut-uAyjrC_oA=I$UCTqw1?#a!f%qf3)+&LiS8|`@#$0BClWav`G6GPCpG`@>H;YGD z1o>zV-eWhq(4?$+&b>QH^-)rL;0;`!gBJL!I>oKZ(i=Rtz~@v;>x@)n9B(GFLmva+T99y6F~Lfh)s zrCt}h!xiab`@|=*>-`Pdvvp=N%Y^f|qd%O!PvDIJX3PK7QLD9XwZXY$f&Bc&4H?XKA9#Ef3`^q*M+ow{uE)~Som@BE!6 zc1)h|&!_6Hd%0BC|IUbE>wQ>ImS9BMD9|EcQ zojy$ix(xB?qBpm7`D}QCgpu)sbQHQPt-JWWVh`&QAK6o?kov>) zfKgBCtX@M(CcrWNnsJGY)omc!3A3k(+CwAc|OtOAf9Czrq?f1*~Pi-S4-jyUm zyJWILNx0P4UjiLEU|0Sl`1f!=TE8w2lii~VZmrrsbpsjVEMBl(-$}rs>lJ`FU1&cT z9yjEx92?=wf61@QJN?e_4F#$9t_ymq6`^!57OpCp7oEpGO58oc8hA+CGB+BNXW&{^ zYX?`WXo-o)_|{cLM>{?kwSi{AT^QqJH2^PG2%;XVmc4^DmB*X<56hj#+FntK6uK!i zQT?1%N^rA|uxzzHc2!8yPLIi3ZP#J7J~&nxrpw;rnz9-jUN#dR9P?RpVG~IW_T@UI zdC2U0xOPX>XZQB}+nxsn)EDu&f^`FTYfCF2@V1cUXKg9(Z+(x$60ODT3|cvp?@qSs~ z{uT>>=`9ZLC1b`PJ=7Vi>)gbjI?biolRaWr`|@si!f!Ft8yg)@9{l|B`5W(t)OMe7 z5J$CM%n9T$m5nPRbJixa;mIiF=NO^8zsF-##?2m@^(7o>eG(Y6)l~= zGx$RBJMSL$*Jj)^%WY9re@grFBx$*>arvRX`uWa8mI7v<)ib&|pHBe2Crp07S*n?4 zz%N4b2_$wPB`q*{+~TrQhpWaf6v_=NXxGk?0f`19_4Ay`y|+eOdqgn9o!hmJWp$zt zM~ioNM)A(|&DL>8NuG?pB>XxdZqeDtl+R(eOGI1-S}kbUxIki z6#rw*krdx;BwlqJf>5tI{~uL|VY@L3v4w}-887Dh9!B!;dK=g7PMGlqeu$B%X}wM^ z=w6^-pJUTuP|!8@l+(ShMTEfbz9Y5Pv?NjU*52EB+ADVYkCawcZ<6@Vg8O{0hby9L z-~U z5#+*cH*4fA{eb|l*9hhLt7@Y6z8Da+uO!~)uU0%TWTy6GL-6fC$w~HF=uL-xkt@>oZ?ahp^L6q z054G@;EZf-UbgoblQLjw0eU;;wMN!WA-D5y$24y3#kg?my^hZ@F0mVwcJ-cqv7?hR z%UfKd-v$0Xh)1r!jMONzY7>z#_*6yUgs&aBZWby*JJb zUBor;t0?C(>uteCM%&Xmq7N2}XD<)Eozot<>fo1z)^@7)tr#vUcAB)2fGI^_H+6uB znXz^2n}O{58UjdB8&xzgy*m1ekB%^Jz!XywUKv`AIWgTIN-%$fgSjjg*<n9Sy)n= zC1LwD(JE6?@xU8A);U1x0A!(ls$((iZ1;c~-it?Hp3bd(()%(itOBjM0j^k8{KUY@ zL6`}x4yrt=i#2koinXaJ&S!h>2`llt6xM{qj%f~un`o3eN8By)DqA^$`=wMAbUMh| zua<0a0k=Q!FBn#?76aAtl=;Y;Mh41@?XSHjKgtU#@SrPJVu!XM#{JQpdVj&i@ZEHY zpJ_L@vaHrK-&n0*5_|QUPYAuDDY5v*x2w$X^YD9zf77PbFRBZQTteVRstW{4jB7Dt zT^+ZY~%EwL+zn9f<&1GPx*>oeX_*Fha*&}4pik8dQLMcPDK z(PQD8>Vx4*UvK6A$Y4d)?{>vIMbsTCF|*k91B5VG9Cwf{!$uX(4MEfxDQC)-2=%0TwPe3^ zV>UK56(F4@$k~{A`{LI1183iTri>l)z4fYnzeI=(Jv-#>XH^#u$||D{UeQTR!N#&+ zdH*5w35H;s!;QHrM2_tYTJzLA=4z;+-Pse^FFnJq$!|o>BsLbMGxk`>@~nLw%PT&T z*N5(*M{9Lj;{v8HD@#m{c?2`B#NArqkUmuj)aQ`cRrWq02k#I@R^@SXxi~5%^nQ;X zkRs32ptBiC^(nn0o#NRN?y&WiElxvf|7S2}cwt+_sqDs}|K8ELsvHGoLB1&yAPAU# zgJZ5%xP^V^OdHLoFd(|s6LIV6PWv@`^<@vQpXp*e-U^_2^m6_7-Pu%s`{~A$94ZV& zDPA^#%wj<2m*+@c9E76lURMKe#F>vn7b!A_L)Eig9+Dd-vyejJ_S3e91PAp64-!?j zrN>$V4nNHt#89mN^%@9R&EEorGec#d*K^y{t?wj|N1YPRArO4Ka{<9uz}`?4WW@hiQVICJNvAh zfN{*K^ITSifijmU1oBmn+Wgdtfk3_jfJ6+G{h?c#4h83}d8?f|APb2fq18simsxSb zf>>W2+InAjKwF=e$#7;O zV=+gP(Sw*Aq3QPvzp&%$o`73Qga1{m;Hv{~v}+_IL7OXWNjYu|emzNZz2|)N?a128 zo9~05s!NDay#gA)Q^j8+VlO#FG`fljleHH!_LDw;k*DR+UF(C|0RNfMXqB6h4F;UU zgtG>T!h%??CwHkZ^1Ce34n?ugg~Sa$p~j8wmilj_Zrp-m&P}l0c4YD54xi8pl6Yii z@C{7>dbKAv?D2OrqGs7!SUi^N0#!}>%m(TB1!&#-?BMi#*;kgXL$;{By+pC_XIFX1 z<)$W@A0(}yVFYV3$j%T8nfTNxK_*0Ax-yvLBF-GB;zPW)LJ{Brty7Occs>@`-+th1 zn(H-)qNO;t>!O!E1%n!EtQ?1Q&9c~3l_4tpxIhqMaz{jjOc?e?5d1K0fQkaT9Q#%_ zWG$TaBiLWw@gcJK%p%TNiY;uArQ{yBtB{(Y7-_(^j`CRo@g`?^aNOHa_eloV?kG(*SSKL$fS;# z)e09AXgLi-L5Ty&`#ghAozpqI&b4%3(eN2o+=k9D-GaHjG@ooxWaR}Bn3WtzMQBT0 zn+y9V236CUj zowrKvgph-~B~voNZ3WW^&_ZSno`TmmmnIU4gEau8fY)F9w!UBuR2uFAY-GSihtWQ@ zdmJ=IG^nb{B)^u7iwpccq9Hixab+qg3>yKxFB*Ap%?gDgZR$kTsHD6rlo>FOi7YNE z7is6qtq3i$*^3YUQB+H_HXb--$iPYAGJ?XbUfH>*Kjrxt&g^^x{v!#-#RNUiCtP^0 zkI+`k;q>a8OA_U-4%|J(L&>9NIeM0FYzPU57f&yQK0m*HE7CtA;dmxq;(OsL8- zP$k~{BQaycX&BTI(U4DqYgH&%;RCM@$@Qw1B!yb^D|e$#H&-N43C=cAiAR9qhYB>% zs+3=`r&fB1sJF9jI-_;VIUJX@8tW3hE@dCI%arU8Jzt(5to~{ zt4=~N4%lvVk>i<+v{0>(HLXl~OQDCKqvAH6^NV-cgald~4nOkwx0)Ob*4&Wy6P|yU zDkH5^hhYYP&z(2!0I9Gi@YBWVSo^}O#HDb?n0wX#+;3%N8hak-IiaSgZf#efPHdMO zY~PDM`s6{Ks_tCW5lJ<3I23Hs+UFi%0aaXR#d8ANAC@d%3L*NYsu>SPKa6@=%zgux zVK9Dh5Su(^@wre1>riwvuwU+a{{g)ls>|$#xUaikeaxTI z!#GZxZ6@pIXNC8z%~xwyje=$|za|>(-+=j|fs~g+m zR=o8uc$LA9^5<8Ka8)+HQAj`fWF%p=J96ntP&>*vqrfl^j#l|C@9|4&sqbi~8-r6{k9y4_w~lrs zd2rKleVt-dZ%`xs5Q!Xk%e;bGsiG|xxGQf(ET`WLv%3ZVRbokMRE*(#G`Zd`nHz_C zneM*0t4Ad)Agi*iZUQJqBWf@I(yFkSXrjhnklJS(IY+~JtOs0$lppjLtw2iG-y7SY zp8Yn;Z*T6?62Ga0F;E6l-+@yClF1bovGZ|?3wHZQ=0z0T&l`h5;TTOQCn(q4YB;cbfHQ> zvBSUU!Fjf+$(eEY|0fUb%TQ3ykGO%+?s^Dn0$ai#=lfRj7)8F#A3^HW4*-y)uOnf{ zu$X77iKe0ebS#(O{VGlRBi8o(2Fz*=TEihwHN^puyhcKuc19lS9_>Ht{h#dC zuk8l#5}KLQ1KPw?BPg$FsTT+SAjh5eDy{*D3po_c*$1(FUW%kQ$%RF1i+VNyx^)&E zAqDN70lriD1H7Avi!~|AgZhCg*Hj{JGdpwl54At3Qn`cS4YQ1}w|%zMzK~`45m2w^ zx8p6(?xcjc4z*d0;D7xr_gLY0v4cx;`zDJ?*{o!Ec3!OKgbao+GY1fuhj?y2F$FEd z!V>w2;N8WtK0NaOqTtFkm_dM$|LBcn;!{6{PPr^^4@`!G2T7?d@5{Q8$*rurq1qwT zLGi;;U7RC`=dArN1pP1ui=Wze3@mlG7Y9912GN3#9)90cdZ?0z*`j>=aXWTCzCgd& z{8`qR|9|JDq7HtqDP^6dJ4>zoD`J)S`@}acj!gSILA{zF0n;CJI_b;V*Df`^r`lW8 zI)m~wy#aqIDhaZ*NqmV-=fO7vn}a2N(+c1<^|jCdq=-8JejwbjU_i;wezu)+KivJXf8PkeY_tFr zsApq1()&&mP+D^n47}dANFa^6+en^ewK^|6kV-kDJNvoTjdv*8t*zV^5@%Ms9C7n1 zv+t&!@7}=sM`8t`0{gF+{SViM3iS(A;;+aR-}?MvckSuqO79#!9(h4xvv(=;_N(nd z|9|J2!>)_FE>dHo8LMo$;_q8F1pSy# zu;nn&CI@C%FZ}QcSK8YxLqK=_9mHmBTj-Pz{0LN;cfLDlLLycTevpV&SzY~XMyS!e zw-HoxJi;>-3`dCAf9DxFunJdRZi(W2&iL#+OEyr(H1w(u>)D?2izBd4^?@%-I(I6u z`|)1C-i_<;!HtlZMTbkx6J%Pmo?bNT)lVzDKhhoXI04ig*G4e3RShCW>k}Q zje+eJe?`YPAUvXg(E)nSZpDd5*4{Ckf;~UfVx!W%caHam6IR(Xb8uyf{RZ}Oi z`krHc&{oXQ#iczj?&J^92Ztdf>PaeN()#1n1S+QG;il5&m*5BCy<~BtN$y7W7wD$$ zn>crU1kkWNVA7;j|EUE3eQa`}06YGnZRiPv_V%v#R1!4@@r7)m3}Oz z;7Cm9OOvU5sXZl2Kh=EJf=syTrsoM~NUms=?`H;o92-Xr*As?%*;79N{Cp2;TJBEh z)F{+Cx@)^Ct}%dYFcPGLFnKvVktISx04K#BM7b;@LC-C6%=S)v0gWDD_r!4 z&P9jBfIt{f358j5$Y8`I2TZ;{z^t#%W$3Xe2 zlJeqTRopX^h;N5aqOUO-4?knHB#=}eJj;aADAuoEk-g@dOqJZ@jRWW}ho_>hF`Wn6 z+t5djY6^}F3T_5cr_k`dH{};H?TBtOg;NpKpI}7visP(JR>7<%(=;3bh=JA7oVI4} zTY{=nN_gTV*~1~(_8F0?vqj^w?Pn6}F_x1HEK(LiR0n!-6`8nK* z4_$vV?I$p^>(9Kw6PE7GPst#l+n?mj^Ms=I4Sx`%g@qu2o*XMH?cQwOqDFz(5xQ-x z##>S(G(3O>`s_1_A@LBkh4anu7py=U9QuVCN1lL5lU6#g`4Th2_MnH9_PHyGn*eUo zfH7nq{MnZ{tq?nq5Lphe0jYB$+QaBKNAL^h->hFPcPUAIUT7f1S0O?Cq@dSm6 zH-%hztU~+QFx?L=(+lb;Ft1r1kVLX(^QO4y`p|iYP@5Aovd!|1%@#&Vi?QZI=c3eJ zE_3>o<2DWvgK!63x+3KM>?sIp4?~dek)5mJyG|~o+W;4SLN096inZ7s4#Eqh~L zZfE&56@9+h($z~%i~NhMmaTh#-HgXgyPRByM8mk?Rb0g^z{$q(&g2xMqK_1grS#!NK@v`kf&5MRvidk?0S_O=&MqJ1`ZZK^D>3d&x9J7pf%Hrcl)LJ znDmzcl>M9&$_RxtmbQw5K#vok0=8clkTU*)w3P~UK{#p%5Te`|MniqjZtR_Edj*JM zewO^{y`PiE@<^F073(?LPcvv_WUE#WVnjbIJ}iY?BBwuH29Bub^V4ri8W&d%*=`)a z`TW%IR|`wA4zM5USy<^Scyvw-eQ#Pks{hP^C))%-&&f=QBmHD)3leajd}y1{6eC>p z3z}!&jlZ!`op1?VP(94Pc4Kd<4$^`8HaP)2Q$Q9U~BxbL#~7y<(HnnC9F3+OCdcLVB7L_63UyTjyA7Jp*?i`)yqNvf{Zo|QU6B@ zTCXF)r5>H5=6g%fWj1+ZYZDT4T4;el#{<7WmyfLCz>fZ-xKOxwb~0wYO?}gg@vonM z0L<@5V=JG~1Kt%-RiLO2>C#tPg?7E=LGbc83u4j-BY-ZZLSJr4{ru=r90mijN;gf@ zTFkRgUXTWt@!EU=fKE$s{jt62lOna4``Yx-;`>`&JPh(Z*#Z(XDmz(=C?TJhP!P+X{5^m-L;yCqVD)KOML|$P=sJINnltco`5b z#D#!~uk|zCLUlQ3%H#LG?TN2hNth zh{Xe@CvW;$AmR*+Pm}*j?IJxJ$90a@_zx7t?erNOs+9eO!wYP2n89EYihSJGWo4T= zy}&l4{_rcj+d)>oVOeZS9B$ZB*lV4jAUuvT!W3=YQAn>ga~oM6tyQkD*qYxdEa?E( zWd=;mG@xa>WI~5*Cd0AI_S&izjK(4ZyBhuFZ-HIkg!s*%j(D~cSug$max4`aHOL(T zt$KZ6;F`i*vCuP%I2r}gy!lWcgV3SsFYri1V>{^AFI^cNeRAgE=Jv=sz6*tY=XWrf zO+gJUa90AkhDDHxK@6U)S-trxXu4F$Fk81EHTUb+oY(pVtq!%D%Tn??rL9>Oq!(OsA;H}p zo$cRAtAC{uhyV# z-A=-_m|){`iHYP?{1B*sVJLV+1XP01Q^lQ3@V9UU$%nki)1@+9I0fS!u{Oa%&Js>~I-maI894ws3x>9H z13$Q%^srry1|f{D)BTsC3JgX|nG+}(<~W&M*`=Gi-T?B?38$v!cp-v&Vb(V#IHZ2go zWg?sMLOy#4%9>!fOUc=s!<@v`VBI%CjGlH+9NE3MjHPy(&uYCIMr#}qHuq@m>*a2u z+)_`cj4Tkj2GFkeQA}pvjnJl2C_f<|E~?f-0crihV3`Okf`-%iI=H`8gjn&N *> zL*bS}EKPftT474i{Rwz%LfZ`pZs1%S2)De}wb`8({bEk>`z#Y=TQkB!-awW^e>O+s2I+4tm`k_yb!PML@y7zHBBd}pu>V1voPrEu0%!vT zavL@7HM_*ia&))=Z688{7jUbJ$6$nZ18ZXvX# zK$^A+O7U2qaS?~89BBN+x)>0c{)pL(<6xtg{dh(8<7qUq-S))PN$5!Q&bn7@3*zCkAQWv>j%RD4%>m(J z=|=<|cn@HPpRCj8v+oL)H;A*ye&m-^c>tB&Z>D?g{#XEbuV6!MaxCWc9wcb zn2qC?O5{@`=w~8YSn5?FI>SS_Ih1#f%j)1PB-Z9{>RM4R$$ub@1p(cILpGEvcP>y{*sDN zfd_eJ{yQUCnEe;}DS$?noY60Mz<%(T*E-i0FhKv=zZCcMGVr6pCc0RS?))LiZHmwT z9W!1H)%}+sE(v(7ZtI85;0@fh8E&NqMBsc#pNc*_o2n2efam4{Kr#Ktr;>Ba(Idd8 z``*Btyq6MgSr2-DBBbo$`!wo{`JUFfWhh&>eorm1-`mr?v zCvyIu7AirsJqs2yiT8_Vl|HwD(#nFft!R)=L_b`v=YvX!{NUE*?Cw3o{ z>MenfSCNmbazF+l zop;0c$-LLb>@S6ikDzyuKUCILCs&c(KZ{>r)~29e{&pa2{Ru9$(oFvgSW{SAY_9=( zh;E^lx7xrw`|0R%nry`!__}(JeF_j_{i@65t={}HKl-=5C2m5+A|Nx!+$6Q{>-a?Nza@urzCRR<{xUj<6l z;2+7##XCT7FszrTkdn0ZG0|837N~!#LnvA_}V}jXLo26j=N4I53*3q*6A-g>V0i^ z9^`w)bKXXf0pvL;{4MK&Rx3h~EAlUE4FL#YHhqHzD(p(Ud3(>PX1?FSN+JGb5hz8E z0nuP$r*PZ_TXI-H@afv|Lib0FRHH!Lk64` z&eBg!W~^E0_*$9nf*dki@N$gCTTs>qYO?kDlov4hH!1I%g9J7Da+xcy>^zQlS#tz` zq}2xiL0uc47Q`Fspyk_&+tle!h-^s^JsY+@{;&Jb>%i<2Obe4Gj^rxbCtIj`fVcB4 z=62u98I~=n3woP#E+KW7>>xl^3hjnbErxVn&vkj&xMF8iQrjmK?zWQGjqDG zO`aWr8y457ql@B}{vAl;H@_Oc&ieYWzsPv{Rbs@oiQxGlnr*+2TX{LhcPji3Em>+= z&bt1It;25}uE*V)zltoLB|9Lzq@_MsVh}VUjXkNI$-(2$?phpTVsZKA&hk+oP_^b2 z+$xRX3~sv^blFZx(!3$MR^V0i7-o*(G#GWD1xX{if!g~rQpIWy0$??&6&ks1;*Bbn zGnerXA3>%4cRDtu7tEOp?Zm8FP4~MyoUfN2>lONC+)+*~MHMmuWvC-}g6)(H6O`4qM&$xeO%N$(K)o^->KNuSB$4%oOBo zeviqdgkee0G={uUIl)S);CHo_8w8+j&@{`Nu$id6SP}wL#sGzy7ObYQ5M?L978!7^ zz^)RA_`#fb79an71}3~2suaqoSn(W&0(FzX(}YMJ)})k<1;u#+f(4u>Lc#wzu^Wht z)2>0s87Ql!y@7r&BwYmvkyGqmBVm*~8wu58VHWusH^V}dBiTSe31v&H zAK_5pG*IQsY!C=lA%jDhJW7M@Bbh`ir2hv!EeIFvD1{~Q;Cah6F6Y^dF+6v48c7%(QX=b*N*9iprg#;VM z03wBu+aYq$g?xZcQsaHbmy^^;j@GG#q}?+@Go`?Z{(FXE_#&`_9x#JY8CxGn+Y2wB z?Qs6b&+o$bL5xD)ne*R3I_;v{mEyMLc(i3(u(co!`2hY=P{5qGP4#{AQrPIMoZn!nyCj*+?y&EMVY0?y&|8jn(12^7cKXsy+*9CaWh-8eJeb_ zweUYe316+NAcwr3k09346dTzLV~#@*3YBih|3jz17|FfQic3j@&=u;T?o0Rru`!d} z7D)A?aL`r+h$eVQT&*~m)Qg<`)G0j3>|Z|HTw_+J+;-sgS-i3Ljrru3XXMy3mH+B- zaycdt-4QYyeV&S9^q=(xg`nOr#)n{2Jl_ZL37V$}_z68I;G4(NyU)w%T?f&vYF)vLz;sTa_OrGcqsH3tN}qkjk!qJ2-ZlKU*By~AT;H(2#%sv+ zjZyXXMabj)b=Gstd|tJi=l!wc-cR<+-{19M`s$}FU8<$gBwj_3%=!A-`N5Pp;Ap2` zQlFZOrSQ-ve(|>|Jn`NL84zMwdbV{?T(kZsh}3&+AY@hne2??<^^4Q}fH#~Q)%#Gv<*ysf1;rfjZ%{>eAs_h2 zO`FXvH81{lc2h6wOuxBv#u#;(C2{D;iMQ2d+J><>I%2GKyC&?MsDh|HCJ5CMaH4-6 zlt%6P3(3~2sfnQlTme)nQ=|a$>0T+Ybcc`>O=iQ*K;ipa;RxhL@bTEWdKVbee z@etts8*9dfp9=tM?-D1=iQ=WacxUf$nZ2eIM<`Zj5<__eNrY>Ok|lS(E!FN&b}vNjiP#Lkii>Jl zJ@=1}!O;urmS@U~Wu^d~<0|RBZB;Aag-Lil#Q(2%44MjM@KUj^GFXGc&U2vStFvYr za+p#NAPyl#fbMdu!qmO>-0X2z`-)7}-Il(qKf7SOl__G_f3#El zWdkCzg!X(Zl*|L@`8WEHc+^*-x7sg;SbLXt=;}+|1+<3M@sj-=#rZx-4ME6g(G5gz zh591@yy`JaYm(&qHH7C9ia=~#GlsSH?KKdj!1PV7IV~6g_sfT$7 zNwJzhn-z|UjpJHS+|MXxKCzP%kU=qwOy~p|Z%w?n(0GKnf5NFIQb6c}1>6~?nM^q{ z@e*f2tpGhba_EmR^`B4I!mp72-f|bQh)L!@Aad9MDI14K1I6zRJMt`u%xIzAJm4Z| zpXWk*UBFu?J_~{lC=7Pv5IFR1J7PgRN2mh<=oqLjLctGAy(S21=BE+9Z;K|>H_5dA zJ6;<=?I--x@D=ANnIcFo1y-sEYL^go7ZxA`LYXNd zsIeU&<8_A9MC;Uh3JAR-lU8%;EHD=Kl!am^GHmKyIS}GNXj&Mn4j9F%1Z=%PhrIc+ zH3;jVF|FhR+{)v?J*ophoxx0d_%)CLx$kJGGY)h9TP*UJWt zG7ub>V9o*rdnO#9+=4g~6xE>LP1tp~gBH)C9DO|bI(p3@0?R0d_q_&&%9G z*0iPXoSYGWBW%6d88%NRTTf~sUIr-Wcm_B*V!91r9$wD9ynosnPoiZE8wN6AhvZVk#%Vk5Pi+?| z@B1wBY2gD)6HC9f$(WxFwFdNU{Kn|Z>fSNr6t$hjGV*E5t$#6?aL^i+sc|;l*iqa1FEj+It z*1p9!SoC?f8Wh#0;gbF-PhfjnOXx**ybrd5q`3HAxC#U54?c$y>mUBh3T88 z-}0kBRY41p|6fU!{sLaI5bBeKq{zt*3D4yU%WTC~bqB$`Bd?x@%$#v*ayz@szG-ti z%F$-mQ5u&2UwmVnAtO&O7L`SkQYZ`E&+dixh`w6HtcqiFb`pkXBH!Q{CT|VTC~?qg z!=*!R?9*ti<>gVbHGq^5$@}E!#g2{~(b5Iff6D>Z&t= zszjEsQQ}!;SpjM`6-OahN~d37Wv{LEIt8F76{qX%>NFLp6U%tBfx`(lxn}%8q>&UP zYNP~k+~t&GrMG*r{ov}?EEAmtu6!|p(1esGxW>&57O}uSzR#21!OIb#4#f^yf`{D- zj&h^aiF;1X0)kX=^zX_e3S1fe%W|vO9VJ^FpCB0bL;9TMHe5_ z*`wHhggSwl1+C^j4>QKZ*DaLeporidae https://www.ohdsi.org/analytic-tools/whiterabbit-for-etl-design/ + + OHDSI + https://www.ohdsi.org + + + + + Apache License, Version 2.0 + https://www.apache.org/licenses/LICENSE-2.0.txt + + + central @@ -78,7 +90,13 @@ 1.8 1.8 + 1.8 UTF-8 + + false + ${skipTests} + ${skipTests} + 1.8 @@ -106,8 +124,42 @@ org.apache.maven.plugins maven-surefire-plugin - - 3.0.0-M8 + 3.1.2 + + ${skipUnitTests} + 1 + false + + false + com.github.caciocavallosilano.cacio.ctc.CTCToolkit + com.github.caciocavallosilano.cacio.ctc.CTCGraphicsEnvironment + + + -Doracle.jdbc.timezoneAsRegion=false + + --add-exports=java.base/java.util=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + + --add-opens java.base/java.lang.reflect=ALL-UNNAMED + --add-exports java.base/java.lang.reflect=ALL-UNNAMED + --add-exports=java.desktop/java.awt=ALL-UNNAMED + --add-exports=java.desktop/java.awt.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.image=ALL-UNNAMED + --add-exports=java.desktop/sun.java2d=ALL-UNNAMED + --add-exports=java.desktop/java.awt.dnd.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.event=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.datatransfer=ALL-UNNAMED + --add-exports=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.desktop/java.awt=ALL-UNNAMED + --add-opens=java.desktop/sun.java2d=ALL-UNNAMED + --add-opens=java.base/java.lang.reflect=ALL-UNNAMED + --add-opens java.base/java.lang=ALL-UNNAMED + + + org.junit.jupiter @@ -116,14 +168,41 @@ - + + org.apache.maven.plugins + maven-failsafe-plugin + + + **/*IT.java + + ${skipIntegrationTests} + + + + + integration-test + verify + + + + + + org.honton.chas + license-maven-plugin + 0.0.3 + + + org.apache.maven.plugins + maven-clean-plugin + 3.3.1 + maven-clean-plugin - 3.1.0 + 3.3.1 @@ -146,6 +225,125 @@ maven-project-info-reports-plugin 3.0.0 + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.4.1 + + + enforce-java + + enforce + + + + + 17 + + + + + + + + org.honton.chas + license-maven-plugin + 0.0.3 + + + + Oracle Free Use Terms and Conditions \(FUTC\) + + + The GNU General Public License, v2 with Universal FOSS Exception, v1.0 + + + Plexus + https://github.com/dom4j/dom4j/blob/master/LICENSE + + + (The )?Apache( )?(Software )?(License)?(,)? (Version )?(2.0|v2) + https://www.apache.org/licenses/LICENSE-2.0.txt + + + BSD-2-Clause + https://jdbc.postgresql.org/about/license.html + + + HSQLDB License, a BSD open source license + https://hsqldb.org/web/hsqlLicense.html + + + MIT License + https://www.opensource.org/licenses/mit-license.php + + + MIT + https://opensource.org/licenses/MIT + + + Eclipse Public License 1.0 + https://www.eclipse.org/legal/epl-v10.html + + + Eclipse Public License v2.0 + https://www.eclipse.org/legal/epl-v20.html + + + GPL2 with classpath exception + https://openjdk.java.net/legal/gplv2+ce.html + + + + + com.microsoft.sqlserver:sqljdbc4 + + + com.teradata.jdbc:terajdbc4 + com.teradata.tdgss:tdgssconfig + com.simba.googlebigquery.jdbc:GoogleBigQueryJDBC + com.simba.googlebigquery.jdbc:google-api-client + com.simba.googlebigquery.jdbc:google-http-client + com.simba.googlebigquery.jdbc:gax + com.simba.googlebigquery.jdbc:google-http-client-jackson2 + com.simba.googlebigquery.jdbc:google-oauth-client + com.simba.googlebigquery.jdbc:google-auth-library-oauth2-http + com.simba.googlebigquery.jdbc:google-auth-library-credentials + com.simba.googlebigquery.jdbc:jackson-core + com.simba.googlebigquery.jdbc:guava + com.simba.googlebigquery.jdbc:google-api-service-bigquery + com.simba.googlebigquery.jdbc:opencensus-api + com.simba.googlebigquery.jdbc:opencensus-contrib-http-util + com.simba.googlebigquery.jdbc:grpc-context + com.simba.googlebigquery.jdbc:joda-time + + + + + + + compliance + + + + + + com.github.ferstl + depgraph-maven-plugin + 4.0.2 + + diff --git a/rabbit-core/pom.xml b/rabbit-core/pom.xml index dd527e4a..4fdb1ad4 100644 --- a/rabbit-core/pom.xml +++ b/rabbit-core/pom.xml @@ -14,9 +14,26 @@ UTF-8 - 4.1.2 + 5.2.4 + + + + org.apache.maven.plugins + maven-jar-plugin + 2.4 + + + + test-jar + + + + + + + com.oracle.ojdbc @@ -38,50 +55,39 @@ dom4j 2.1.4 + + org.slf4j + slf4j-simple + 1.7.36 + + org.apache.poi poi - 4.1.2 - - - org.apache.ant - ant - - + ${apache-poi-version} org.apache.poi poi-ooxml - 4.1.2 - - - org.apache.commons - commons-compress - - + ${apache-poi-version} org.apache.poi poi-excelant - 4.1.2 - - - org.apache.ant - ant - - + ${apache-poi-version} + org.apache.poi - poi-ooxml-schemas - 4.1.2 + poi-ooxml-lite + ${apache-poi-version} org.apache.xmlbeans xmlbeans - 3.1.0 + 5.1.1 org.postgresql @@ -113,11 +119,6 @@ commons-compress 1.24.0 - - org.hsqldb - hsqldb - 2.7.2 - com.healthmarketscience.jackcess jackcess @@ -127,6 +128,18 @@ net.sf.ucanaccess ucanaccess 5.0.1 + + + org.hsqldb + hsqldb + + + + + org.hsqldb + hsqldb + 2.7.2 + jdk8 com.amazon.redshift @@ -202,6 +215,12 @@ com.simba.googlebigquery.jdbc opencensus-api 0.18.0 + + + com.simba.googlebigquery.jdbc + avro + + com.simba.googlebigquery.jdbc @@ -218,10 +237,17 @@ joda-time 2.10.1 + - com.simba.googlebigquery.jdbc + org.apache.avro avro - 1.8.2 + 1.11.3 + + + com.fasterxml.jackson.core + jackson-core + + com.epam @@ -234,5 +260,30 @@ ant 1.10.14 + + + + net.snowflake + snowflake-jdbc + 3.14.3 + + + org.junit.jupiter + junit-jupiter + RELEASE + test + + + org.apache.httpcomponents + httpclient + 4.5.13 + compile + + + one.util + streamex + 0.8.2 + compile + diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/DBConnection.java b/rabbit-core/src/main/java/org/ohdsi/databases/DBConnection.java new file mode 100644 index 00000000..749c4e76 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/DBConnection.java @@ -0,0 +1,374 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.apache.commons.lang.StringUtils; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.utilities.files.Row; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.sql.*; +import java.text.DecimalFormat; +import java.util.ArrayList; +import java.util.List; + +/* + * DBConnection is a wrapper for java.sql.Connection + * + * + * The latter one instantiates a java.sql.Connection instance itself. + * The constructors of DBConnection ensure that one of the following is true: + * - a java.sql.Connection implementing object is provided, and used it its methods + * - a StorageHandler implementing object is provided, and used to create a java.sql.Connection interface + * - if neither of the above is valid at construction, a RuntimeException is thrown + * + * DBConnection provides a partial subset of the java.sql.Connection interface, just enough to satisfy the + * needs of WhiteRabbit + */ +public class DBConnection { + Logger logger = LoggerFactory.getLogger(DBConnection.class); + + private final Connection connection; + private final DbType dbType; + private boolean verbose; + private final StorageHandler connectorInterface; + private static DecimalFormat decimalFormat = new DecimalFormat("#.#"); + + + public DBConnection(Connection connection, DbType dbType, boolean verbose) { + this.connection = connection; + this.dbType = dbType; + this.connectorInterface = null; + this.verbose = verbose; + } + + public DBConnection(StorageHandler connectorInterface, DbType dbType, boolean verbose) { + this.connectorInterface = connectorInterface; + connectorInterface.checkInitialised(); + this.connection = connectorInterface.getDBConnection().getConnection(); + this.dbType = dbType; + this.verbose = verbose; + } + + public Connection getConnection() { + return this.connection; + } + + public StorageHandler getStorageHandler() { + this.connectorInterface.checkInitialised(); + return this.connectorInterface; + } + + public void setVerbose(boolean verbose) { + this.verbose = verbose; + } + + public boolean isVerbose() { + return verbose; + } + + public boolean hasStorageHandler() { + return this.connectorInterface != null; + } + + public Statement createStatement(int typeForwardOnly, int concurReadOnly) throws SQLException { + return this.connection.createStatement(typeForwardOnly, concurReadOnly); + } + + public DatabaseMetaData getMetaData() throws SQLException { + return this.connection.getMetaData(); + } + + public void use(String database, DbType dbType) { + if (this.hasStorageHandler()) { + this.getStorageHandler().use(database); + } else { + if (database == null || dbType == DbType.MS_ACCESS || dbType == DbType.BIGQUERY || dbType == DbType.AZURE) { + return; + } + + if (dbType == DbType.ORACLE) { + execute("ALTER SESSION SET current_schema = " + database); + } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) { + execute("SET search_path TO " + database); + } else if (dbType == DbType.TERADATA) { + execute("database " + database); + } else { + execute("USE " + database); + } + } + } + + public void execute(String sql) { + execute(sql, false); + } + + public void execute(String sql, boolean verbose) { + Statement statement = null; + try { + if (StringUtils.isEmpty(sql)) { + return; + } + + statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); + for (String subQuery : sql.split(";")) { + if (verbose) { + String abbrSQL = subQuery.replace('\n', ' ').replace('\t', ' ').trim(); + if (abbrSQL.length() > 100) + abbrSQL = abbrSQL.substring(0, 100).trim() + "..."; + logger.info("Adding query to batch: " + abbrSQL); + } + + statement.addBatch(subQuery); + } + long start = System.currentTimeMillis(); + if (verbose) { + logger.info("Executing batch"); + } + statement.executeBatch(); + if (verbose) { + outputQueryStats(statement, System.currentTimeMillis() - start); + } + } catch (SQLException e) { + logger.error(sql); + logger.error(e.getMessage(), e); + } finally { + if (statement != null) { + try { + statement.close(); + } catch (SQLException e) { + logger.error(e.getMessage()); + } + } + } + } + + void outputQueryStats(Statement statement, long ms) throws SQLException { + Throwable warning = statement.getWarnings(); + if (warning != null) + logger.info("- SERVER: " + warning.getMessage()); + String timeString; + if (ms < 1000) + timeString = ms + " ms"; + else if (ms < 60000) + timeString = decimalFormat.format(ms / 1000d) + " seconds"; + else if (ms < 3600000) + timeString = decimalFormat.format(ms / 60000d) + " minutes"; + else + timeString = decimalFormat.format(ms / 3600000d) + " hours"; + logger.info("- Query completed in " + timeString); + } + + public List getTableNames(String database) { + if (this.hasStorageHandler()) { + return this.getStorageHandler().getTableNames(); + } else { + return getTableNamesClassic(database); + } + } + + public List fetchTableStructure(RichConnection connection, String database, String table, ScanParameters scanParameters) { + List fieldInfos = new ArrayList<>(); + + if (dbType.supportsStorageHandler()) { + fieldInfos = dbType.getStorageHandler().fetchTableStructure(table, scanParameters); + } else if (dbType == DbType.MS_ACCESS) { + ResultSet rs = getFieldNamesFromJDBC(table); + try { + while (rs.next()) { + FieldInfo fieldInfo = new FieldInfo(scanParameters, rs.getString("COLUMN_NAME")); + fieldInfo.type = rs.getString("TYPE_NAME"); + fieldInfo.rowCount = connection.getTableSize(table); + fieldInfos.add(fieldInfo); + } + } catch (SQLException e) { + throw new RuntimeException(e.getMessage()); + } + } else { + String query = null; + if (dbType == DbType.ORACLE) + query = "SELECT COLUMN_NAME,DATA_TYPE FROM ALL_TAB_COLUMNS WHERE table_name = '" + table + "' AND owner = '" + database.toUpperCase() + "'"; + else if (dbType == DbType.SQL_SERVER || dbType == DbType.PDW) { + String trimmedDatabase = database; + if (database.startsWith("[") && database.endsWith("]")) + trimmedDatabase = database.substring(1, database.length() - 1); + String[] parts = table.split("\\."); + query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG='" + trimmedDatabase + "' AND TABLE_SCHEMA='" + parts[0] + + "' AND TABLE_NAME='" + parts[1] + "';"; + } else if (dbType == DbType.AZURE) { + String[] parts = table.split("\\."); + query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='" + parts[0] + + "' AND TABLE_NAME='" + parts[1] + "';"; + } else if (dbType == DbType.MYSQL) + query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '" + database + "' AND TABLE_NAME = '" + table + + "';"; + else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) + query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '" + database.toLowerCase() + "' AND TABLE_NAME = '" + + table.toLowerCase() + "' ORDER BY ordinal_position;"; + else if (dbType == DbType.TERADATA) { + query = "SELECT ColumnName, ColumnType FROM dbc.columns WHERE DatabaseName= '" + database.toLowerCase() + "' AND TableName = '" + + table.toLowerCase() + "';"; + } else if (dbType == DbType.BIGQUERY) { + query = "SELECT column_name AS COLUMN_NAME, data_type as DATA_TYPE FROM " + database + ".INFORMATION_SCHEMA.COLUMNS WHERE table_name = \"" + table + "\";"; + } + + if (StringUtils.isEmpty(query)) { + throw new RuntimeException("No query was specified to obtain the table structure for DbType = " + dbType.name()); + } + + for (org.ohdsi.utilities.files.Row row : connection.query(query)) { + row.upperCaseFieldNames(); + org.ohdsi.databases.FieldInfo fieldInfo; + if (dbType == DbType.TERADATA) { + fieldInfo = new org.ohdsi.databases.FieldInfo(scanParameters, row.get("COLUMNNAME")); + } else { + fieldInfo = new org.ohdsi.databases.FieldInfo(scanParameters, row.get("COLUMN_NAME")); + } + if (dbType == DbType.TERADATA) { + fieldInfo.type = row.get("COLUMNTYPE"); + } else { + fieldInfo.type = row.get("DATA_TYPE"); + } + fieldInfo.rowCount = connection.getTableSize(table); + fieldInfos.add(fieldInfo); + } + } + return fieldInfos; + } + + public ResultSet getFieldNamesFromJDBC(String table) { + if (dbType == DbType.MS_ACCESS) { + try { + DatabaseMetaData metadata = connection.getMetaData(); + return metadata.getColumns(null, null, table, null); + } catch (SQLException e) { + throw new RuntimeException(e.getMessage()); + } + } else { + throw new RuntimeException("DB is not of supported type"); + } + } + + public QueryResult fetchRowsFromTable(String table, long rowCount, ScanParameters scanParameters) { + String query = null; + int sampleSize = scanParameters.getSampleSize(); + + if (dbType.supportsStorageHandler()) { + query = dbType.getStorageHandler().getRowSampleQuery(table, rowCount, sampleSize); + } else if (sampleSize == -1) { + if (dbType == DbType.MS_ACCESS) + query = "SELECT * FROM [" + table + "]"; + else if (dbType == DbType.SQL_SERVER || dbType == DbType.PDW || dbType == DbType.AZURE) + query = "SELECT * FROM [" + table.replaceAll("\\.", "].[") + "]"; + else + query = "SELECT * FROM " + table; + } else { + if (dbType == DbType.SQL_SERVER || dbType == DbType.AZURE) + query = "SELECT * FROM [" + table.replaceAll("\\.", "].[") + "] TABLESAMPLE (" + sampleSize + " ROWS)"; + else if (dbType == DbType.MYSQL) + query = "SELECT * FROM " + table + " ORDER BY RAND() LIMIT " + sampleSize; + else if (dbType == DbType.PDW) + query = "SELECT TOP " + sampleSize + " * FROM [" + table.replaceAll("\\.", "].[") + "] ORDER BY RAND()"; + else if (dbType == DbType.ORACLE) { + if (sampleSize < rowCount) { + double percentage = 100 * sampleSize / (double) rowCount; + if (percentage < 100) + query = "SELECT * FROM " + table + " SAMPLE(" + percentage + ")"; + } else { + query = "SELECT * FROM " + table; + } + } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) { + query = "SELECT * FROM " + table + " ORDER BY RANDOM() LIMIT " + sampleSize; + } + else if (dbType == DbType.MS_ACCESS) { + query = "SELECT " + "TOP " + sampleSize + " * FROM [" + table + "]"; + } + else if (dbType == DbType.BIGQUERY) { + query = "SELECT * FROM " + table + " ORDER BY RAND() LIMIT " + sampleSize; + } + } + + + if (StringUtils.isEmpty(query)) { + throw new RuntimeException("No query was generated for database type " + dbType.name()); + } + + return createQueryResult(query); + } + + + private List getTableNamesClassic(String database) { + List names = new ArrayList<>(); + String query = null; + if (dbType == DbType.MYSQL) { + query = "SHOW TABLES IN " + database; + } else if (dbType == DbType.SQL_SERVER || dbType == DbType.PDW || dbType == DbType.AZURE) { + query = "SELECT CONCAT(schemas.name, '.', tables_views.name) FROM " + + "(SELECT schema_id, name FROM %1$s.sys.tables UNION ALL SELECT schema_id, name FROM %1$s.sys.views) tables_views " + + "INNER JOIN %1$s.sys.schemas ON tables_views.schema_id = schemas.schema_id " + + "ORDER BY schemas.name, tables_views.name"; + query = String.format(query, database); + logger.info(query); + } else if (dbType == DbType.ORACLE) { + query = "SELECT table_name FROM " + + "(SELECT table_name, owner FROM all_tables UNION ALL SELECT view_name, owner FROM all_views) tables_views " + + "WHERE owner='" + database.toUpperCase() + "'"; + } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) { + query = "SELECT table_name FROM information_schema.tables WHERE table_schema = '" + database.toLowerCase() + "' ORDER BY table_name"; + } else if (dbType == DbType.MS_ACCESS) { + query = "SELECT Name FROM sys.MSysObjects WHERE (Type=1 OR Type=5) AND Flags=0;"; + } else if (dbType == DbType.TERADATA) { + query = "SELECT TableName from dbc.tables WHERE tablekind IN ('T','V') and databasename='" + database + "'"; + } else if (dbType == DbType.BIGQUERY) { + query = "SELECT table_name from " + database + ".INFORMATION_SCHEMA.TABLES ORDER BY table_name;"; + } + + for (Row row : createQueryResult(query)) + names.add(row.get(row.getFieldNames().get(0))); + return names; + } + + private QueryResult createQueryResult(String sql) { + return new QueryResult(sql, this, verbose); + } + + public void close() throws SQLException { + if (this.hasStorageHandler()) { + this.getStorageHandler().close(); + } else { + this.connection.close(); + } + } + + public void setAutoCommit(boolean b) throws SQLException { + this.connection.setAutoCommit(b); + } + + public PreparedStatement prepareStatement(String statement) throws SQLException { + return this.connection.prepareStatement(statement); + } + + public void commit() throws SQLException { + this.connection.commit(); + } + + public void clearWarnings() throws SQLException { + this.connection.clearWarnings(); + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/DBConnector.java b/rabbit-core/src/main/java/org/ohdsi/databases/DBConnector.java index 33bc159a..4c3c8f63 100644 --- a/rabbit-core/src/main/java/org/ohdsi/databases/DBConnector.java +++ b/rabbit-core/src/main/java/org/ohdsi/databases/DBConnector.java @@ -18,37 +18,46 @@ package org.ohdsi.databases; import java.sql.Connection; +import java.sql.Driver; import java.sql.DriverManager; import java.sql.SQLException; +import java.util.Enumeration; import java.util.regex.Matcher; import java.util.regex.Pattern; import oracle.jdbc.pool.OracleDataSource; -import org.apache.tools.ant.types.selectors.SelectSelector; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; public class DBConnector { - public static void main(String[] args) { + public static DBConnection connect(DbSettings dbSettings, boolean verbose) { + assert dbSettings.dbType != null; + if (dbSettings.dbType.supportsStorageHandler()) { + return new DBConnection(dbSettings.dbType.getStorageHandler().getInstance(dbSettings), dbSettings.dbType, verbose); + } else { + return connect(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType, verbose); + } } // If dbType.BIGQUERY: domain field has been replaced with database field - public static Connection connect(String server, String domain, String user, String password, DbType dbType) { - if (dbType.equals(DbType.MYSQL)) - return DBConnector.connectToMySQL(server, user, password); - else if (dbType.equals(DbType.MSSQL) || dbType.equals(DbType.PDW) || dbType.equals(DbType.AZURE)) - return DBConnector.connectToMSSQL(server, domain, user, password); - else if (dbType.equals(DbType.ORACLE)) - return DBConnector.connectToOracle(server, domain, user, password); - else if (dbType.equals(DbType.POSTGRESQL)) - return DBConnector.connectToPostgreSQL(server, user, password); - else if (dbType.equals(DbType.MSACCESS)) - return DBConnector.connectToMsAccess(server, user, password); - else if (dbType.equals(DbType.REDSHIFT)) - return DBConnector.connectToRedshift(server, user, password); - else if (dbType.equals(DbType.TERADATA)) - return DBConnector.connectToTeradata(server, user, password); - else if (dbType.equals(DbType.BIGQUERY)) - return DBConnector.connectToBigQuery(server, domain, user, password); + private static DBConnection connect(String server, String domain, String user, String password, DbType dbType, boolean verbose) { + if (dbType.equalsDbType(DbType.MYSQL)) + return new DBConnection(DBConnector.connectToMySQL(server, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.SQL_SERVER) || dbType.equalsDbType(DbType.PDW) || dbType.equalsDbType(DbType.AZURE)) + return new DBConnection(DBConnector.connectToMSSQL(server, domain, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.ORACLE)) + return new DBConnection(DBConnector.connectToOracle(server, domain, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.POSTGRESQL)) + return new DBConnection(DBConnector.connectToPostgreSQL(server, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.MS_ACCESS)) + return new DBConnection(DBConnector.connectToMsAccess(server, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.REDSHIFT)) + return new DBConnection(DBConnector.connectToRedshift(server, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.TERADATA)) + return new DBConnection(DBConnector.connectToTeradata(server, user, password), dbType, verbose); + else if (dbType.equalsDbType(DbType.BIGQUERY)) + return new DBConnection(DBConnector.connectToBigQuery(server, domain, user, password), dbType, verbose); else return null; } @@ -110,10 +119,9 @@ public static Connection connectToPostgreSQL(String server, String user, String final String jdbcProtocol = "jdbc:postgresql://"; String url = (!server.startsWith(jdbcProtocol) ? jdbcProtocol : "") + server; try { - System.out.printf("DriverManager.getConnection(%s, %s, %s)%n", url, user, password); return DriverManager.getConnection(url, user, password); } catch (SQLException e1) { - throw new RuntimeException("Cannot connect to DB server: " + e1.getMessage() + " for url: " + url); + throw new RuntimeException("Cannot connect to DB server: " + e1.getMessage()); } } @@ -253,5 +261,45 @@ public static Connection connectToBigQuery(String server, String domain, String throw new RuntimeException("Simba URL failed: Cannot connect to DB server: " + e1.getMessage()); } } + + /* + * main() can be run to verify that all configured JDBC drivers are loadable + */ + public static void main(String[] args) { + verifyDrivers(); + } + + public static final String ALL_JDBC_DRIVERS_LOADABLE = "All configured JDBC drivers could be loaded."; + static void verifyDrivers() { + // verify that a JDBC driver that is not included/supported cannot be loaded + String notSupportedDriver = "org.sqlite.JDBC"; // change this if WhiteRabbit starts supporting SQLite + if (DbType.driverNames().contains(notSupportedDriver)) { + throw new RuntimeException("Cannot run this test for a supported driver."); + } + try { + testJDBCDriverAndVersion(notSupportedDriver); + throw new RuntimeException(String.format("JDBC driver was not expected to be loaded: %s", notSupportedDriver)); + } catch (ClassNotFoundException ignored) {} + + DbType.driverNames().forEach(driver -> { + try { + testJDBCDriverAndVersion(driver); + } catch (ClassNotFoundException e) { + throw new RuntimeException(String.format("JDBC driver class could not be loaded: %s", driver)); + } + }); + System.out.println(ALL_JDBC_DRIVERS_LOADABLE); + } + + static void testJDBCDriverAndVersion(String driverName) throws ClassNotFoundException { + Enumeration drivers = DriverManager.getDrivers(); + while (drivers.hasMoreElements()) { + Driver driver = drivers.nextElement(); + Class driverClass = Class.forName(driverName); + if (driver.getClass().isAssignableFrom(driverClass)) { + int ignoredMajorVersion = driver.getMajorVersion(); + } + } + } } diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/DBRowIterator.java b/rabbit-core/src/main/java/org/ohdsi/databases/DBRowIterator.java new file mode 100644 index 00000000..20a6687d --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/DBRowIterator.java @@ -0,0 +1,123 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.ohdsi.utilities.files.Row; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.HashSet; +import java.util.Iterator; +import java.util.Set; + +class DBRowIterator implements Iterator { + static Logger logger = LoggerFactory.getLogger(DBRowIterator.class); + + private ResultSet resultSet; + + private boolean hasNext; + + private Set columnNames = new HashSet<>(); + + public DBRowIterator(String sql, RichConnection richConnection) { + new DBRowIterator(sql, richConnection.getConnection(), richConnection.isVerbose()); + } + public DBRowIterator(String sql, DBConnection dbConnection, boolean verbose) { + Statement statement; + try { + sql.trim(); + if (sql.endsWith(";")) + sql = sql.substring(0, sql.length() - 1); + if (verbose) { + String abbrSQL = sql.replace('\n', ' ').replace('\t', ' ').trim(); + if (abbrSQL.length() > 100) + abbrSQL = abbrSQL.substring(0, 100).trim() + "..."; + logger.info("Executing query: {}", abbrSQL); + } + long start = System.currentTimeMillis(); + statement = dbConnection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); + resultSet = statement.executeQuery(sql); + hasNext = resultSet.next(); + if (verbose) + dbConnection.outputQueryStats(statement, System.currentTimeMillis() - start); + } catch (SQLException e) { + logger.error(sql, e.getMessage()); + throw new RuntimeException(e); + } + } + + public void close() { + if (resultSet != null) { + try { + resultSet.close(); + } catch (SQLException e) { + e.printStackTrace(); + } + resultSet = null; + hasNext = false; + } + } + + @Override + public boolean hasNext() { + return hasNext; + } + + @Override + public Row next() { + try { + Row row = new Row(); + ResultSetMetaData metaData; + metaData = resultSet.getMetaData(); + columnNames.clear(); + + for (int i = 1; i < metaData.getColumnCount() + 1; i++) { + String columnName = metaData.getColumnName(i); + if (columnNames.add(columnName)) { + String value; + try { + value = resultSet.getString(i); + } catch (Exception e) { + value = ""; + } + if (value == null) + value = ""; + + row.add(columnName, value.replace(" 00:00:00", "")); + } + } + hasNext = resultSet.next(); + if (!hasNext) { + resultSet.close(); + resultSet = null; + } + return row; + } catch (SQLException e) { + e.printStackTrace(); + throw new RuntimeException(e); + } + } + + @Override + public void remove() { + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/DataType.java b/rabbit-core/src/main/java/org/ohdsi/databases/DataType.java new file mode 100644 index 00000000..0b82546e --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/DataType.java @@ -0,0 +1,22 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +public enum DataType { + EMPTY, TEXT, DATE, INT, REAL, VARCHAR; +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/DbType.java b/rabbit-core/src/main/java/org/ohdsi/databases/DbType.java deleted file mode 100644 index 4942deb0..00000000 --- a/rabbit-core/src/main/java/org/ohdsi/databases/DbType.java +++ /dev/null @@ -1,52 +0,0 @@ -/******************************************************************************* - * Copyright 2019 Observational Health Data Sciences and Informatics - * - * This file is part of WhiteRabbit - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - ******************************************************************************/ -package org.ohdsi.databases; - -public class DbType { - public static DbType MYSQL = new DbType("mysql"); - public static DbType MSSQL = new DbType("mssql"); - public static DbType PDW = new DbType("pdw"); - public static DbType ORACLE = new DbType("oracle"); - public static DbType POSTGRESQL = new DbType("postgresql"); - public static DbType MSACCESS = new DbType("msaccess"); - public static DbType REDSHIFT = new DbType("redshift"); - public static DbType TERADATA = new DbType("teradata"); - public static DbType BIGQUERY = new DbType("bigquery"); - public static DbType AZURE = new DbType("azure"); - - private enum Type { - MYSQL, MSSQL, PDW, ORACLE, POSTGRESQL, MSACCESS, REDSHIFT, TERADATA, BIGQUERY, AZURE - }; - - private Type type; - - public DbType(String type) { - this.type = Type.valueOf(type.toUpperCase()); - } - - public boolean equals(Object other) { - if (other instanceof DbType && ((DbType) other).type == type) - return true; - else - return false; - } - - public String getTypeName() { - return this.type.name(); - } -} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/FieldInfo.java b/rabbit-core/src/main/java/org/ohdsi/databases/FieldInfo.java new file mode 100644 index 00000000..8d19fcb7 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/FieldInfo.java @@ -0,0 +1,255 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.ohdsi.utilities.DateUtilities; +import org.ohdsi.utilities.StringUtilities; +import org.ohdsi.utilities.collections.CountingSet; +import org.ohdsi.utilities.collections.Pair; + +import java.time.LocalDate; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class FieldInfo { + private final ScanParameters scanParameters; + public String type; + public String name; + public String label; + public CountingSet valueCounts = new CountingSet<>(); + public long sumLength = 0; + public int maxLength = 0; + public long nProcessed = 0; + public long emptyCount = 0; + public long uniqueCount = 0; + public long rowCount = -1; + public boolean isInteger = true; + public boolean isReal = true; + public boolean isDate = true; + public boolean isFreeText = false; + public boolean tooManyValues = false; + public UniformSamplingReservoir samplingReservoir; + public Object average; + public Object stdev; + public Object minimum; + public Object maximum; + public Object q1; + public Object q2; + public Object q3; + + public FieldInfo(ScanParameters scanParameters, String name) { + this.scanParameters = scanParameters; + this.name = name; + if (scanParameters.doCalculateNumericStats()) { + this.samplingReservoir = new UniformSamplingReservoir(scanParameters.getNumStatsSamplerSize()); + } + } + + public void trim() { + // Only keep values that are used in scan report + if (valueCounts.size() > scanParameters.getMaxValues()) { + valueCounts.keepTopN(scanParameters.getMaxValues()); + } + + // Calculate numeric stats and dereference sampling reservoir to save memory. + if (scanParameters.doCalculateNumericStats()) { + average = getAverage(); + stdev = getStandardDeviation(); + minimum = getMinimum(); + maximum = getMaximum(); + q1 = getQ1(); + q2 = getQ2(); + q3 = getQ3(); + } + samplingReservoir = null; + } + + public boolean hasValuesTrimmed() { + return tooManyValues; + } + + public Double getFractionEmpty() { + if (nProcessed == 0) + return 1d; + else + return emptyCount / (double) nProcessed; + } + + public String getTypeDescription() { + if (type != null) + return type; + else if (!scanParameters.doScanValues()) // If not type assigned and not values scanned, do not derive + return ""; + else if (nProcessed == emptyCount) + return DataType.EMPTY.name(); + else if (isFreeText) + return DataType.TEXT.name(); + else if (isDate) + return DataType.DATE.name(); + else if (isInteger) + return DataType.INT.name(); + else if (isReal) + return DataType.REAL.name(); + else + return DataType.VARCHAR.name(); + } + + public Double getFractionUnique() { + if (nProcessed == 0 || uniqueCount == 1) { + return 0d; + } else { + return uniqueCount / (double) nProcessed; + } + + } + + public void processValue(String value) { + nProcessed++; + sumLength += value.length(); + if (value.length() > maxLength) + maxLength = value.length(); + + String trimValue = value.trim(); + if (trimValue.length() == 0) + emptyCount++; + + if (!isFreeText) { + boolean newlyAdded = valueCounts.add(value); + if (newlyAdded) uniqueCount++; + + if (trimValue.length() != 0) { + evaluateDataType(trimValue); + } + + if (nProcessed == ScanParameters.N_FOR_FREE_TEXT_CHECK && !isInteger && !isReal && !isDate) { + doFreeTextCheck(); + } + } else { + valueCounts.addAll(StringUtilities.mapToWords(trimValue.toLowerCase())); + } + + // if over this large constant number, then trimmed back to size used in report (maxValues). + if (!tooManyValues && valueCounts.size() > ScanParameters.MAX_VALUES_IN_MEMORY) { + tooManyValues = true; + this.trim(); + } + + if (scanParameters.doCalculateNumericStats() && !trimValue.isEmpty()) { + if (isInteger || isReal) { + samplingReservoir.add(Double.parseDouble(trimValue)); + } else if (isDate) { + samplingReservoir.add(DateUtilities.parseDate(trimValue)); + } + } + + } + + public List> getSortedValuesWithoutSmallValues() { + List> result = valueCounts.key2count.entrySet().stream() + .filter(e -> e.getValue().count >= scanParameters.getMinCellCount()) + .sorted(Comparator.>comparingInt(e -> e.getValue().count).reversed()) + .limit(scanParameters.getMaxValues()) + .map(e -> new Pair<>(e.getKey(), e.getValue().count)) + .collect(Collectors.toCollection(ArrayList::new)); + + if (result.size() < valueCounts.key2count.size()) { + result.add(new Pair<>("List truncated...", -1)); + } + return result; + } + + private void evaluateDataType(String value) { + if (isReal && !StringUtilities.isNumber(value)) + isReal = false; + if (isInteger && !StringUtilities.isLong(value)) + isInteger = false; + if (isDate && !StringUtilities.isDate(value)) + isDate = false; + } + + private void doFreeTextCheck() { + double averageLength = sumLength / (double) (nProcessed - emptyCount); + if (averageLength >= ScanParameters.MIN_AVERAGE_LENGTH_FOR_FREE_TEXT) { + isFreeText = true; + // Reset value count to word count + CountingSet wordCounts = new CountingSet<>(); + for (Map.Entry entry : valueCounts.key2count.entrySet()) + for (String word : StringUtilities.mapToWords(entry.getKey().toLowerCase())) + wordCounts.add(word, entry.getValue().count); + valueCounts = wordCounts; + } + } + + private Object formatNumericValue(double value) { + return formatNumericValue(value, false); + } + + private Object formatNumericValue(double value, boolean dateAsDays) { + if (nProcessed == 0) { + return Double.NaN; + } else if (getTypeDescription().equals(DataType.EMPTY.name())) { + return Double.NaN; + } else if (isInteger || isReal) { + return value; + } else if (isDate && dateAsDays) { + return value; + } else if (isDate) { + return LocalDate.ofEpochDay((long) value).toString(); + } else { + return Double.NaN; + } + } + + private Object getMinimum() { + double min = samplingReservoir.getPopulationMinimum(); + return formatNumericValue(min); + } + + private Object getMaximum() { + double max = samplingReservoir.getPopulationMaximum(); + return formatNumericValue(max); + } + + private Object getAverage() { + double average = samplingReservoir.getPopulationMean(); + return formatNumericValue(average); + } + + private Object getStandardDeviation() { + double stddev = samplingReservoir.getSampleStandardDeviation(); + return formatNumericValue(stddev, true); + } + + private Object getQ1() { + double q1 = samplingReservoir.getSampleQuartiles().get(0); + return formatNumericValue(q1); + } + + private Object getQ2() { + double q2 = samplingReservoir.getSampleQuartiles().get(1); + return formatNumericValue(q2); + } + + private Object getQ3() { + double q3 = samplingReservoir.getSampleQuartiles().get(2); + return formatNumericValue(q3); + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/QueryResult.java b/rabbit-core/src/main/java/org/ohdsi/databases/QueryResult.java new file mode 100644 index 00000000..e44de6f5 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/QueryResult.java @@ -0,0 +1,53 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.ohdsi.utilities.files.Row; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +public class QueryResult implements Iterable { + private String sql; + + private List iterators = new ArrayList<>(); + private DBConnection dbConnection; + + public QueryResult(String sql, DBConnection dbConnection) { + this(sql, dbConnection, false); + } + + public QueryResult(String sql, DBConnection dbConnection, boolean verbose) { + this.sql = sql; + this.dbConnection = dbConnection; + } + + @Override + public Iterator iterator() { + DBRowIterator iterator = new DBRowIterator(sql, dbConnection, false); + iterators.add(iterator); + return iterator; + } + + public void close() { + for (DBRowIterator iterator : iterators) { + iterator.close(); + } + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/RichConnection.java b/rabbit-core/src/main/java/org/ohdsi/databases/RichConnection.java index 17f691e7..77b563d1 100644 --- a/rabbit-core/src/main/java/org/ohdsi/databases/RichConnection.java +++ b/rabbit-core/src/main/java/org/ohdsi/databases/RichConnection.java @@ -19,168 +19,76 @@ import java.io.Closeable; import java.sql.BatchUpdateException; -import java.sql.Connection; import java.sql.DatabaseMetaData; import java.sql.PreparedStatement; import java.sql.ResultSet; -import java.sql.ResultSetMetaData; import java.sql.SQLException; -import java.sql.Statement; import java.sql.Types; -import java.text.DecimalFormat; import java.util.ArrayList; import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Set; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; import org.ohdsi.utilities.SimpleCounter; import org.ohdsi.utilities.StringUtilities; import org.ohdsi.utilities.files.Row; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; public class RichConnection implements Closeable { - public static int INSERT_BATCH_SIZE = 100000; - private Connection connection; - private boolean verbose = false; - private static DecimalFormat decimalFormat = new DecimalFormat("#.#"); - private DbType dbType; + Logger logger = LoggerFactory.getLogger(RichConnection.class); - public RichConnection(String server, String domain, String user, String password, DbType dbType) { - this.connection = DBConnector.connect(server, domain, user, password, dbType); - this.dbType = dbType; + public static int INSERT_BATCH_SIZE = 100000; + private DBConnection connection; + private boolean verbose = false; + private DbType dbType; + + public RichConnection(DbSettings dbSettings) { + this.connection = DBConnector.connect(dbSettings, verbose); + this.dbType = dbSettings.dbType; } /** * Execute the given SQL statement. - * + * * @param sql */ public void execute(String sql) { - Statement statement = null; - try { - if (sql.length() == 0) - return; - - statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); - for (String subQuery : sql.split(";")) { - if (verbose) { - String abbrSQL = subQuery.replace('\n', ' ').replace('\t', ' ').trim(); - if (abbrSQL.length() > 100) - abbrSQL = abbrSQL.substring(0, 100).trim() + "..."; - System.out.println("Adding query to batch: " + abbrSQL); - } - - statement.addBatch(subQuery); - } - long start = System.currentTimeMillis(); - if (verbose) - System.out.println("Executing batch"); - statement.executeBatch(); - if (verbose) - outputQueryStats(statement, System.currentTimeMillis() - start); - } catch (SQLException e) { - System.err.println(sql); - e.printStackTrace(); - } finally { - if (statement != null) { - try { - statement.close(); - } catch (SQLException e) { - // TODO Auto-generated catch block - System.err.println(e.getMessage()); - } - } - } - } - - private void outputQueryStats(Statement statement, long ms) throws SQLException { - Throwable warning = statement.getWarnings(); - if (warning != null) - System.out.println("- SERVER: " + warning.getMessage()); - String timeString; - if (ms < 1000) - timeString = ms + " ms"; - else if (ms < 60000) - timeString = decimalFormat.format(ms / 1000d) + " seconds"; - else if (ms < 3600000) - timeString = decimalFormat.format(ms / 60000d) + " minutes"; - else - timeString = decimalFormat.format(ms / 3600000d) + " hours"; - System.out.println("- Query completed in " + timeString); + connection.execute(sql, verbose); } /** * Query the database using the provided SQL statement. - * + * * @param sql * @return */ public QueryResult query(String sql) { - return new QueryResult(sql); + return new QueryResult(sql, connection, verbose); } /** * Switch the database to use. - * + * * @param database */ public void use(String database) { - if (database == null || dbType == DbType.MSACCESS || dbType == DbType.BIGQUERY || dbType == DbType.AZURE) { - return; - } - - if (dbType == DbType.ORACLE) { - execute("ALTER SESSION SET current_schema = " + database); - } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) { - execute("SET search_path TO " + database); - } else if (dbType == DbType.TERADATA) { - execute("database " + database); - } else { - execute("USE " + database); - } + connection.use(database, dbType); } public List getTableNames(String database) { - List names = new ArrayList<>(); - String query = null; - if (dbType == DbType.MYSQL) { - query = "SHOW TABLES IN " + database; - } else if (dbType == DbType.MSSQL || dbType == DbType.PDW || dbType == DbType.AZURE) { - query = "SELECT CONCAT(schemas.name, '.', tables_views.name) FROM " + - "(SELECT schema_id, name FROM %1$s.sys.tables UNION ALL SELECT schema_id, name FROM %1$s.sys.views) tables_views " + - "INNER JOIN %1$s.sys.schemas ON tables_views.schema_id = schemas.schema_id " + - "ORDER BY schemas.name, tables_views.name"; - query = String.format(query, database); - System.out.println(query); - } else if (dbType == DbType.ORACLE) { - query = "SELECT table_name FROM " + - "(SELECT table_name, owner FROM all_tables UNION ALL SELECT view_name, owner FROM all_views) tables_views " + - "WHERE owner='" + database.toUpperCase() + "'"; - } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) { - query = "SELECT table_name FROM information_schema.tables WHERE table_schema = '" + database.toLowerCase() + "' ORDER BY table_name"; - } else if (dbType == DbType.MSACCESS) { - query = "SELECT Name FROM sys.MSysObjects WHERE (Type=1 OR Type=5) AND Flags=0;"; - } else if (dbType == DbType.TERADATA) { - query = "SELECT TableName from dbc.tables WHERE tablekind IN ('T','V') and databasename='" + database + "'"; - } else if (dbType == DbType.BIGQUERY) { - query = "SELECT table_name from " + database + ".INFORMATION_SCHEMA.TABLES ORDER BY table_name;"; - } + return connection.getTableNames(database); + } - for (Row row : query(query)) - names.add(row.get(row.getFieldNames().get(0))); - return names; + public List fetchTableStructure(RichConnection connection, String database, String table, ScanParameters scanParameters) { + return this.connection.fetchTableStructure(this, database, table, scanParameters); } - public ResultSet getMsAccessFieldNames(String table) { - if (dbType == DbType.MSACCESS) { - try { - DatabaseMetaData metadata = connection.getMetaData(); - return metadata.getColumns(null, null, table, null); - } catch (SQLException e) { - throw new RuntimeException(e.getMessage()); - } - } else - throw new RuntimeException("DB is not of type MS Access"); + public QueryResult fetchRowsFromTable(String table, long rowCount, ScanParameters scanParameters) { + return this.connection.fetchRowsFromTable(table, rowCount, scanParameters); } /** @@ -192,9 +100,9 @@ public ResultSet getMsAccessFieldNames(String table) { public long getTableSize(String tableName) { QueryResult qr; long returnVal; - if (dbType == DbType.MSSQL || dbType == DbType.PDW || dbType == DbType.AZURE) + if (dbType == DbType.SQL_SERVER || dbType == DbType.PDW || dbType == DbType.AZURE) qr = query("SELECT COUNT_BIG(*) FROM [" + tableName.replaceAll("\\.", "].[") + "];"); - else if (dbType == DbType.MSACCESS) + else if (dbType == DbType.MS_ACCESS) qr = query("SELECT COUNT(*) FROM [" + tableName + "];"); else qr = query("SELECT COUNT(*) FROM " + tableName + ";"); @@ -217,37 +125,19 @@ public void close() { try { connection.close(); } catch (SQLException e) { - e.printStackTrace(); + logger.error(e.getMessage(), e); } } public void setVerbose(boolean verbose) { - this.verbose = verbose; + this.connection.setVerbose(verbose); } - public class QueryResult implements Iterable { - private String sql; - - private List iterators = new ArrayList<>(); - - public QueryResult(String sql) { - this.sql = sql; - } - - @Override - public Iterator iterator() { - DBRowIterator iterator = new DBRowIterator(sql); - iterators.add(iterator); - return iterator; - } - - public void close() { - for (DBRowIterator iterator : iterators) { - iterator.close(); - } - } + public DBConnection getConnection() { + return connection; } + /** * Inserts the rows into a table in the database. * @@ -263,8 +153,9 @@ public void insertIntoTable(Iterator iterator, String table, boolean create SimpleCounter counter = new SimpleCounter(1000000, true); while (iterator.hasNext()) { if (batch.size() == INSERT_BATCH_SIZE) { - if (first && create) + if (first && create) { createTable(table, batch); + } insert(table, batch); batch.clear(); first = false; @@ -272,24 +163,29 @@ public void insertIntoTable(Iterator iterator, String table, boolean create batch.add(iterator.next()); counter.count(); } - if (batch.size() != 0) { - if (first && create) + if (!batch.isEmpty()) { + if (first && create) { createTable(table, batch); + } insert(table, batch); } } + boolean isVerbose() { + return connection.isVerbose(); + } + private void insert(String tableName, List rows) { List columns; columns = rows.get(0).getFieldNames(); - for (int i = 0; i < columns.size(); i++) - columns.set(i, columnNameToSqlName(columns.get(i))); + columns.replaceAll(this::columnNameToSqlName); StringBuilder sql = new StringBuilder("INSERT INTO " + tableName); sql.append(" (").append(StringUtilities.join(columns, ",")).append(")"); sql.append(" VALUES (?"); - for (int i = 1; i < columns.size(); i++) + for (int i = 1; i < columns.size(); i++) { sql.append(",?"); + } sql.append(")"); try { connection.setAutoCommit(false); @@ -297,16 +193,16 @@ private void insert(String tableName, List rows) { for (Row row : rows) { for (int i = 0; i < columns.size(); i++) { String value = row.get(columns.get(i)); - if (value == null) - System.out.println(row.toString()); - else if (value.length() == 0) + if (value == null) { + logger.info(row.toString()); + } else if (value.isEmpty()) { value = null; - // System.out.println(value); - if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) // PostgreSQL does not allow unspecified types + } + if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) {// PostgreSQL does not allow unspecified types statement.setObject(i + 1, value, Types.OTHER); + } else if (dbType == DbType.ORACLE) { if (isDate(value)) { - // System.out.println(value); statement.setDate(i + 1, java.sql.Date.valueOf(value)); } else @@ -322,9 +218,9 @@ else if (dbType == DbType.ORACLE) { connection.setAutoCommit(true); connection.clearWarnings(); } catch (SQLException e) { - e.printStackTrace(); + logger.error(e.getMessage(), e); if (e instanceof BatchUpdateException) { - System.err.println(e.getNextException().getMessage()); + logger.error(e.getNextException().getMessage()); } } } @@ -351,25 +247,25 @@ private static boolean isDate(String string) { private Set createTable(String tableName, List rows) { Set numericFields = new HashSet<>(); Row firstRow = rows.get(0); - List fields = new ArrayList<>(rows.size()); + List fields = new ArrayList<>(rows.size()); for (String field : firstRow.getFieldNames()) - fields.add(new FieldInfo(field)); + fields.add(new NumericFieldInfo(field)); for (Row row : rows) { - for (FieldInfo fieldInfo : fields) { - String value = row.get(fieldInfo.name); - if (fieldInfo.isNumeric && !StringUtilities.isInteger(value)) - fieldInfo.isNumeric = false; - if (value.length() > fieldInfo.maxLength) - fieldInfo.maxLength = value.length(); + for (NumericFieldInfo numericFieldInfo : fields) { + String value = row.get(numericFieldInfo.name); + if (numericFieldInfo.isNumeric && !StringUtilities.isInteger(value)) + numericFieldInfo.isNumeric = false; + if (value.length() > numericFieldInfo.maxLength) + numericFieldInfo.maxLength = value.length(); } } StringBuilder sql = new StringBuilder(); sql.append("CREATE TABLE ").append(tableName).append(" (\n"); - for (FieldInfo fieldInfo : fields) { - sql.append(" ").append(fieldInfo.toString()).append(",\n"); - if (fieldInfo.isNumeric) - numericFields.add(fieldInfo.name); + for (NumericFieldInfo numericFieldInfo : fields) { + sql.append(" ").append(numericFieldInfo.toString()).append(",\n"); + if (numericFieldInfo.isNumeric) + numericFields.add(numericFieldInfo.name); } sql.append(");"); execute(sql.toString()); @@ -380,12 +276,12 @@ private String columnNameToSqlName(String name) { return name.replaceAll(" ", "_").replace("-", "_").replace(",", "_").replaceAll("_+", "_"); } - private class FieldInfo { + private class NumericFieldInfo { public String name; public boolean isNumeric = true; public int maxLength = 0; - public FieldInfo(String name) { + public NumericFieldInfo(String name) { this.name = name; } @@ -397,7 +293,7 @@ else if (maxLength > 255) return columnNameToSqlName(name) + " text"; else return columnNameToSqlName(name) + " varchar(255)"; - } else if (dbType == DbType.MSSQL || dbType == DbType.PDW || dbType == DbType.AZURE) { + } else if (dbType == DbType.SQL_SERVER || dbType == DbType.PDW || dbType == DbType.AZURE) { if (isNumeric) { if (maxLength < 10) return columnNameToSqlName(name) + " int"; @@ -411,94 +307,4 @@ else if (maxLength > 255) throw new RuntimeException("Create table syntax not specified for type " + dbType); } } - - private class DBRowIterator implements Iterator { - - private ResultSet resultSet; - - private boolean hasNext; - - private Set columnNames = new HashSet<>(); - - public DBRowIterator(String sql) { - Statement statement; - try { - sql.trim(); - if (sql.endsWith(";")) - sql = sql.substring(0, sql.length() - 1); - if (verbose) { - String abbrSQL = sql.replace('\n', ' ').replace('\t', ' ').trim(); - if (abbrSQL.length() > 100) - abbrSQL = abbrSQL.substring(0, 100).trim() + "..."; - System.out.println("Executing query: " + abbrSQL); - } - long start = System.currentTimeMillis(); - statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); - resultSet = statement.executeQuery(sql); - hasNext = resultSet.next(); - if (verbose) - outputQueryStats(statement, System.currentTimeMillis() - start); - } catch (SQLException e) { - System.err.println(sql); - System.err.println(e.getMessage()); - throw new RuntimeException(e); - } - } - - public void close() { - if (resultSet != null) { - try { - resultSet.close(); - } catch (SQLException e) { - e.printStackTrace(); - } - resultSet = null; - hasNext = false; - } - } - - @Override - public boolean hasNext() { - return hasNext; - } - - @Override - public Row next() { - try { - Row row = new Row(); - ResultSetMetaData metaData; - metaData = resultSet.getMetaData(); - columnNames.clear(); - - for (int i = 1; i < metaData.getColumnCount() + 1; i++) { - String columnName = metaData.getColumnName(i); - if (columnNames.add(columnName)) { - String value; - try { - value = resultSet.getString(i); - } catch (Exception e) { - value = ""; - } - if (value == null) - value = ""; - - row.add(columnName, value.replace(" 00:00:00", "")); - } - } - hasNext = resultSet.next(); - if (!hasNext) { - resultSet.close(); - resultSet = null; - } - return row; - } catch (SQLException e) { - e.printStackTrace(); - throw new RuntimeException(e); - } - } - - @Override - public void remove() { - } - } } diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/ScanParameters.java b/rabbit-core/src/main/java/org/ohdsi/databases/ScanParameters.java new file mode 100644 index 00000000..8cbe5669 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/ScanParameters.java @@ -0,0 +1,38 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +public interface ScanParameters { + + public boolean doCalculateNumericStats(); + + public int getNumStatsSamplerSize(); + + public int getMaxValues(); + + public boolean doScanValues(); + + public int getMinCellCount(); + + public int getSampleSize(); + + public static int MAX_VALUES_IN_MEMORY = 100000; + public static int MIN_CELL_COUNT_FOR_CSV = 1000000; + public static int N_FOR_FREE_TEXT_CHECK = 1000; + public static int MIN_AVERAGE_LENGTH_FOR_FREE_TEXT = 100; +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/SnowflakeHandler.java b/rabbit-core/src/main/java/org/ohdsi/databases/SnowflakeHandler.java new file mode 100644 index 00000000..1c32670b --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/SnowflakeHandler.java @@ -0,0 +1,289 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.apache.commons.lang.StringUtils; + +import java.sql.*; +import java.util.Arrays; +import java.util.List; + +import org.ohdsi.databases.configuration.*; +import org.ohdsi.utilities.collections.Pair; +import org.ohdsi.utilities.files.IniFile; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.ohdsi.databases.SnowflakeHandler.SnowflakeConfiguration.*; + +/* + * SnowflakeHandler implements all Snowflake specific logic required to connect to, and query, a Snowflake instance. + * + * It is implemented as a Singleton, using the enum pattern es described here: https://www.baeldung.com/java-singleton + */ +public enum SnowflakeHandler implements StorageHandler { + INSTANCE(); + + final static Logger logger = LoggerFactory.getLogger(SnowflakeHandler.class); + + DBConfiguration configuration = new SnowflakeConfiguration(); + private DBConnection snowflakeConnection = null; + + private final DbType dbType = DbType.SNOWFLAKE; + public static final String ERROR_NO_FIELD_OF_TYPE = "No value was specified for type"; + public static final String ERROR_INCORRECT_SCHEMA_SPECIFICATION = + "Database should be specified as 'warehouse.database.schema', " + + "e.g. 'computewh.snowflake_sample_data.weather"; + public static final String ERROR_CONNECTION_NOT_INITIALIZED = + "Snowflake Database connection has not been initialized."; + + SnowflakeHandler() { + } + + public void resetConnection() throws SQLException { + if (this.snowflakeConnection != null) { + this.snowflakeConnection.close(); + } + this.snowflakeConnection = null; + } + + @Override + public StorageHandler getInstance(DbSettings dbSettings) { + if (snowflakeConnection == null) { + snowflakeConnection = connectToSnowflake(dbSettings); + } + + return INSTANCE; + } + + public static Pair getConfiguration(IniFile iniFile, ValidationFeedback feedback) { + SnowflakeConfiguration configuration = new SnowflakeConfiguration(); + ValidationFeedback currentFeedback = configuration.loadAndValidateConfiguration(iniFile); + if (feedback != null) { + feedback.add(currentFeedback); + } + + String warehouse = configuration.getValue(SNOWFLAKE_WAREHOUSE); + DbSettings dbSettings = new DbSettings(); + dbSettings.dbType = DbType.SNOWFLAKE; + dbSettings.server = String.format("https://%s.snowflakecomputing.com", configuration.getValue(SNOWFLAKE_ACCOUNT)); + dbSettings.database = String.format("%s.%s.%s", + warehouse, + configuration.getValue(SNOWFLAKE_DATABASE), + configuration.getValue(SNOWFLAKE_SCHEMA)); + dbSettings.domain = dbSettings.database; + dbSettings.user = configuration.getValue(SNOWFLAKE_USER); + dbSettings.password = configuration.getValue(SNOWFLAKE_PASSWORD); + dbSettings.sourceType = DbSettings.SourceType.DATABASE; + + return new Pair<>(configuration, dbSettings); + } + + public DBConnection getDBConnection() { + this.checkInitialised(); + return this.snowflakeConnection; + } + + @Override + public String getTableSizeQuery(String tableName) { + return String.format("SELECT COUNT(*) FROM %s.%s.%s;", this.getDatabase(), this.getSchema(), tableName); + } + + public String getRowSampleQuery(String table, long rowCount, long sampleSize) { + return String.format("SELECT * FROM %s ORDER BY RANDOM() LIMIT %s", table, sampleSize); + } + + public String getTablesQuery(String database) { + return String.format("SELECT TABLE_NAME FROM %s.INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = '%s'", this.getDatabase().toUpperCase(), this.getSchema().toUpperCase()); + } + + @Override + public void checkInitialised() throws DBConfigurationException { + if (this.snowflakeConnection == null) { + throw new DBConfigurationException("Snowflake DB/connection was not initialized"); + } + } + + public DbType getDbType() { + return this.dbType; + } + + private static DBConnection connectToSnowflake(DbSettings dbSettings) { + try { + Class.forName("net.snowflake.client.jdbc.SnowflakeDriver"); + } catch (ClassNotFoundException ex) { + throw new RuntimeException("Cannot find JDBC driver. Make sure the file snowflake-jdbc-x.xx.xx.jar is in the path: " + ex.getMessage()); + } + String url = buildUrl(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, INSTANCE.configuration.getValue(SNOWFLAKE_AUTHENTICATOR)); + try { + return new DBConnection(DriverManager.getConnection(url), DbType.SNOWFLAKE, false); + } catch (SQLException ex) { + throw new RuntimeException("Cannot connect to Snowflake server: " + ex.getMessage()); + } + } + + public ResultSet getFieldNames(String table) { + try { + DatabaseMetaData metadata = this.snowflakeConnection.getMetaData(); + return metadata.getColumns(null, null, table, null); + } catch (SQLException e) { + throw new RuntimeException(e.getMessage()); + } + } + + public DBConfiguration getDBConfiguration() { + + return this.configuration; + } + public static class SnowflakeConfiguration extends DBConfiguration { + public static final String SNOWFLAKE_ACCOUNT = "SNOWFLAKE_ACCOUNT"; + public static final String TOOLTIP_SNOWFLAKE_ACCOUNT = "Account for the Snowflake instance"; + public static final String SNOWFLAKE_USER = "SNOWFLAKE_USER"; + public static final String SNOWFLAKE_PASSWORD = "SNOWFLAKE_PASSWORD"; + public static final String SNOWFLAKE_AUTHENTICATOR = "SNOWFLAKE_AUTHENTICATOR"; + public static final String SNOWFLAKE_WAREHOUSE = "SNOWFLAKE_WAREHOUSE"; + public static final String SNOWFLAKE_DATABASE = "SNOWFLAKE_DATABASE"; + public static final String SNOWFLAKE_SCHEMA = "SNOWFLAKE_SCHEMA"; + public static final String ERROR_MUST_SET_PASSWORD_OR_AUTHENTICATOR = "Either password or authenticator must be specified for Snowflake"; + public static final String ERROR_MUST_NOT_SET_PASSWORD_AND_AUTHENTICATOR = "Specify only one of password or authenticator Snowflake"; + public static final String ERROR_VALUE_CAN_ONLY_BE_ONE_OF = "Error can only be one of "; + public SnowflakeConfiguration() { + super( + ConfigurationField.create( + SNOWFLAKE_ACCOUNT, + "Account", + TOOLTIP_SNOWFLAKE_ACCOUNT) + .required(), + ConfigurationField.create( + SNOWFLAKE_USER, + "User", + "User for the Snowflake instance") + .required(), + ConfigurationField.create( + SNOWFLAKE_PASSWORD, + "Password", + "Password for the Snowflake instance"), + ConfigurationField.create( + SNOWFLAKE_WAREHOUSE, + "Warehouse", + "Warehouse for the Snowflake instance") + .required(), + ConfigurationField.create( + SNOWFLAKE_DATABASE, + "Database", + "Database for the Snowflake instance") + .required(), + ConfigurationField.create( + SNOWFLAKE_SCHEMA, + "Schema", + "Schema for the Snowflake instance") + .required(), + ConfigurationField.create( + SNOWFLAKE_AUTHENTICATOR, + "Authenticator method", + "Snowflake JDBC authenticator method (only 'externalbrowser' is currently supported)") + .addValidator(new FieldValidator() { + private final List allowedValues = Arrays.asList("externalbrowser"); + @Override + public ValidationFeedback validate(ConfigurationField field) { + ValidationFeedback feedback = new ValidationFeedback(); + if (StringUtils.isNotEmpty(field.getValue())) { + if (!allowedValues.contains(field.getValue().toLowerCase())) { + feedback.addError(String.format("%s (%s)", ERROR_VALUE_CAN_ONLY_BE_ONE_OF, + String.join(", ", allowedValues)), field); + } else { + field.setValue(field.getValue().toLowerCase()); + } + } + return feedback; + } + }) + ); + this.configurationFields.addValidator(new PasswordXORAuthenticatorValidator()); + } + + static class PasswordXORAuthenticatorValidator implements ConfigurationValidator { + + @Override + public ValidationFeedback validate(ConfigurationFields fields) { + ValidationFeedback feedback = new ValidationFeedback(); + String password = fields.getValue(SNOWFLAKE_PASSWORD); + String authenticator = fields.getValue(SNOWFLAKE_AUTHENTICATOR); + if (StringUtils.isEmpty(password) && StringUtils.isEmpty(authenticator)) { + feedback.addError(ERROR_MUST_SET_PASSWORD_OR_AUTHENTICATOR, fields.get(SNOWFLAKE_PASSWORD)); + feedback.addError(ERROR_MUST_SET_PASSWORD_OR_AUTHENTICATOR, fields.get(SNOWFLAKE_AUTHENTICATOR)); + } else if (!StringUtils.isEmpty(password) && !StringUtils.isEmpty(authenticator)) { + feedback.addError(ERROR_MUST_NOT_SET_PASSWORD_AND_AUTHENTICATOR, fields.get(SNOWFLAKE_PASSWORD)); + feedback.addError(ERROR_MUST_NOT_SET_PASSWORD_AND_AUTHENTICATOR, fields.get(SNOWFLAKE_AUTHENTICATOR)); + } + + return feedback; + } + } + @Override + public DbSettings toDbSettings(ValidationFeedback feedback) { + return getConfiguration(this.toIniFile(),feedback ).getItem2(); + } + + } + + private static String buildUrl(String server, String schema, String user, String password, String authenticator) { + final String jdbcPrefix = "jdbc:snowflake://"; + String url = (!server.startsWith(jdbcPrefix) ? jdbcPrefix : "") + server; + if (!url.contains("?")) { + url += "?"; + } + + String[] parts = splitDatabaseName(schema); + url = appendParameterIfSet(url, "warehouse", parts[0]); + url = appendParameterIfSet(url, "db", parts[1]); + url = appendParameterIfSet(url, "schema", parts[2]); + url = appendParameterIfSet(url, "user", user); + if (!StringUtils.isEmpty(authenticator)) { + url = appendParameterIfSet(url, "authenticator", authenticator); + } else { + url = appendParameterIfSet(url, "password", password); + } + + return url; + } + private static String appendParameterIfSet(String url, String name, String value) { + if (!StringUtils.isEmpty(value)) { + return String.format("%s%s%s=%s", url, (url.endsWith("?") ? "" : "&"), name, value); + } + else { + throw new RuntimeException(String.format(ERROR_NO_FIELD_OF_TYPE + " %s", name)); + } + } + private static String[] splitDatabaseName(String databaseName) { + String[] parts = databaseName.split("\\."); + if (parts.length != 3) { + throw new RuntimeException(ERROR_INCORRECT_SCHEMA_SPECIFICATION); + } + + return parts; + } + + public String getDatabase() { + return this.configuration.getValue(SNOWFLAKE_DATABASE); + } + + private String getSchema() { + return this.configuration.getValue(SNOWFLAKE_SCHEMA); + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/StorageHandler.java b/rabbit-core/src/main/java/org/ohdsi/databases/StorageHandler.java new file mode 100644 index 00000000..f241cd5d --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/StorageHandler.java @@ -0,0 +1,233 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.ohdsi.databases.configuration.*; +import org.ohdsi.utilities.files.IniFile; +import org.ohdsi.utilities.files.Row; + +import java.io.PrintStream; +import java.sql.DatabaseMetaData; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +/** + * StorageHandler defines the interface that a database connection class must implement. + * + */ +public interface StorageHandler { + + /** + * Creates an instance of the implementing class, or can return the singleton for. + * + * @param dbSettings Configuration parameters for the implemented database + * @return instance of a StorageHandler implementing class + */ + StorageHandler getInstance(DbSettings dbSettings); + + /** + * Returns the DBConnection object associated with the database connection + * + * @return DBConnection object + */ + DBConnection getDBConnection(); + + /** + * @return the DbType enum constant associated with the implementation + */ + DbType getDbType(); + + /** + * + * @param tableName name of the table to get the size (number of rows) for + * @return Implementation specific query to get the size of the table + */ + String getTableSizeQuery(String tableName); + + /** + * Verifies if the implementing object was properly configured for use. Should throw a DBConfigurationException + * if this is not the case. + * + * @throws DBConfigurationException Object not ready for use + */ + void checkInitialised() throws DBConfigurationException; + + /** + * Returns the row count of the specified table. + * + * @param tableName name of table + * @return size of table in rows + */ + default long getTableSize(String tableName ) { + long returnVal; + QueryResult qr = new QueryResult(getTableSizeQuery(tableName), getDBConnection()); + try { + returnVal = Long.parseLong(qr.iterator().next().getCells().get(0)); + } catch (Exception e) { + throw new RuntimeException(e); + } finally { + qr.close(); + } + return returnVal; + } + + /** + * Executes an SQL use statement (or similar) if the underlying database requires it. + * + * No-op by default. + * + * @param ignoredDatabase provided for compatibility + */ + default void use(String ignoredDatabase) {} + + /** + * closes the connection to the database. No-op by default. + */ + default void close() { + // no-op by default, so singletons don't need to implement it + } + + /** + * Returns the name of the database the connection was initiated for. + * + * @return name of (current) database + */ + String getDatabase(); + + /** + * + * @return List of table names in current database + */ + default List getTableNames() { + List names = new ArrayList<>(); + String query = this.getTablesQuery(getDatabase()); + + for (Row row : new QueryResult(query, new DBConnection(this, getDbType(), false))) { + names.add(row.getCells().get(0)); + } + + return names; + } + + /** + * Fetches the structure of a table as a list of FieldInfo objects. + * + * The default implementation should work for some/most/all JDBC databases and only needs to be overridden + * for databases where this is not the case. + * + * @param table name of the table to fetch the structure for + * @param scanParameters parameters that are to be used for scanning the table + * @return + */ + default List fetchTableStructure(String table, ScanParameters scanParameters) { + List fieldInfos = new ArrayList<>(); + ResultSet rs = getFieldNamesFromJDBC(table); + try { + while (rs.next()) { + FieldInfo fieldInfo = new FieldInfo(scanParameters, rs.getString("COLUMN_NAME")); + fieldInfo.type = rs.getString("TYPE_NAME"); + fieldInfo.rowCount = getTableSize(table); + fieldInfos.add(fieldInfo); + } + } catch ( + SQLException e) { + throw new RuntimeException(e.getMessage()); + } + return fieldInfos; + } + + /** + * Retrieves column names (fields) for a table. + * + * The default implementation uses the JDBC metadata. Should only be overridden if this approach does not work + * for the underlying database. + * + * @param table name of the table to get the column names for + * @return java.sql.ResultSet + */ + default ResultSet getFieldNamesFromJDBC(String table) { + try { + DatabaseMetaData metadata = getDBConnection().getMetaData(); + return metadata.getColumns(null, null, table, null); + } catch (SQLException e) { + throw new RuntimeException(e.getMessage()); + } + } + + /** + * Returns the database specific query to obtain the table names in the database. + * See getTableNames(), which calls this method + * + * @param database + * @return + */ + String getTablesQuery(String database); + + /** + * Returns the database specific query that should be used to obtain a sample of rows from a table. + * + * @param table table to get sample from + * @param rowCount known rowcount for the table + * @param sampleSize size of the sample + * @return Database specific SQL query + */ + String getRowSampleQuery(String table, long rowCount, long sampleSize); + + /** + * @return the DbSettings object used to initialize the database connection + */ + default DbSettings getDbSettings(ValidationFeedback feedback) { + return getDBConfiguration().toDbSettings(feedback); + } + + /** + * Returns a validated DbSettings object with values based on the IniFile object + * + * @param iniFile IniFile object containing database configuration values for the class + * that implements the StorageHandler + * + * @return DbSettings object + */ + default DbSettings getDbSettings(IniFile iniFile, ValidationFeedback feedback, PrintStream outStream) { + ValidationFeedback validationFeedback = getDBConfiguration().loadAndValidateConfiguration(iniFile); + if (feedback != null) { + feedback.add(validationFeedback); + } + if (outStream != null) { + if (validationFeedback.hasErrors()) { + outStream.println("There are errors for the configuration file:"); + validationFeedback.getErrors().forEach((error, fields) -> + outStream.printf("\t%s (%s)%n", error, fields.stream().map(f -> f.name).collect(Collectors.joining(",")))); + } + if (validationFeedback.hasWarnings()) { + outStream.println("There are errors for the configuration file:"); + validationFeedback.getWarnings().forEach((warning, fields) -> + outStream.printf("\t%s (%s)%n", warning, fields.stream().map(f -> f.name).collect(Collectors.joining(",")))); + } + } + return getDBConfiguration().toDbSettings(feedback); + } + + /** + * Returns the DBConfiguration object for the implementing class + */ + DBConfiguration getDBConfiguration(); +} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/UniformSamplingReservoir.java b/rabbit-core/src/main/java/org/ohdsi/databases/UniformSamplingReservoir.java similarity index 79% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/UniformSamplingReservoir.java rename to rabbit-core/src/main/java/org/ohdsi/databases/UniformSamplingReservoir.java index d7b8a068..8680d989 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/UniformSamplingReservoir.java +++ b/rabbit-core/src/main/java/org/ohdsi/databases/UniformSamplingReservoir.java @@ -1,4 +1,24 @@ -package org.ohdsi.whiteRabbit.scan; +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.math.BigDecimal; import java.math.RoundingMode; @@ -18,13 +38,15 @@ * calculated exactly. */ public class UniformSamplingReservoir { + static Logger logger = LoggerFactory.getLogger(UniformSamplingReservoir.class); + private double[] samples; private int maxSize; private long populationCount; private BigDecimal populationSum; private double populationMinimum = Double.POSITIVE_INFINITY; private double populationMaximum = Double.NEGATIVE_INFINITY; - private transient int currentSampleLength; + private int currentSampleLength; /** * Create an empty reservoir. @@ -211,14 +233,16 @@ public static void main(String[] args) { us.add(i); } - System.out.println(us.getSamples().toString()); - System.out.println(us.getCount()); - System.out.println(us.getSampleQuartiles().toString()); - System.out.println(us.populationSum.doubleValue()); - System.out.println(us.getPopulationMean()); - System.out.println(us.getPopulationMinimum()); - System.out.println(us.getPopulationMaximum()); - System.out.println(us.getSampleMean()); - System.out.println(us.getSampleStandardDeviation()); + if (logger.isInfoEnabled()) { + logger.info(us.getSamples().toString()); + logger.info(String.valueOf(us.getCount())); + logger.info(us.getSampleQuartiles().toString()); + logger.info(String.valueOf(us.populationSum.doubleValue())); + logger.info(String.valueOf(us.getPopulationMean())); + logger.info(String.valueOf(us.getPopulationMinimum())); + logger.info(String.valueOf(us.getPopulationMaximum())); + logger.info(String.valueOf(us.getSampleMean())); + logger.info(String.valueOf(us.getSampleStandardDeviation())); + } } } \ No newline at end of file diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationField.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationField.java new file mode 100644 index 00000000..63cb2254 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationField.java @@ -0,0 +1,137 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import org.apache.commons.lang.StringUtils; + +import java.util.ArrayList; +import java.util.List; +import java.util.regex.Pattern; + +public class ConfigurationField { + public final String name; + public final String label; + public final String toolTip; + private String value; + private String defaultValue; + + public static final String VALUE_REQUIRED_FORMAT_STRING = "A non-empty value is required for field %s (name %s)"; + public static final String INTEGER_VALUE_REQUIRED_FORMAT_STRING = "An integer value is allowed for field %s (name %s)"; + public static final String ONLY_YESNO_ALLOWED_FORMAT_STRING = "Only the values 'yes' or 'no' are allowed for field %s (name %s)"; + + List validators = new ArrayList<>(); + + private static final FieldValidator fieldRequiredValidator = new FieldRequiredValidator(); + private static final FieldValidator integerValueValidator = new IntegerValueValidator(); + private static final FieldValidator onlyYesNoAllowed = new YesNoValidator(); + + private ConfigurationField(String name, String label, String toolTip) { + this.name = name; + this.label = label; + this.toolTip = toolTip; + this.defaultValue = null; + this.value = null; + } + + public static ConfigurationField create(String name, String label, String toolTip) { + return new ConfigurationField(name, label, toolTip); + } + + public ConfigurationField required() { + this.addValidator(fieldRequiredValidator); + return this; + } + public ConfigurationField integerValue() { + this.addValidator(integerValueValidator); + return this; + } + public ConfigurationField yesNoValue() { + this.addValidator(onlyYesNoAllowed); + return this; + } + + public ConfigurationField defaultValue(String value) { + this.defaultValue = value; + return this; + } + + public ConfigurationField addValidator(FieldValidator validator) { + this.validators.add(validator); + return this; + } + + public ConfigurationField setValue(String value) { + this.value = value; + return this; + } + + public String getValue() { + return this.value; + } + + public String getDefaultValue() { + return this.defaultValue; + } + + public String getValueOrDefault() { + if (this.value != null) { + return this.value; + } else if (this.defaultValue != null){ + return this.defaultValue; + } + return null; + } + + private static class FieldRequiredValidator implements FieldValidator { + public ValidationFeedback validate(ConfigurationField field) { + ValidationFeedback feedback = new ValidationFeedback(); + if (StringUtils.isEmpty(field.getValue())) { + feedback.addError(String.format(VALUE_REQUIRED_FORMAT_STRING, field.label, field.name), field); + } + + return feedback; + } + } + + private static class IntegerValueValidator implements FieldValidator { + static Pattern integerPattern = Pattern.compile("^\\d*$"); + public ValidationFeedback validate(ConfigurationField field) { + ValidationFeedback feedback = new ValidationFeedback(); + if (StringUtils.isNotEmpty(field.getValue()) && (!integerPattern.matcher(field.getValue()).matches())) { + feedback.addError(String.format(INTEGER_VALUE_REQUIRED_FORMAT_STRING, field.label, field.name), field); + } + + return feedback; + } + } + private static class YesNoValidator implements FieldValidator { + static Pattern yesNoPattern = Pattern.compile("^(yes|no)$", Pattern.CASE_INSENSITIVE); + public ValidationFeedback validate(ConfigurationField field) { + ValidationFeedback feedback = new ValidationFeedback(); + if (StringUtils.isNotEmpty(field.getValue())) { + if (!yesNoPattern.matcher(field.getValue()).matches()) { + feedback.addError(String.format(ONLY_YESNO_ALLOWED_FORMAT_STRING, field.label, field.name), field); + } else { + field.setValue(field.getValue().toLowerCase()); + } + } + + return feedback; + } + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationFields.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationFields.java new file mode 100644 index 00000000..d5658a3e --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationFields.java @@ -0,0 +1,63 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; + +public class ConfigurationFields { + List fields; + List validators = new ArrayList<>(); + + public ConfigurationFields(ConfigurationField... fields) { + this.fields = new ArrayList<>(Arrays.asList(fields)); + } + + public void addValidator(ConfigurationValidator validator) { + this.validators.add(validator); + } + + public List getFields() { + return this.fields; + } + + public ConfigurationField get(String fieldName) { + Optional field = fields.stream().filter(f -> fieldName.equalsIgnoreCase(f.name)).findFirst(); + if (field.isPresent()) { + return field.get(); + } + + throw new DBConfigurationException(String.format("No ConfigurationField object found for field name '%s'", fieldName)); + } + + public String getValue(String fieldName) { + Optional value = this.fields.stream().filter(f -> fieldName.equalsIgnoreCase(f.name)).map(ConfigurationField::getValue).findFirst(); + return (value.orElse("")); + } + + public ValidationFeedback validate() { + ValidationFeedback allFeedback = new ValidationFeedback(); + for (ConfigurationValidator validator : this.validators) { + allFeedback.add(validator.validate(this)); + } + + return allFeedback; + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationValidator.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationValidator.java new file mode 100644 index 00000000..ec37f3d9 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ConfigurationValidator.java @@ -0,0 +1,23 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +@FunctionalInterface +public interface ConfigurationValidator { + ValidationFeedback validate(ConfigurationFields fields); +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfiguration.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfiguration.java new file mode 100644 index 00000000..2336a0d6 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfiguration.java @@ -0,0 +1,168 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import org.apache.commons.lang.StringUtils; +import org.ohdsi.utilities.files.IniFile; + +import java.io.PrintStream; +import java.util.*; + +public class DBConfiguration { + public static final String DATA_TYPE_FIELD = "DATA_TYPE"; + public static final String DELIMITER_FIELD = "DELIMITER"; + public static final String TABLES_TO_SCAN_FIELD = "TABLES_TO_SCAN"; + public static final String SCAN_FIELD_VALUES_FIELD = "SCAN_FIELD_VALUES"; + public static final String MIN_CELL_COUNT_FIELD = "MIN_CELL_COUNT"; + public static final String MAX_DISTINCT_VALUES_FIELD = "MAX_DISTINCT_VALUES"; + public static final String ROWS_PER_TABLE_FIELD = "ROWS_PER_TABLE"; + public static final String CALCULATE_NUMERIC_STATS_FIELD = "CALCULATE_NUMERIC_STATS"; + public static final String NUMERIC_STATS_SAMPLER_SIZE_FIELD = "NUMERIC_STATS_SAMPLER_SIZE"; + public static final String ERROR_DUPLICATE_DEFINITIONS_FOR_FIELD = "Multiple definitions for field "; + protected ConfigurationFields configurationFields; + + private DBConfiguration() { + } + + + public DBConfiguration(ConfigurationField... fields) { + this.checkForDuplicates(fields); + this.configurationFields = new ConfigurationFields(fields); + } + + public static ConfigurationField[] createScanConfigurationFields() { + return new ConfigurationField[]{ + ConfigurationField.create(DELIMITER_FIELD, + "", + "") + .defaultValue(",") + .required(), + ConfigurationField.create(TABLES_TO_SCAN_FIELD, + "", + "") + .defaultValue("*") + .required(), + ConfigurationField.create(SCAN_FIELD_VALUES_FIELD, + "", + "") + .defaultValue("yes") + .required(), + ConfigurationField.create(MIN_CELL_COUNT_FIELD, + "", + "") + .defaultValue("5") + .integerValue() + .required(), + ConfigurationField.create(MAX_DISTINCT_VALUES_FIELD, + "", + "") + .defaultValue("1000") + .integerValue() + .required(), + ConfigurationField.create(ROWS_PER_TABLE_FIELD, + "", + "") + .defaultValue("100000") + .integerValue() + .required(), + ConfigurationField.create(CALCULATE_NUMERIC_STATS_FIELD, + "", + "") + .defaultValue("no") + .yesNoValue() + .required(), + ConfigurationField.create(NUMERIC_STATS_SAMPLER_SIZE_FIELD, + "", + "") + .defaultValue("500") + .integerValue() + .required() + }; + } + + public IniFile toIniFile() { + IniFile iniFile = new IniFile(); + this.configurationFields.getFields().forEach(f -> { + iniFile.set(f.name, f.getValue()); + }); + + return iniFile; + } + + public DbSettings toDbSettings(ValidationFeedback feedback) { + throw new DBConfigurationException("Should be implemented by inheriting classes"); + } + + private void checkForDuplicates(ConfigurationField... fields) { + Set names = new HashSet<>(); + for (ConfigurationField field : fields) { + if (names.contains(field.name)) { + throw new DBConfigurationException(ERROR_DUPLICATE_DEFINITIONS_FOR_FIELD + field.name); + } + names.add(field.name); + } + } + + public ValidationFeedback loadAndValidateConfiguration(IniFile iniFile) throws DBConfigurationException { + for (ConfigurationField field : this.getFields()) { + field.setValue(iniFile.get(field.name)); + } + + return this.validateAll(); + } + + public ValidationFeedback validateAll() { + ValidationFeedback configurationFeedback = new ValidationFeedback(); + for (ConfigurationField field : this.getFields()) { + for (FieldValidator validator : field.validators) { + ValidationFeedback feedback = validator.validate(field); + configurationFeedback.add(feedback); + } + } + + configurationFeedback.add(configurationFields.validate()); + + return configurationFeedback; + } + + public List getFields() { + return configurationFields.getFields(); + } + + public ConfigurationField getField(String fieldName) { + return this.getFields().stream().filter(f -> f.name.equalsIgnoreCase(fieldName)).findFirst().orElse(null); + } + + public String getValue(String fieldName) { + Optional field = getFields().stream().filter(f -> fieldName.equalsIgnoreCase(f.name)).findFirst(); + if (field.isPresent()) { + return field.get().getValue(); + } else { + return ""; + } + } + + public void printIniFileTemplate(PrintStream stream) { + for (ConfigurationField field : this.configurationFields.getFields()) { + stream.printf("%s: %s\t%s%n", + field.name, + StringUtils.isEmpty(field.getDefaultValue()) ? "_" : field.getDefaultValue(), + field.toolTip); + } + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfigurationException.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfigurationException.java new file mode 100644 index 00000000..a216f700 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DBConfigurationException.java @@ -0,0 +1,24 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +public class DBConfigurationException extends RuntimeException { + public DBConfigurationException(String s) { + super(s); + } +} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/DbSettings.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbSettings.java similarity index 76% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/DbSettings.java rename to rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbSettings.java index 72637902..459898c6 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/DbSettings.java +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbSettings.java @@ -15,13 +15,13 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit; +package org.ohdsi.databases.configuration; import java.util.ArrayList; import java.util.List; import org.apache.commons.csv.CSVFormat; -import org.ohdsi.databases.DbType; +import org.ohdsi.databases.configuration.DbType; public class DbSettings { public enum SourceType { @@ -36,10 +36,17 @@ public enum SourceType { public String user; public String password; public String database; + public String warehouse; + public String schema; public String server; public String domain; // CSV file settings public char delimiter = ','; public CSVFormat csvFormat = CSVFormat.RFC4180; + + public String toString() { + return String.format("sourceType: %s; dbType: %s; user: %s; password: xxxx; database:%s; tables: %s", + sourceType, (dbType == null) ? "null" : dbType.name(), user, database, tables); + } } diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbType.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbType.java new file mode 100644 index 00000000..75e09f1e --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/DbType.java @@ -0,0 +1,119 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import org.apache.commons.lang.StringUtils; +import org.ohdsi.databases.StorageHandler; +import org.ohdsi.databases.SnowflakeHandler; + +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public enum DbType { + /* + * Please note: the names and strings and the Type enum below must match when String.toUpperCase().replace(" ", "_") + * is applied (see constructor and the normalizedName() method). This is enforced when the enum values are constructed, + * and a violation of this rule will result in a DBConfigurationException being thrown. + */ + DELIMITED_TEXT_FILES("Delimited text files", null), + MYSQL("MySQL", "com.mysql.cj.jdbc.Driver"), + ORACLE("Oracle", "oracle.jdbc.driver.OracleDriver"), + SQL_SERVER("SQL Server", "com.microsoft.sqlserver.jdbc.SQLServerDriver"), + POSTGRESQL("PostgreSQL", "org.postgresql.Driver"), + MS_ACCESS("MS Access", "net.ucanaccess.jdbc.UcanaccessDriver"), + PDW("PDW", "com.microsoft.sqlserver.jdbc.SQLServerDriver"), + REDSHIFT("Redshift", "com.amazon.redshift.jdbc42.Driver"), + TERADATA("Teradata", "com.teradata.jdbc.TeraDriver"), + BIGQUERY("BigQuery", "com.simba.googlebigquery.jdbc42.Driver"), + AZURE("Azure", "com.microsoft.sqlserver.jdbc.SQLServerDriver"), + SNOWFLAKE("Snowflake", "net.snowflake.client.jdbc.SnowflakeDriver", SnowflakeHandler.INSTANCE), + SAS7BDAT("Sas7bdat", null); + + private final String label; + private final String driverName; + private final StorageHandler implementingClass; + + DbType(String type, String driverName) { + this(type, driverName, null); + } + + DbType(String label, String driverName, StorageHandler implementingClass) { + this.label = label; + this.driverName = driverName; + this.implementingClass = implementingClass; + if (!this.name().equals(normalizedName(label))) { + throw new DBConfigurationException(String.format( + "%s: the normalized value of label '%s' (%s) must match the name of the enum constant (%s)", + DbType.class.getName(), + label, + normalizedName(label), + this.name() + )); + } + } + + public boolean equalsDbType(DbType other) { + return (other != null && other.equals(this)); + } + + public boolean supportsStorageHandler() { + return this.implementingClass != null; + } + + public StorageHandler getStorageHandler() throws DBConfigurationException { + if (this.supportsStorageHandler()) { + return this.implementingClass; + } else { + throw new DBConfigurationException(String.format("Class %s does not implement interface %s", + this.implementingClass.getClass().getName(), + StorageHandler.class.getName())); + } + } + + public static DbType getDbType(String name) { + return Enum.valueOf(DbType.class, normalizedName(name)); + } + + /** + * Returns the list of supported database in the order that they should appear in the GUI. + * + * @return Array of labels for the supported database, intended for use in a selector (like a Swing JComboBox) + */ + public static String[] pickList() { + return Stream.of(DELIMITED_TEXT_FILES, SAS7BDAT, MYSQL, ORACLE, SQL_SERVER, POSTGRESQL, MS_ACCESS, PDW, REDSHIFT, TERADATA, BIGQUERY, AZURE, SNOWFLAKE) + .map(DbType::label).toArray(String[]::new); + } + + public static List driverNames() { + // return a list of unique names, without null values + return Stream.of(values()).filter(v -> StringUtils.isNotEmpty(v.driverName)).map(d -> d.driverName).distinct().collect(Collectors.toList()); + } + + public String label() { + return this.label; + } + + public String driverName() { + return this.driverName; + } + + private static String normalizedName(String name) { + return name.toUpperCase().replace(" ", "_"); + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/FieldValidator.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/FieldValidator.java new file mode 100644 index 00000000..aa1ab734 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/FieldValidator.java @@ -0,0 +1,23 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +@FunctionalInterface +public interface FieldValidator { + ValidationFeedback validate(ConfigurationField field); +} diff --git a/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ValidationFeedback.java b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ValidationFeedback.java new file mode 100644 index 00000000..531d7910 --- /dev/null +++ b/rabbit-core/src/main/java/org/ohdsi/databases/configuration/ValidationFeedback.java @@ -0,0 +1,77 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import one.util.streamex.EntryStream; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class ValidationFeedback { + private Map> warnings = new HashMap<>(); + private Map> errors = new HashMap<>(); + + public boolean isFullyValid() { + return warnings.isEmpty() && errors.isEmpty(); + } + + public boolean hasWarnings() { + return !warnings.isEmpty(); + } + + public boolean hasErrors() { + return !errors.isEmpty(); + } + + public Map> getWarnings() { + return this.warnings; + } + + public Map> getErrors() { + return this.errors; + } + + public void addWarning(String warning, ConfigurationField field) { + if (this.warnings.containsKey(warning)) { + this.warnings.get(warning).add(field); + } else { + this.warnings.put(warning, Collections.singletonList(field)); + } + } + + public void addError(String error, ConfigurationField field) { + if (this.errors.containsKey(error)) { + this.errors.get(error).add(field); + } else { + this.errors.put(error, Stream.of(field).collect(Collectors.toList())); + } + } + + public void add(ValidationFeedback feedback) { + this.warnings = EntryStream.of(this.warnings) + .append(EntryStream.of(feedback.getWarnings())) + .toMap((e1, e2) -> e1); + this.errors = EntryStream.of(this.errors) + .append(EntryStream.of(feedback.getErrors())) + .toMap((e1, e2) -> e1); + } +} diff --git a/rabbit-core/src/main/java/org/ohdsi/ooxml/ReadXlsxFileWithHeader.java b/rabbit-core/src/main/java/org/ohdsi/ooxml/ReadXlsxFileWithHeader.java index ed26bdfa..867f5267 100644 --- a/rabbit-core/src/main/java/org/ohdsi/ooxml/ReadXlsxFileWithHeader.java +++ b/rabbit-core/src/main/java/org/ohdsi/ooxml/ReadXlsxFileWithHeader.java @@ -83,7 +83,7 @@ public Row next() { List cells = new ArrayList(fieldName2ColumnIndex.size()); for (Cell cell : iterator.next()) { String text; - if (cell.getCellTypeEnum() == CellType.NUMERIC) + if (cell.getCellType() == CellType.NUMERIC) text = myFormatter.format(cell.getNumericCellValue()); else text = cell.toString(); diff --git a/rabbit-core/src/main/java/org/ohdsi/rabbitInAHat/dataModel/TableCellLongTextRenderer.java b/rabbit-core/src/main/java/org/ohdsi/rabbitInAHat/dataModel/TableCellLongTextRenderer.java index 9e49e383..4d5a40dd 100644 --- a/rabbit-core/src/main/java/org/ohdsi/rabbitInAHat/dataModel/TableCellLongTextRenderer.java +++ b/rabbit-core/src/main/java/org/ohdsi/rabbitInAHat/dataModel/TableCellLongTextRenderer.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat.dataModel; import java.awt.Component; diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/ScanFieldName.java b/rabbit-core/src/main/java/org/ohdsi/utilities/ScanFieldName.java index 0110e696..721cb9e1 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/ScanFieldName.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/ScanFieldName.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.utilities; public interface ScanFieldName { diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/ScanSheetName.java b/rabbit-core/src/main/java/org/ohdsi/utilities/ScanSheetName.java index 56e65c7a..216c2db4 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/ScanSheetName.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/ScanSheetName.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.utilities; public interface ScanSheetName { diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/SimpleCounter.java b/rabbit-core/src/main/java/org/ohdsi/utilities/SimpleCounter.java index 8a066408..fa7939ce 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/SimpleCounter.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/SimpleCounter.java @@ -17,7 +17,13 @@ ******************************************************************************/ package org.ohdsi.utilities; +import org.ohdsi.databases.UniformSamplingReservoir; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + public class SimpleCounter { + static Logger logger = LoggerFactory.getLogger(SimpleCounter.class); + private int reportN; private long count; private long lastTime; @@ -43,14 +49,17 @@ public void count() { } private void report() { - if (reportRate){ - long interval = System.currentTimeMillis() - lastTime; - long processed = count - lastCount; - System.out.println(count + " (time per unit = " + interval/(double)processed + "ms)"); - lastTime = System.currentTimeMillis(); - lastCount = count; - } else - System.out.println(count); + if (logger.isInfoEnabled()) { + if (reportRate) { + long interval = System.currentTimeMillis() - lastTime; + long processed = count - lastCount; + logger.info("{} (time per unit = {} ms", count, interval / (double) processed); + lastTime = System.currentTimeMillis(); + lastCount = count; + } else { + logger.info(String.valueOf(count)); + } + } } public void finish() { diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/StringUtilities.java b/rabbit-core/src/main/java/org/ohdsi/utilities/StringUtilities.java index 70aeb954..e5b9a969 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/StringUtilities.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/StringUtilities.java @@ -17,6 +17,10 @@ ******************************************************************************/ package org.ohdsi.utilities; +import org.ohdsi.utilities.collections.CountingSet; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.text.DateFormat; @@ -32,7 +36,8 @@ import java.util.zip.DataFormatException; public class StringUtilities { - + static Logger logger = LoggerFactory.getLogger(StringUtilities.class); + public static long SECOND = 1000; public static long MINUTE = 60 * SECOND; public static long HOUR = 60 * MINUTE; @@ -328,7 +333,8 @@ public static String now() { DateFormat df = DateFormat.getDateTimeInstance(DateFormat.MEDIUM, DateFormat.MEDIUM); return df.format(d); } - + + @SuppressWarnings("java:S106") // System.out is intended here public static void outputWithTime(String message) { System.out.println(now() + "\t" + message); } @@ -767,7 +773,6 @@ public static String getMD5Digest(String str) { md5.update(buffer); result = md5.digest(); - // System.out.println(result); // create hex string from the 16-byte hash buf = new StringBuffer(result.length * 2); for (int i = 0; i < result.length; i++) { @@ -779,9 +784,7 @@ public static String getMD5Digest(String str) { } return buf.toString(); } catch (NoSuchAlgorithmException e) { - System.err.println("Exception caught: " + e); - e.printStackTrace(); - + logger.error(e.getMessage(), e); } return null; } @@ -802,7 +805,6 @@ public static String getSHA256Digest(String str) { sha256.update(buffer); result = sha256.digest(); - // System.out.println(result); // create hex string from the 16-byte hash buf = new StringBuffer(result.length * 2); for (int i = 0; i < result.length; i++) { diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/Version.java b/rabbit-core/src/main/java/org/ohdsi/utilities/Version.java index 79f4be67..5ee076f0 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/Version.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/Version.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.utilities; import java.io.IOException; diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/collections/CountingSet.java b/rabbit-core/src/main/java/org/ohdsi/utilities/collections/CountingSet.java index a70241c5..12d19144 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/collections/CountingSet.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/collections/CountingSet.java @@ -17,6 +17,10 @@ ******************************************************************************/ package org.ohdsi.utilities.collections; +import org.ohdsi.utilities.files.QuickAndDirtyXlsxReader; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.AbstractSet; import java.util.Comparator; import java.util.HashMap; @@ -32,7 +36,8 @@ * @param */ public class CountingSet extends AbstractSet { - + static Logger logger = LoggerFactory.getLogger(CountingSet.class); + public Map key2count; public CountingSet() { @@ -188,7 +193,7 @@ public int compareTo(Count o) { public void printCounts() { decliningCountStream() - .forEach(entry -> System.out.println(entry.getKey() + "\t" + entry.getValue().count)); + .forEach(entry -> logger.info("{}\t{}", entry.getKey(), entry.getValue().count)); } private Stream> decliningCountStream() { diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/files/IniFile.java b/rabbit-core/src/main/java/org/ohdsi/utilities/files/IniFile.java index e2605d76..9bf348fd 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/files/IniFile.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/files/IniFile.java @@ -17,12 +17,18 @@ ******************************************************************************/ package org.ohdsi.utilities.files; +import org.apache.commons.lang.StringUtils; +import org.ohdsi.databases.configuration.DBConfiguration; + import java.util.HashMap; import java.util.Map; public class IniFile { private Map settings = new HashMap(); + public IniFile() { + + } public IniFile(String filename){ for (String line : new ReadTextFile(filename)){ int indexOfHash = line.lastIndexOf('#'); @@ -43,4 +49,20 @@ public String get(String fieldName){ else return value; } + + public void set(String fieldName, String value) { + settings.put(fieldName.trim().toLowerCase(), value); + } + + public String getOrFail(String fieldName){ + String value = this.get(fieldName); + if (StringUtils.isEmpty(value)) { + throw new RuntimeException("Ini file should contain a value for '" + fieldName + "'"); + } + return value; + } + + public String getDataType() { + return getOrFail(DBConfiguration.DATA_TYPE_FIELD); + } } diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/files/QuickAndDirtyXlsxReader.java b/rabbit-core/src/main/java/org/ohdsi/utilities/files/QuickAndDirtyXlsxReader.java index f62f74eb..dbb43037 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/files/QuickAndDirtyXlsxReader.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/files/QuickAndDirtyXlsxReader.java @@ -32,12 +32,16 @@ import java.util.zip.ZipEntry; import java.util.zip.ZipInputStream; +import org.ohdsi.utilities.SimpleCounter; import org.ohdsi.utilities.StringUtilities; import org.ohdsi.utilities.collections.IntegerComparator; import org.ohdsi.utilities.files.QuickAndDirtyXlsxReader.Sheet; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; public class QuickAndDirtyXlsxReader extends ArrayList { + static Logger logger = LoggerFactory.getLogger(QuickAndDirtyXlsxReader.class); private static final long serialVersionUID = 25124428448185386L; private static final Pattern DOUBLE_IGNORE_PATTERN = Pattern.compile("[<>= ]+"); @@ -142,7 +146,6 @@ private void processSharedStrings(ZipInputStream inputStream) throws IOException private void processSheet(String filename, ZipInputStream inputStream) throws IOException { Sheet sheet = filenameToSheet.get(filename); - //System.out.println(filename + "\t" + sheet.name); BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8)); String line; StringBuilder fullSheet = new StringBuilder(); @@ -152,8 +155,9 @@ private void processSheet(String filename, ZipInputStream inputStream) throws IO for (String rowLine : StringUtilities.multiFindBetween(fullSheet.toString(), "")) { Row row = new Row(sheet); row.addAll(findCellValues(rowLine)); - if (row.size() != 0) + if (!row.isEmpty()) { sheet.add(row); + } } } diff --git a/rabbit-core/src/main/java/org/ohdsi/utilities/files/WriteCSVFileWithHeader.java b/rabbit-core/src/main/java/org/ohdsi/utilities/files/WriteCSVFileWithHeader.java index f6329154..0c2604f9 100644 --- a/rabbit-core/src/main/java/org/ohdsi/utilities/files/WriteCSVFileWithHeader.java +++ b/rabbit-core/src/main/java/org/ohdsi/utilities/files/WriteCSVFileWithHeader.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.utilities.files; import java.io.FileWriter; diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/DBConfigurationTest.java b/rabbit-core/src/test/java/org/ohdsi/databases/DBConfigurationTest.java new file mode 100644 index 00000000..ff7019a1 --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/DBConfigurationTest.java @@ -0,0 +1,59 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.ohdsi.databases.configuration.ConfigurationField; +import org.ohdsi.databases.configuration.DBConfiguration; +import org.ohdsi.databases.configuration.DBConfigurationException; + +import static org.junit.jupiter.api.Assertions.*; +import static org.ohdsi.databases.configuration.DBConfiguration.ERROR_DUPLICATE_DEFINITIONS_FOR_FIELD; + +class DBConfigurationTest { + + private final String NAME_FIELD1 = "FIELD_1"; + private final String LABEL_FIELD1 = "Field one"; + private final String TOOLTIP_FIELD1 = "Tooltip for field one"; + private final String NAME_FIELD2 = "FIELD_2"; + private final String LABEL_FIELD2 = "Field two"; + private final String TOOLTIP_FIELD2 = "Tooltip for field two"; + + @BeforeEach + void setUp() { + } + + @Test + void doNotAcceptDuplicateDefinitionsForField() { + Exception exception = assertThrows(DBConfigurationException.class, () -> { + DBConfiguration testConfiguration = new DBConfiguration( + ConfigurationField.create(NAME_FIELD1, LABEL_FIELD1, TOOLTIP_FIELD1).required(), + ConfigurationField.create(NAME_FIELD1, LABEL_FIELD2, TOOLTIP_FIELD2)); + }); + assertTrue(exception.getMessage().startsWith(ERROR_DUPLICATE_DEFINITIONS_FOR_FIELD)); + } + + @Test + void getFields() { + } + + @Test + void printIniFileTemplate() { + } +} \ No newline at end of file diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/DBConnectorTest.java b/rabbit-core/src/test/java/org/ohdsi/databases/DBConnectorTest.java new file mode 100644 index 00000000..d6883e1a --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/DBConnectorTest.java @@ -0,0 +1,46 @@ +package org.ohdsi.databases; + +import org.junit.jupiter.api.Test; +import org.ohdsi.databases.configuration.DbType; + +import java.sql.Driver; +import java.sql.DriverManager; +import java.util.Enumeration; + +import static org.junit.jupiter.api.Assertions.*; + +class DBConnectorTest { + + public static void main(String[] args) { + DBConnectorTest dbConnectorTest = new DBConnectorTest(); + dbConnectorTest.verifyDrivers(); + } + + @Test + void verifyDrivers() { + // verify that a JDBC driver that is not included/supported cannot be loaded + String notSupportedDriver = "org.sqlite.JDBC"; // change this if WhiteRabbit starts supporting SQLite + assertFalse(DbType.driverNames().contains(notSupportedDriver), "Cannot test this for a supported driver."); + assertThrows(ClassNotFoundException.class, () -> + testJDBCDriverAndVersion(notSupportedDriver)); + DbType.driverNames().forEach(driver -> { + try { + testJDBCDriverAndVersion(driver); + } catch (ClassNotFoundException e) { + fail(String.format("JDBC driver class could not be loaded for %s", driver)); + } + }); + System.out.println("All configured JDBC drivers could be loaded."); + } + + void testJDBCDriverAndVersion(String driverName) throws ClassNotFoundException { + Enumeration drivers = DriverManager.getDrivers(); + while (drivers.hasMoreElements()) { + Driver driver = drivers.nextElement(); + Class driverClass = Class.forName(driverName); + if (driver.getClass().isAssignableFrom(driverClass)) { + int ignoredMajorVersion = driver.getMajorVersion(); + } + } + } +} \ No newline at end of file diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/SnowflakeTestUtils.java b/rabbit-core/src/test/java/org/ohdsi/databases/SnowflakeTestUtils.java new file mode 100644 index 00000000..1f40631f --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/SnowflakeTestUtils.java @@ -0,0 +1,91 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.apache.commons.lang.StringUtils; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.function.BooleanSupplier; + +public class SnowflakeTestUtils { + + public static String getEnvOrFail(String name) { + String value = System.getenv(name); + if (StringUtils.isEmpty(value)) { + throw new RuntimeException(String.format("Environment variable '%s' is not set.", name)); + } + + return value; + } + + public static String getPropertyOrFail(String name) { + String value = System.getProperty(name); + if (StringUtils.isEmpty(value)) { + throw new RuntimeException(String.format("System property '%s' is not set.", name)); + } + + return value; + } + + @FunctionalInterface + public interface ReaderInterface { + String getOrFail(String name); + } + + public static class EnvironmentReader implements ReaderInterface { + public String getOrFail(String name) { + return getEnvOrFail(name); + } + } + public static class PropertyReader implements ReaderInterface { + public String getOrFail(String name) { + return getPropertyOrFail(name); + } + } + + public static class SnowflakeSystemPropertiesFileChecker implements BooleanSupplier { + @Override + public boolean getAsBoolean() { + String buildDirectory = System.getProperty("projectBuildDirectory"); + Path snowflakeEnvVarPath = Paths.get(buildDirectory,"../..", "snowflake.env"); + if (StringUtils.isNotEmpty(buildDirectory) && Files.exists(snowflakeEnvVarPath)) { + try { + loadSystemProperties(snowflakeEnvVarPath); + } catch (IOException e) { + throw new RuntimeException(e); + } + return true; + } + // check the endpoint here and return either true or false + return false; + } + + private void loadSystemProperties(Path envVarFile) throws IOException { + Files.lines(envVarFile) + .map(line -> line.replaceAll("^export ", "")) + .map(line2 -> line2.split("=", 2)) + .forEach(v -> System.setProperty(v[0], v[1])); + } + } +} diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/TestConfigurationField.java b/rabbit-core/src/test/java/org/ohdsi/databases/TestConfigurationField.java new file mode 100644 index 00000000..f5462857 --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/TestConfigurationField.java @@ -0,0 +1,142 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.junit.jupiter.api.Test; +import org.ohdsi.databases.configuration.ConfigurationField; +import org.ohdsi.databases.configuration.DBConfiguration; +import org.ohdsi.databases.configuration.FieldValidator; +import org.ohdsi.databases.configuration.ValidationFeedback; + +import static org.junit.jupiter.api.Assertions.*; +import static org.ohdsi.databases.configuration.ConfigurationField.*; + +class TestConfigurationField { + + @Test + void testStandardValidators() { + final String REQUIRED_FIELD = "REQUIRED_FIELD"; + final String OPTIONAL_INTEGER_FIELD = "OPTIONAL_INTEGER_FIELD"; + final String REQUIRED_INTEGER_FIELD = "REQUIRED_INTEGER_FIELD"; + final String OPTIONAL_YESNO_FIELD = "OPTIONAL_YESNO_FIELD"; + final String REQUIRED_YESNO_FIELD = "REQUIRED_YESNO_FIELD"; + DBConfiguration configuration = new DBConfiguration( + ConfigurationField + .create(REQUIRED_FIELD, REQUIRED_FIELD, "") + .required(), + ConfigurationField + .create(OPTIONAL_INTEGER_FIELD, OPTIONAL_INTEGER_FIELD, "") + .integerValue(), + ConfigurationField + .create(REQUIRED_INTEGER_FIELD, REQUIRED_INTEGER_FIELD, "") + .integerValue() + .required(), + ConfigurationField + .create(OPTIONAL_YESNO_FIELD, OPTIONAL_YESNO_FIELD, "") + .yesNoValue(), + ConfigurationField + .create(REQUIRED_YESNO_FIELD, REQUIRED_YESNO_FIELD, "") + .yesNoValue() + .required() + ); + + // test for values in required fields + ValidationFeedback feedback = configuration.validateAll(); + assertEquals(0, feedback.getWarnings().size()); + assertEquals(3, feedback.getErrors().size()); + String expectedErrorKey = String.format(VALUE_REQUIRED_FORMAT_STRING, REQUIRED_FIELD, REQUIRED_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(REQUIRED_FIELD)); + expectedErrorKey = String.format(VALUE_REQUIRED_FORMAT_STRING, REQUIRED_INTEGER_FIELD, REQUIRED_INTEGER_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(REQUIRED_INTEGER_FIELD)); + expectedErrorKey = String.format(VALUE_REQUIRED_FORMAT_STRING, REQUIRED_YESNO_FIELD, REQUIRED_YESNO_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(REQUIRED_YESNO_FIELD)); + + // set (valid) values where required + configuration.getField(REQUIRED_FIELD).setValue("some value"); + configuration.getField(REQUIRED_INTEGER_FIELD).setValue("123"); + configuration.getField(REQUIRED_YESNO_FIELD).setValue("yes"); + feedback = configuration.validateAll(); + assertEquals(0, feedback.getWarnings().size()); + assertEquals(0, feedback.getErrors().size()); + + // set some bogus values + configuration.getField(REQUIRED_INTEGER_FIELD).setValue("abc"); + configuration.getField(REQUIRED_YESNO_FIELD).setValue("of course!"); + configuration.getField(OPTIONAL_YESNO_FIELD).setValue("maybe not?"); + feedback = configuration.validateAll(); + assertEquals(0, feedback.getWarnings().size()); + assertEquals(3, feedback.getErrors().size()); + expectedErrorKey = String.format(INTEGER_VALUE_REQUIRED_FORMAT_STRING, REQUIRED_INTEGER_FIELD, REQUIRED_INTEGER_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(REQUIRED_INTEGER_FIELD)); + expectedErrorKey = String.format(ONLY_YESNO_ALLOWED_FORMAT_STRING, REQUIRED_YESNO_FIELD, REQUIRED_YESNO_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(REQUIRED_YESNO_FIELD)); + expectedErrorKey = String.format(ONLY_YESNO_ALLOWED_FORMAT_STRING, OPTIONAL_YESNO_FIELD, OPTIONAL_YESNO_FIELD); + assertTrue(feedback.getErrors().containsKey(expectedErrorKey)); + assertTrue(feedback.getErrors().get(expectedErrorKey).get(0).name.equalsIgnoreCase(OPTIONAL_YESNO_FIELD)); + + // and test the normalization of a yes/no field + configuration.getField(REQUIRED_INTEGER_FIELD).setValue("0"); // no error wanted here either + configuration.getField(REQUIRED_YESNO_FIELD).setValue("YeS"); + configuration.getField(OPTIONAL_YESNO_FIELD).setValue("NO"); + feedback = configuration.validateAll(); + assertEquals(0, feedback.getWarnings().size()); + assertEquals(0, feedback.getErrors().size()); + assertEquals("yes", configuration.getField(REQUIRED_YESNO_FIELD).getValue()); + assertEquals("no", configuration.getField(OPTIONAL_YESNO_FIELD).getValue()); + } + + static class WarningValidator implements FieldValidator { + final static String expectedValue = "Expected value"; + final static String warning = "Field does not contain the expected value!"; + @Override + public ValidationFeedback validate(ConfigurationField field) { + ValidationFeedback feedback = new ValidationFeedback(); + + if (!field.getValue().equalsIgnoreCase(expectedValue)) { + feedback.addWarning(warning, field); + } + + return feedback; + } + } + + @Test + void testBespokeWarningValidator() { + final String FIELD_NAME = "FieldName"; + DBConfiguration configuration = new DBConfiguration( + ConfigurationField + .create(FIELD_NAME, FIELD_NAME, "") + .addValidator(new WarningValidator()) + .setValue("")); + + ValidationFeedback feedback = configuration.validateAll(); + assertEquals(1, feedback.getWarnings().size()); + assertEquals(0, feedback.getErrors().size()); + assertTrue(feedback.getWarnings().get(WarningValidator.warning).get(0).name.equalsIgnoreCase(FIELD_NAME)); + + configuration.getFields().get(0).setValue(WarningValidator.expectedValue); + feedback = configuration.validateAll(); + assertEquals(0, feedback.getWarnings().size()); + assertEquals(0, feedback.getErrors().size()); + } +} \ No newline at end of file diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/TestSnowflakeHandler.java b/rabbit-core/src/test/java/org/ohdsi/databases/TestSnowflakeHandler.java new file mode 100644 index 00000000..bb942336 --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/TestSnowflakeHandler.java @@ -0,0 +1,100 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases; + +import org.apache.commons.lang.StringUtils; +import org.junit.jupiter.api.Test; +import org.ohdsi.databases.configuration.*; +import org.ohdsi.utilities.files.IniFile; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.PrintStream; +import java.util.ArrayList; + +import static org.junit.jupiter.api.Assertions.*; +import static org.ohdsi.databases.SnowflakeHandler.*; + +class TestSnowflakeHandler { + + Logger logger = LoggerFactory.getLogger(TestSnowflakeHandler.class); + + @Test + void testPrintIniFileTemplate() throws IOException { + String output; + try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); PrintStream printStream = new PrintStream(outputStream)) { + DBConfiguration configuration = new SnowflakeConfiguration(); + configuration.printIniFileTemplate(printStream); + output = outputStream.toString(); + for (ConfigurationField field: configuration.getFields()) { + assertTrue(output.contains(field.name), String.format("ini file template should contain field name (%s)", field.name)); + assertTrue(output.contains(field.toolTip), String.format("ini file template should contain tool tip (%s)", field.toolTip)); + if (!StringUtils.isEmpty(field.getDefaultValue())) { + assertTrue(output.contains(field.getDefaultValue()), String.format("ini file template should contain default value (%s)", field.getDefaultValue())); + } + } + } + } + + @Test + void testLoadAndValidateConfiguration() { + DBConfiguration snowflakeConfiguration = new SnowflakeConfiguration(); + IniFile iniFile = new IniFile(); + + iniFile.set(DBConfiguration.DATA_TYPE_FIELD, DbType.SNOWFLAKE.name()); + + // start with no values set, should generate an error for each required field + ValidationFeedback feedback = snowflakeConfiguration.loadAndValidateConfiguration(iniFile); + assertFalse(feedback.hasWarnings()); + assertTrue(feedback.hasErrors()); + assertEquals(6,feedback.getErrors().size()); + + // fill in all required fields, verify no errors + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_ACCOUNT, "some-account"); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_USER, "some-user"); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_PASSWORD, "some-password"); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_WAREHOUSE, "some-warehouse"); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_DATABASE, "some-database"); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_SCHEMA, "some-schema"); + + feedback = snowflakeConfiguration.loadAndValidateConfiguration(iniFile); + assertFalse(feedback.hasWarnings()); + assertFalse(feedback.hasErrors()); + + // add (invalid) value for authenticator field, should generate two errors + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_AUTHENTICATOR, "some-value"); + + feedback = snowflakeConfiguration.loadAndValidateConfiguration(iniFile); + assertFalse(feedback.hasWarnings()); + assertTrue(feedback.hasErrors()); + assertEquals(2,feedback.getErrors().size()); + assertTrue(feedback.getErrors().containsKey(SnowflakeConfiguration.ERROR_MUST_NOT_SET_PASSWORD_AND_AUTHENTICATOR), + "there should be an error indicating that both password and authenticator were set"); + assertEquals(1, + (int) new ArrayList<>(feedback.getErrors().keySet()).stream().filter(k -> k.startsWith(SnowflakeConfiguration.ERROR_VALUE_CAN_ONLY_BE_ONE_OF)).count(), + "there should be an error indicating that a wrong value was set for the authenticator field"); + + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_PASSWORD, null); + iniFile.set(SnowflakeConfiguration.SNOWFLAKE_AUTHENTICATOR, "externalbrowser"); + feedback = snowflakeConfiguration.loadAndValidateConfiguration(iniFile); + assertFalse(feedback.hasWarnings()); + assertFalse(feedback.hasErrors()); + } +} \ No newline at end of file diff --git a/rabbit-core/src/test/java/org/ohdsi/databases/configuration/DbTypeTest.java b/rabbit-core/src/test/java/org/ohdsi/databases/configuration/DbTypeTest.java new file mode 100644 index 00000000..c3fbd5b7 --- /dev/null +++ b/rabbit-core/src/test/java/org/ohdsi/databases/configuration/DbTypeTest.java @@ -0,0 +1,37 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.databases.configuration; + +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.*; + +class DbTypeTest { + + @Test + void testPickList() { + List labelsFromAllDbTypeValues = Stream.of(DbType.values()).map(DbType::label).sorted().collect(Collectors.toList()); + List labelsFromPickList = Stream.of(DbType.pickList()).sorted().collect(Collectors.toList()); + + assertEquals(labelsFromAllDbTypeValues, labelsFromPickList, "The picklist should contain all the labels defined in the DbType enum"); + } +} \ No newline at end of file diff --git a/rabbit-core/src/test/resources/snowflake.ini b/rabbit-core/src/test/resources/snowflake.ini new file mode 100644 index 00000000..6299dcea --- /dev/null +++ b/rabbit-core/src/test/resources/snowflake.ini @@ -0,0 +1,16 @@ +# Usage: dist/bin/whiteRabbit -ini +WORKING_FOLDER = . +DATA_TYPE = Snowflake +SNOWFLAKE_ACCOUNT = some-account +SNOWFLAKE_USER = some-user +SNOWFLAKE_PASSWORD = some-password +SNOWFLAKE_WAREHOUSE = some-warehouse +SNOWFLAKE_DATABASE = some-database +SNOWFLAKE_SCHEMA = some-schema +TABLES_TO_SCAN = * # Comma-delimited list of table names to scan. Use "*" (asterix) to include all tables in the database +SCAN_FIELD_VALUES = yes # Include the frequency of field values in the scan report? "yes" or "no" +MIN_CELL_COUNT = 5 # Minimum frequency for a field value to be included in the report +MAX_DISTINCT_VALUES = 1000 # Maximum number of distinct values per field to be reported +ROWS_PER_TABLE = 100000 # Maximum number of rows per table to be scanned for field values +CALCULATE_NUMERIC_STATS = no # Include average, standard deviation and quartiles in the scan report? "yes" or "no" +NUMERIC_STATS_SAMPLER_SIZE = 500 # Maximum number of rows used to calculate numeric statistics diff --git a/rabbitinahat/pom.xml b/rabbitinahat/pom.xml index 533b81d4..8876ed55 100644 --- a/rabbitinahat/pom.xml +++ b/rabbitinahat/pom.xml @@ -36,7 +36,28 @@ org.apache.maven.plugins maven-surefire-plugin - 2.22.2 + + + false + com.github.caciocavallosilano.cacio.ctc.CTCToolkit + com.github.caciocavallosilano.cacio.ctc.CTCGraphicsEnvironment + + + --add-exports=java.desktop/java.awt=ALL-UNNAMED + --add-exports=java.desktop/java.awt.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.image=ALL-UNNAMED + --add-exports=java.desktop/sun.java2d=ALL-UNNAMED + --add-exports=java.desktop/java.awt.dnd.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.event=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.datatransfer=ALL-UNNAMED + --add-exports=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.desktop/java.awt=ALL-UNNAMED + --add-opens=java.desktop/sun.java2d=ALL-UNNAMED + --add-opens=java.base/java.lang.reflect=ALL-UNNAMED + + @@ -56,6 +77,35 @@ + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.4.1 + + + enforce-max-java-version + verify + enforce + + + + ${org.ohdsi.whiterabbit.maxjdkversion} + test + + + true + + + + + + org.codehaus.mojo + extra-enforcer-rules + 1.7.0 + + + @@ -64,47 +114,12 @@ - org.ohdsi rabbit-core ${project.version} - - - org.apache.maven.plugins - maven-surefire-plugin - 2.22.2 - test - - junit junit @@ -127,7 +142,7 @@ com.github.caciocavallosilano cacio-tta - 1.10 + 1.17.1 test @@ -137,6 +152,11 @@ 3.1.0 test + + org.junit.jupiter + junit-jupiter-api + 5.9.2 + test + - diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/DescriptionTextArea.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/DescriptionTextArea.java index 4d2db70e..9cfdeafb 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/DescriptionTextArea.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/DescriptionTextArea.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat; import javax.swing.JTextArea; diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FetchCDMModelFromServer.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FetchCDMModelFromServer.java index 7dc47aee..cad7a708 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FetchCDMModelFromServer.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FetchCDMModelFromServer.java @@ -17,7 +17,8 @@ ******************************************************************************/ package org.ohdsi.rabbitInAHat; -import org.ohdsi.databases.DbType; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; import org.ohdsi.databases.RichConnection; import org.ohdsi.utilities.files.Row; import org.ohdsi.utilities.files.WriteCSVFileWithHeader; @@ -31,7 +32,12 @@ public class FetchCDMModelFromServer { public static void main(String[] args) { - RichConnection connection = new RichConnection("127.0.0.1/ohdsi", null, "postgres", "F1r3starter", DbType.POSTGRESQL); + DbSettings dbSettings = new DbSettings(); + dbSettings.server = "127.0.0.1/ohdsi"; + dbSettings.user = "postgres"; + dbSettings.password = "F1r3starter"; + dbSettings.dbType = DbType.POSTGRESQL; + RichConnection connection = new RichConnection(dbSettings); connection.use("cdm5"); WriteCSVFileWithHeader out = new WriteCSVFileWithHeader("c:/temp/CDMV5Model.csv"); diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FilterDialog.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FilterDialog.java index a39f6b14..33fe2e6d 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FilterDialog.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/FilterDialog.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat; import java.awt.Container; diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/MappingPanel.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/MappingPanel.java index 0cfd1e0b..2a935797 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/MappingPanel.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/MappingPanel.java @@ -62,6 +62,8 @@ public class MappingPanel extends JPanel implements MouseListener, MouseMotionLi public static final int MIN_SPACE_BETWEEN_COLUMNS = 200; public static final int ARROW_START_WIDTH = 50; public static final int BORDER_HEIGHT = 25; + // Extra margin between header and first item when using stem table + public static final int STEM_HEIGHT_MARGIN = ITEM_HEIGHT / 2; private int sourceX; private int cdmX; @@ -240,7 +242,7 @@ private void setLabeledRectanglesLocation(List components, int int y = HEADER_HEIGHT + HEADER_TOP_MARGIN; if (ObjectExchange.etl.hasStemTable()) { // Move all non-stem items - y = HEADER_TOP_MARGIN + ITEM_HEIGHT; + y += STEM_HEIGHT_MARGIN; } for (LabeledRectangle component : components) { // Exception for laying out the stem table @@ -264,8 +266,18 @@ private void setLabeledRectanglesLocation(List components, int public Dimension getMinimumSize() { Dimension dimension = new Dimension(); dimension.width = 2 * (ITEM_WIDTH + MARGIN) + MIN_SPACE_BETWEEN_COLUMNS; - dimension.height = Math.min(HEADER_HEIGHT + HEADER_TOP_MARGIN + Math.max(sourceComponents.size(), cdmComponents.size()) * (ITEM_HEIGHT + MARGIN), - maxHeight); + int maxComponentsSize = Math.max(sourceComponents.size(), cdmComponents.size()); + int componentsHeight = maxComponentsSize * (ITEM_HEIGHT + MARGIN); + dimension.height = Math.min(HEADER_HEIGHT + HEADER_TOP_MARGIN + componentsHeight, maxHeight); + + if (ObjectExchange.etl.hasStemTable()) { + dimension.height += STEM_HEIGHT_MARGIN; + // For the table mapping panel, deduct the stem table from the items (as it's not shown as a normal item in the list) + boolean isTablesPanel = cdmComponents.stream().allMatch(n -> (n.getItem() instanceof Table)); + if (!cdmComponents.isEmpty() && isTablesPanel) { + dimension.height -= (ITEM_HEIGHT + MARGIN); + } + } return dimension; } diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/RabbitInAHatMain.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/RabbitInAHatMain.java index 81fbdd9f..6d6ecfd6 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/RabbitInAHatMain.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/RabbitInAHatMain.java @@ -739,7 +739,6 @@ private Dimension getPreferredDimension() { if (matcher.groupCount() == 2) { preferredHeight = Integer.parseInt(matcher.group(1)); preferredWidth = Integer.parseInt(matcher.group(2)); - //System.out.println("Using cacio screen size: " + cacioScreenSize); } } } diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/SQLGenerator.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/SQLGenerator.java index 442a3bb9..fdf79454 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/SQLGenerator.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/SQLGenerator.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat; import org.ohdsi.rabbitInAHat.dataModel.*; diff --git a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/dataModel/StemTableFactory.java b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/dataModel/StemTableFactory.java index ca8602aa..250265b3 100644 --- a/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/dataModel/StemTableFactory.java +++ b/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/dataModel/StemTableFactory.java @@ -1,3 +1,20 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat.dataModel; import java.io.IOException; diff --git a/rabbitinahat/src/test/java/org/ohdsi/rabbitInAHat/TestRabbitInAHatMain.java b/rabbitinahat/src/test/java/org/ohdsi/rabbitInAHat/TestRabbitInAHatMain.java index bf0fea59..b6cc22b1 100644 --- a/rabbitinahat/src/test/java/org/ohdsi/rabbitInAHat/TestRabbitInAHatMain.java +++ b/rabbitinahat/src/test/java/org/ohdsi/rabbitInAHat/TestRabbitInAHatMain.java @@ -1,18 +1,33 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.rabbitInAHat; -import com.github.caciocavallosilano.cacio.ctc.junit.CacioAssertJRunner; +import com.github.caciocavallosilano.cacio.ctc.junit.CacioTest; import org.assertj.swing.annotation.GUITest; import org.assertj.swing.core.ComponentDragAndDrop; import org.assertj.swing.edt.GuiActionRunner; import org.assertj.swing.finder.JFileChooserFinder; import org.assertj.swing.fixture.FrameFixture; import org.assertj.swing.fixture.JFileChooserFixture; -import org.assertj.swing.image.ScreenshotTaker; import org.junit.After; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; -import org.junit.runner.RunWith; import java.awt.*; import java.awt.event.KeyEvent; @@ -27,13 +42,13 @@ import static org.ohdsi.rabbitInAHat.RabbitInAHatMain.*; /* - * CacioTestRunner enables running the Swing GUI tests in a virtual screen. This allows the integration tests to run + * The @CacioTest annotation below enables running the Swing GUI tests in a virtual screen. This allows the integration tests to run * anywhere without being blocked by the absence of a real screen (e.g. github actions), and without being * disrupted by unrelated user activity on workstations/laptops (any keyboard or mouse action). * For debugging purposes, you can disable the annotation below to have the tests run on your screen. Be aware that * any interaction with mouse or keyboard can (will) disrupt the tests if they run on your screen. */ -@RunWith(CacioAssertJRunner.class) +@CacioTest public class TestRabbitInAHatMain { private static FrameFixture window; @@ -81,7 +96,6 @@ public void openReport() throws URISyntaxException { fileChooser.selectFile(new File(Objects.requireNonNull(scanReportUrl).toURI())).approve(); } - @GUITest @Test public void openAndVerifySavedETLSpecs() throws URISyntaxException { // open the test ETL specification @@ -115,8 +129,8 @@ private void openETLSpecs(String specName) throws URISyntaxException { URL etlSpecsUrl = this.getClass().getClassLoader().getResource(specName); fileChooser.selectFile(new File(Objects.requireNonNull(etlSpecsUrl).toURI())).approve(); MappingPanel tablesPanel = window.panel(PANEL_TABLE_MAPPING).targetCastedTo(MappingPanel.class); - assertTrue("There should be source items", tablesPanel.getVisibleSourceComponents().size() > 0); - assertTrue("There should be target items", tablesPanel.getVisibleTargetComponents().size() > 0); + assertFalse("There should be source items", tablesPanel.getVisibleSourceComponents().isEmpty()); + assertFalse("There should be target items", tablesPanel.getVisibleTargetComponents().isEmpty()); } private void verifyTableMapping(MappingPanel tablesPanel, String sourceName, String targetName) { LabeledRectangle sourceTable = findMappableItem(tablesPanel.getVisibleSourceComponents(), sourceName); @@ -150,11 +164,6 @@ private void clickAndVerifyLabeledRectangles(MappingPanel tablesPanel, LabeledRe if (rectangles.length > 1) { window.robot().releaseKey(KeyEvent.VK_SHIFT); } -// if (!r.isSelected()) { -// ScreenshotTaker screenshotTaker = new ScreenshotTaker(); -// screenshotTaker.saveDesktopAsPng("problem.png"); -// System.out.println("Problem!"); -// } assertTrue(r.isSelected()); }); } diff --git a/whiterabbit/pom.xml b/whiterabbit/pom.xml index 0ea7943b..23234e3a 100644 --- a/whiterabbit/pom.xml +++ b/whiterabbit/pom.xml @@ -27,28 +27,126 @@ -Xmx1200m - org.ohdsi.whiteRabbit.WhiteRabbitMain + org.ohdsi.whiterabbit.WhiteRabbitMain whiteRabbit - org.apache.maven.plugins - maven-surefire-plugin - - 3.0.0-M8 + org.apache.maven.plugins + maven-surefire-plugin + - -Doracle.jdbc.timezoneAsRegion=false + 1 + false + ${skipUnitTests} + + false + com.github.caciocavallosilano.cacio.ctc.CTCToolkit + com.github.caciocavallosilano.cacio.ctc.CTCGraphicsEnvironment + + + -Doracle.jdbc.timezoneAsRegion=false + + --add-exports=java.base/java.util=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + + --add-opens java.base/java.lang.reflect=ALL-UNNAMED + --add-exports java.base/java.lang.reflect=ALL-UNNAMED + --add-exports=java.desktop/java.awt=ALL-UNNAMED + --add-exports=java.desktop/java.awt.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.image=ALL-UNNAMED + --add-exports=java.desktop/sun.java2d=ALL-UNNAMED + --add-exports=java.desktop/java.awt.dnd.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.event=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.datatransfer=ALL-UNNAMED + --add-exports=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.desktop/java.awt=ALL-UNNAMED + --add-opens=java.desktop/sun.java2d=ALL-UNNAMED + --add-opens=java.base/java.lang.reflect=ALL-UNNAMED + --add-opens java.base/java.lang=ALL-UNNAMED + - - - org.junit.jupiter - junit-jupiter - 5.8.2 - - - + + + org.apache.maven.plugins + maven-failsafe-plugin + + + propertyValue + ${project.build.directory} + + 1 + false + + **/*IT.java + + ${skipIntegrationTests} + + + --add-opens java.base/java.nio=ALL-UNNAMED + + --add-opens java.base/java.lang.reflect=ALL-UNNAMED + --add-exports java.base/java.lang.reflect=ALL-UNNAMED + --add-exports=java.desktop/java.awt=ALL-UNNAMED + --add-exports=java.desktop/java.awt.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.image=ALL-UNNAMED + --add-exports=java.desktop/sun.java2d=ALL-UNNAMED + --add-exports=java.desktop/java.awt.dnd.peer=ALL-UNNAMED + --add-exports=java.desktop/sun.awt=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.event=ALL-UNNAMED + --add-exports=java.desktop/sun.awt.datatransfer=ALL-UNNAMED + --add-exports=java.base/sun.security.action=ALL-UNNAMED + --add-opens=java.base/java.util=ALL-UNNAMED + --add-opens=java.desktop/java.awt=ALL-UNNAMED + --add-opens=java.desktop/sun.java2d=ALL-UNNAMED + --add-opens=java.base/java.lang.reflect=ALL-UNNAMED + --add-opens java.base/java.lang=ALL-UNNAMED + + + + + + integration-test + verify + + + + + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.4.1 + + + enforce-max-java-version + verify + enforce + + + + ${org.ohdsi.whiterabbit.maxjdkversion} + test + + + true + + + + + + org.codehaus.mojo + extra-enforcer-rules + 1.7.0 + + + @@ -99,6 +197,12 @@ org.testcontainers testcontainers test + + + com.fasterxml.jackson.core + jackson-annotations + + @@ -130,6 +234,55 @@ oracle-xe test + + org.ohdsi + rabbit-core + ${project.version} + test-jar + test + + + + + + + + one.util + streamex + 0.8.2 + + + org.apache.poi + poi-ooxml-lite + 5.2.4 + compile + + + com.github.caciocavallosilano + cacio-tta + 1.17.3 + test + + + + org.assertj + assertj-swing-junit + 3.17.1 + test + + + + org.assertj + assertj-swing + 3.17.1 + test + + + + org.apache.logging.log4j + log4j-core + 2.21.1 + diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/DataType.java b/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/DataType.java deleted file mode 100644 index ee6eaf18..00000000 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/DataType.java +++ /dev/null @@ -1,5 +0,0 @@ -package org.ohdsi.whiteRabbit.scan; - -public enum DataType { - EMPTY, TEXT, DATE, INT, REAL, VARCHAR; -} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/Console.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/Console.java similarity index 85% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/Console.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/Console.java index 374be963..1b0b389d 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/Console.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/Console.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit; +package org.ohdsi.whiterabbit; import java.io.IOException; import java.io.OutputStream; @@ -27,16 +27,10 @@ public class Console extends OutputStream { - private StringBuffer buffer = new StringBuffer(); + private StringBuilder buffer = new StringBuilder(); private WriteTextFile debug = null; private JTextArea textArea; - - public void println(String string) { - textArea.append(string + "\n"); - textArea.repaint(); - System.out.println(string); - } - + public void setTextArea(JTextArea textArea) { this.textArea = textArea; } @@ -66,8 +60,7 @@ public void write(int b) throws IOException { debug.writeln(buffer.toString()); debug.flush(); } - buffer = new StringBuffer(); + buffer = new StringBuilder(); } } - } diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ErrorReport.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/ErrorReport.java similarity index 96% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ErrorReport.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/ErrorReport.java index 28fe0eea..b7cd1059 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ErrorReport.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/ErrorReport.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit; +package org.ohdsi.whiterabbit; import java.io.File; import java.text.DecimalFormat; diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ObjectExchange.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/ObjectExchange.java similarity index 94% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ObjectExchange.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/ObjectExchange.java index 643e87a0..a03891ee 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/ObjectExchange.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/ObjectExchange.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit; +package org.ohdsi.whiterabbit; import javax.swing.JFrame; diff --git a/whiterabbit/src/main/java/org/ohdsi/whiterabbit/PanelsManager.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/PanelsManager.java new file mode 100644 index 00000000..a7e7e8c3 --- /dev/null +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/PanelsManager.java @@ -0,0 +1,32 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit; + +import javax.swing.*; +import java.util.List; + +/** + * Defines the interface between the application's main class and its (Swing) components (panels). + */ +public interface PanelsManager { + void runConnectionTest(); + + JButton getAddAllButton(); + + List getComponentsToDisableWhenRunning(); +} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/WhiteRabbitMain.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/WhiteRabbitMain.java similarity index 60% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/WhiteRabbitMain.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/WhiteRabbitMain.java index 13b34149..f42ceeeb 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/WhiteRabbitMain.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/WhiteRabbitMain.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit; +package org.ohdsi.whiterabbit; import java.awt.BorderLayout; import java.awt.Color; @@ -29,63 +29,57 @@ import java.awt.Toolkit; import java.awt.event.ActionEvent; import java.awt.event.ActionListener; -import java.awt.event.WindowAdapter; -import java.awt.event.WindowEvent; -import java.io.File; -import java.io.IOException; -import java.io.PrintStream; +import java.io.*; import java.net.URI; import java.net.URISyntaxException; import java.util.ArrayList; import java.util.List; +import java.util.Set; import java.util.Vector; -import javax.swing.BorderFactory; -import javax.swing.Box; -import javax.swing.BoxLayout; -import javax.swing.JButton; -import javax.swing.JCheckBox; -import javax.swing.JComboBox; -import javax.swing.JComponent; -import javax.swing.JDialog; -import javax.swing.JFileChooser; -import javax.swing.JFrame; -import javax.swing.JLabel; -import javax.swing.JList; -import javax.swing.JMenu; -import javax.swing.JMenuBar; -import javax.swing.JMenuItem; -import javax.swing.JOptionPane; -import javax.swing.JPanel; -import javax.swing.JPasswordField; -import javax.swing.JScrollPane; -import javax.swing.JSpinner; -import javax.swing.JTabbedPane; -import javax.swing.JTextArea; -import javax.swing.JTextField; +import javax.swing.*; import javax.swing.border.TitledBorder; import javax.swing.filechooser.FileNameExtensionFilter; import org.apache.commons.csv.CSVFormat; -import org.ohdsi.databases.DbType; -import org.ohdsi.databases.RichConnection; +import org.apache.commons.io.output.TeeOutputStream; +import org.ohdsi.databases.*; +import org.ohdsi.databases.configuration.*; import org.ohdsi.utilities.DirectoryUtilities; import org.ohdsi.utilities.StringUtilities; import org.ohdsi.utilities.Version; import org.ohdsi.utilities.files.IniFile; -import org.ohdsi.whiteRabbit.fakeDataGenerator.FakeDataGenerator; -import org.ohdsi.whiteRabbit.scan.SourceDataScan; +import org.ohdsi.whiterabbit.fakeDataGenerator.FakeDataGenerator; +import org.ohdsi.whiterabbit.gui.LocationsPanel; +import org.ohdsi.whiterabbit.scan.SourceDataScan; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * This is the WhiteRabbit main class */ -public class WhiteRabbitMain implements ActionListener { +public class WhiteRabbitMain implements ActionListener, PanelsManager { - public final static String DOCUMENTATION_URL = "http://ohdsi.github.io/WhiteRabbit"; - public final static String ACTION_CMD_HELP = "Open documentation"; + Logger logger = LoggerFactory.getLogger(WhiteRabbitMain.class); + + public static final String DOCUMENTATION_URL = "http://ohdsi.github.io/WhiteRabbit"; + public static final String ACTION_CMD_HELP = "Open documentation"; + + public static final String DELIMITED_TEXT_FILES = DbType.DELIMITED_TEXT_FILES.label(); + + public static final String LABEL_TEST_CONNECTION = "Test connection"; + public static final String LABEL_CONNECTION_SUCCESSFUL = "Connection successful"; + public static final String NAME_TABBED_PANE = "TabbedPane"; + public static final String LABEL_LOCATIONS = "Locations"; + public static final String LABEL_SCAN = "Scan"; + public static final String LABEL_SCAN_TABLES = "Scan tables"; + + public static final String LABEL_ADD_ALL_IN_DB = "Add all in DB"; + + public static final String TITLE_ERRORS_IN_DATABASE_CONFIGURATION = "There are errors in the database configuration"; + public static final String TITLE_WARNINGS_ABOUT_DATABASE_CONFIGURATION = "There are warnings about the database configuration"; private JFrame frame; - private JTextField folderField; private JTextField scanReportFileField; private JComboBox scanRowCount; @@ -95,43 +89,40 @@ public class WhiteRabbitMain implements ActionListener { private JComboBox numericStatsSampleSize; private JSpinner scanMinCellCount; private JSpinner generateRowCount; - private JComboBox sourceType; private JComboBox targetType; private JTextField targetUserField; private JTextField targetPasswordField; private JTextField targetServerField; private JTextField targetDatabaseField; - private JTextField sourceDelimiterField; private JComboBox targetCSVFormat; private JCheckBox doUniformSampling; - private JTextField sourceServerField; - private JTextField sourceUserField; - private JTextField sourcePasswordField; - private JTextField sourceDatabaseField; private JButton addAllButton; private JList tableList; private Vector tables = new Vector(); - private boolean sourceIsFiles = true; - private boolean sourceIsSas = false; private boolean targetIsFiles = false; + private LocationsPanel locationsPanel; + + private Console console; + + private boolean teeOutputStreams; // for testing/debugging purposes private List componentsToDisableWhenRunning = new ArrayList(); - public static void main(String[] args) { - new WhiteRabbitMain(args); + public String reportFilePath = ""; + + public static void main(String[] args) throws IOException { + new WhiteRabbitMain(false, args); } - public WhiteRabbitMain(String[] args) { - if (args.length == 2 && args[0].equalsIgnoreCase("-ini")) + public WhiteRabbitMain(boolean teeOutputStreams, String[] args) throws IOException { + this.teeOutputStreams = teeOutputStreams; + if (args.length == 2 && (args[0].equalsIgnoreCase("-ini") || args[0].equalsIgnoreCase("--ini"))) launchCommandLine(args[1]); else { frame = new JFrame("White Rabbit"); - frame.addWindowListener(new WindowAdapter() { - public void windowClosing(WindowEvent e) { - System.exit(0); - } - }); + frame.setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE); + frame.setLayout(new BorderLayout()); frame.setJMenuBar(createMenuBar()); @@ -148,72 +139,82 @@ public void windowClosing(WindowEvent e) { } } - private void launchCommandLine(String iniFileName) { + public JButton getAddAllButton() { + return this.addAllButton; + } + + public List getComponentsToDisableWhenRunning() { + return this.componentsToDisableWhenRunning; + } + + private void launchCommandLine(String iniFileName) throws IOException { IniFile iniFile = new IniFile(iniFileName); - DbSettings dbSettings = new DbSettings(); - if (iniFile.get("DATA_TYPE").equalsIgnoreCase("Delimited text files")) { - dbSettings.sourceType = DbSettings.SourceType.CSV_FILES; - if (iniFile.get("DELIMITER").equalsIgnoreCase("tab")) - dbSettings.delimiter = '\t'; - else - dbSettings.delimiter = iniFile.get("DELIMITER").charAt(0); - } else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("SAS7bdat")) { - dbSettings.sourceType = DbSettings.SourceType.SAS_FILES; + DbSettings dbSettings = getDbSettings(iniFile); + findTablesToScan(iniFile, dbSettings); + performSourceDataScan(iniFile, dbSettings); + } + + private DbSettings getDbSettings(IniFile iniFile) { + DbSettings dbSettings; + + DbType dbType = DbType.getDbType(iniFile.getDataType()); + if (dbType.supportsStorageHandler()) { + dbSettings = dbType.getStorageHandler().getDbSettings(iniFile, null, System.out); } else { - dbSettings.sourceType = DbSettings.SourceType.DATABASE; - dbSettings.user = iniFile.get("USER_NAME"); - dbSettings.password = iniFile.get("PASSWORD"); - dbSettings.server = iniFile.get("SERVER_LOCATION"); - dbSettings.database = iniFile.get("DATABASE_NAME"); - if (iniFile.get("DATA_TYPE").equalsIgnoreCase("MySQL")) - dbSettings.dbType = DbType.MYSQL; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("Oracle")) - dbSettings.dbType = DbType.ORACLE; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("PostgreSQL")) - dbSettings.dbType = DbType.POSTGRESQL; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("Redshift")) - dbSettings.dbType = DbType.REDSHIFT; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("SQL Server")) { - dbSettings.dbType = DbType.MSSQL; - if (iniFile.get("USER_NAME").length() != 0) { // Not using windows authentication - String[] parts = iniFile.get("USER_NAME").split("/"); - if (parts.length == 2) { - dbSettings.user = parts[1]; - dbSettings.domain = parts[0]; + dbSettings = new DbSettings(); + if (iniFile.get(DBConfiguration.DATA_TYPE_FIELD).equalsIgnoreCase(DELIMITED_TEXT_FILES)) { + dbSettings.sourceType = DbSettings.SourceType.CSV_FILES; + if (iniFile.get("DELIMITER").equalsIgnoreCase("tab")) + dbSettings.delimiter = '\t'; + else + dbSettings.delimiter = iniFile.get("DELIMITER").charAt(0); + } else if (iniFile.get(DBConfiguration.DATA_TYPE_FIELD).equalsIgnoreCase(DbType.SAS7BDAT.label())) { + dbSettings.sourceType = DbSettings.SourceType.SAS_FILES; + } else { + dbSettings.sourceType = DbSettings.SourceType.DATABASE; + dbSettings.user = iniFile.get("USER_NAME"); + dbSettings.password = iniFile.get("PASSWORD"); + dbSettings.server = iniFile.get("SERVER_LOCATION"); + dbSettings.database = iniFile.get("DATABASE_NAME"); + dbSettings.dbType = dbType; + if (dbType == DbType.SQL_SERVER) { + if (!iniFile.get("USER_NAME").isEmpty()) { // Not using windows authentication + String[] parts = iniFile.get("USER_NAME").split("/"); + if (parts.length == 2) { + dbSettings.user = parts[1]; + dbSettings.domain = parts[0]; + } } - } - } else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("Azure")) { - dbSettings.dbType = DbType.AZURE; - if (iniFile.get("USER_NAME").length() != 0) { // Not using windows authentication - String[] parts = iniFile.get("USER_NAME").split("/"); - if (parts.length == 2) { - dbSettings.user = parts[1]; - dbSettings.domain = parts[0]; + } else if (dbType == DbType.AZURE) { + if (!iniFile.get("USER_NAME").isEmpty()) { // Not using windows authentication + String[] parts = iniFile.get("USER_NAME").split("/"); + if (parts.length == 2) { + dbSettings.user = parts[1]; + dbSettings.domain = parts[0]; + } } - } - } else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("PDW")) { - dbSettings.dbType = DbType.PDW; - if (iniFile.get("USER_NAME").length() != 0) { // Not using windows authentication - String[] parts = iniFile.get("USER_NAME").split("/"); - if (parts.length == 2) { - dbSettings.user = parts[1]; - dbSettings.domain = parts[0]; + } else if (dbType == DbType.PDW) { + if (!iniFile.get("USER_NAME").isEmpty()) { // Not using windows authentication + String[] parts = iniFile.get("USER_NAME").split("/"); + if (parts.length == 2) { + dbSettings.user = parts[1]; + dbSettings.domain = parts[0]; + } } + } else if (dbType == DbType.BIGQUERY) { + /* GBQ requires database. Putting database into domain var for connect() */ + dbSettings.domain = dbSettings.database; } - } else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("MS Access")) - dbSettings.dbType = DbType.MSACCESS; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("Teradata")) - dbSettings.dbType = DbType.TERADATA; - else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("BigQuery")) { - dbSettings.dbType = DbType.BIGQUERY; - /* GBQ requires database. Putting database into domain var for connect() */ - dbSettings.domain = dbSettings.database; } } + return dbSettings; + } + + private void findTablesToScan(IniFile iniFile, DbSettings dbSettings) { if (iniFile.get("TABLES_TO_SCAN").equalsIgnoreCase("*")) { if (dbSettings.sourceType == DbSettings.SourceType.DATABASE) { - try (RichConnection connection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { + try (RichConnection connection = new RichConnection(dbSettings)) { dbSettings.tables.addAll(connection.getTableNames(dbSettings.database)); } } else { @@ -242,7 +243,9 @@ else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("BigQuery")) { dbSettings.tables.add(table); } } + } + private void performSourceDataScan(IniFile iniFile, DbSettings dbSettings) throws IOException { SourceDataScan sourceDataScan = new SourceDataScan(); int maxRows = Integer.parseInt(iniFile.get("ROWS_PER_TABLE")); boolean scanValues = iniFile.get("SCAN_FIELD_VALUES").equalsIgnoreCase("yes"); @@ -265,17 +268,19 @@ else if (iniFile.get("DATA_TYPE").equalsIgnoreCase("BigQuery")) { sourceDataScan.setMaxValues(maxValues); sourceDataScan.setCalculateNumericStats(calculateNumericStats); sourceDataScan.setNumStatsSamplerSize(numericStatsSamplerSize); - sourceDataScan.process(dbSettings, iniFile.get("WORKING_FOLDER") + "/" + SourceDataScan.SCAN_REPORT_FILE_NAME); + reportFilePath = iniFile.get("WORKING_FOLDER") + "/" + SourceDataScan.SCAN_REPORT_FILE_NAME; + sourceDataScan.process(dbSettings, reportFilePath); } private JComponent createTabsPanel() { JTabbedPane tabbedPane = new JTabbedPane(); + tabbedPane.setName(NAME_TABBED_PANE); - JPanel locationPanel = createLocationsPanel(); - tabbedPane.addTab("Locations", null, locationPanel, "Specify the location of the source data and the working folder"); + this.locationsPanel = createLocationsPanel(componentsToDisableWhenRunning); + tabbedPane.addTab(LABEL_LOCATIONS, null, locationsPanel, "Specify the location of the source data and the working folder"); JPanel scanPanel = createScanPanel(); - tabbedPane.addTab("Scan", null, scanPanel, "Create a scan of the source data"); + tabbedPane.addTab(LABEL_SCAN, null, scanPanel, "Create a scan of the source data"); JPanel fakeDataPanel = createFakeDataPanel(); tabbedPane.addTab("Fake data generation", null, fakeDataPanel, "Create fake data based on a scan report for development purposes"); @@ -283,136 +288,10 @@ private JComponent createTabsPanel() { return tabbedPane; } - private JPanel createLocationsPanel() { - JPanel panel = new JPanel(); - - panel.setLayout(new GridBagLayout()); - GridBagConstraints c = new GridBagConstraints(); - c.fill = GridBagConstraints.BOTH; - c.weightx = 0.5; - - JPanel folderPanel = new JPanel(); - folderPanel.setLayout(new BoxLayout(folderPanel, BoxLayout.X_AXIS)); - folderPanel.setBorder(BorderFactory.createTitledBorder("Working folder")); - folderField = new JTextField(); - folderField.setText((new File("").getAbsolutePath())); - folderField.setToolTipText("The folder where all output will be written"); - folderPanel.add(folderField); - JButton pickButton = new JButton("Pick folder"); - pickButton.setToolTipText("Pick a different working folder"); - folderPanel.add(pickButton); - pickButton.addActionListener(e -> pickFolder()); - componentsToDisableWhenRunning.add(pickButton); - c.gridx = 0; - c.gridy = 0; - c.gridwidth = 1; - panel.add(folderPanel, c); - - JPanel sourcePanel = new JPanel(); - sourcePanel.setLayout(new GridLayout(0, 2)); - sourcePanel.setBorder(BorderFactory.createTitledBorder("Source data location")); - sourcePanel.add(new JLabel("Data type")); - sourceType = new JComboBox<>(new String[] { "Delimited text files", "SAS7bdat", "MySQL", "Oracle", "SQL Server", "PostgreSQL", "MS Access", "PDW", "Redshift", "Teradata", "BigQuery", "Azure"}); - sourceType.setToolTipText("Select the type of source data available"); - sourceType.addItemListener(itemEvent -> { - String selectedSourceType = itemEvent.getItem().toString(); - sourceIsFiles = selectedSourceType.equals("Delimited text files"); - sourceIsSas = selectedSourceType.equals("SAS7bdat"); - boolean sourceIsDatabase = !(sourceIsFiles || sourceIsSas); - - sourceServerField.setEnabled(sourceIsDatabase); - sourceUserField.setEnabled(sourceIsDatabase); - sourcePasswordField.setEnabled(sourceIsDatabase); - sourceDatabaseField.setEnabled(sourceIsDatabase && !selectedSourceType.equals("Azure")); - sourceDelimiterField.setEnabled(sourceIsFiles); - addAllButton.setEnabled(sourceIsDatabase); - - if (sourceIsDatabase && selectedSourceType.equals("Oracle")) { - sourceServerField.setToolTipText("For Oracle servers this field contains the SID, servicename, and optionally the port: '/', ':/', '/', or ':/'"); - sourceUserField.setToolTipText("For Oracle servers this field contains the name of the user used to log in"); - sourcePasswordField.setToolTipText("For Oracle servers this field contains the password corresponding to the user"); - sourceDatabaseField.setToolTipText("For Oracle servers this field contains the schema (i.e. 'user' in Oracle terms) containing the source tables"); - } else if (sourceIsDatabase && selectedSourceType.equals("PostgreSQL")) { - sourceServerField.setToolTipText("For PostgreSQL servers this field contains the host name and database name (/)"); - sourceUserField.setToolTipText("The user used to log in to the server"); - sourcePasswordField.setToolTipText("The password used to log in to the server"); - sourceDatabaseField.setToolTipText("For PostgreSQL servers this field contains the schema containing the source tables"); - } else if (sourceIsDatabase && selectedSourceType.equals("BigQuery")) { - sourceServerField.setToolTipText("GBQ SA & UA: ProjectID"); - sourceUserField.setToolTipText("GBQ SA only: OAuthServiceAccountEMAIL"); - sourcePasswordField.setToolTipText("GBQ SA only: OAuthPvtKeyPath"); - sourceDatabaseField.setToolTipText("GBQ SA & UA: Data Set within ProjectID"); - } else if (sourceIsDatabase) { - if (selectedSourceType.equals("Azure")) { - sourceServerField.setToolTipText("For Azure, this field contains the host name and database name (;database=)"); - } else { - sourceServerField.setToolTipText("This field contains the name or IP address of the database server"); - } - if (selectedSourceType.equals("SQL Server")) { - sourceUserField.setToolTipText("The user used to log in to the server. Optionally, the domain can be specified as / (e.g. 'MyDomain/Joe')"); - } else { - sourceUserField.setToolTipText("The user used to log in to the server"); - } - sourcePasswordField.setToolTipText("The password used to log in to the server"); - if (selectedSourceType.equals("Azure")) { - sourceDatabaseField.setToolTipText("For Azure, leave this empty"); - } else { - sourceDatabaseField.setToolTipText("The name of the database containing the source tables"); - } - } - }); - sourcePanel.add(sourceType); - - sourcePanel.add(new JLabel("Server location")); - sourceServerField = new JTextField("127.0.0.1"); - sourceServerField.setEnabled(false); - sourcePanel.add(sourceServerField); - sourcePanel.add(new JLabel("User name")); - sourceUserField = new JTextField(""); - sourceUserField.setEnabled(false); - sourcePanel.add(sourceUserField); - sourcePanel.add(new JLabel("Password")); - sourcePasswordField = new JPasswordField(""); - sourcePasswordField.setEnabled(false); - sourcePanel.add(sourcePasswordField); - sourcePanel.add(new JLabel("Database name")); - sourceDatabaseField = new JTextField(""); - sourceDatabaseField.setEnabled(false); - sourcePanel.add(sourceDatabaseField); - - sourcePanel.add(new JLabel("Delimiter")); - sourceDelimiterField = new JTextField(","); - sourceDelimiterField.setToolTipText("The delimiter that separates values. Enter 'tab' for tab."); - sourcePanel.add(sourceDelimiterField); - - c.gridx = 0; - c.gridy = 1; - c.gridwidth = 1; - panel.add(sourcePanel, c); - - JPanel testConnectionButtonPanel = new JPanel(); - testConnectionButtonPanel.setLayout(new BoxLayout(testConnectionButtonPanel, BoxLayout.X_AXIS)); - testConnectionButtonPanel.add(Box.createHorizontalGlue()); - - JButton testConnectionButton = new JButton("Test connection"); - testConnectionButton.setBackground(new Color(151, 220, 141)); - testConnectionButton.setToolTipText("Test the connection"); - testConnectionButton.addActionListener(new ActionListener() { - public void actionPerformed(ActionEvent e) { - testConnection(getSourceDbSettings()); - } - }); - componentsToDisableWhenRunning.add(testConnectionButton); - testConnectionButtonPanel.add(testConnectionButton); - - c.gridx = 0; - c.gridy = 2; - c.gridwidth = 1; - panel.add(testConnectionButtonPanel, c); - - return panel; + private LocationsPanel createLocationsPanel(List componentsToDisableWhenRunning) { + return new LocationsPanel(frame, this); } - + private JPanel createScanPanel() { JPanel panel = new JPanel(); panel.setLayout(new BorderLayout()); @@ -421,12 +300,14 @@ private JPanel createScanPanel() { tablePanel.setLayout(new BorderLayout()); tablePanel.setBorder(new TitledBorder("Tables to scan")); tableList = new JList(); + tableList.setName("TableList"); tableList.setToolTipText("Specify the tables (or CSV files) to be scanned here"); tablePanel.add(new JScrollPane(tableList), BorderLayout.CENTER); JPanel tableButtonPanel = new JPanel(); tableButtonPanel.setLayout(new GridLayout(3, 1)); - addAllButton = new JButton("Add all in DB"); + addAllButton = new JButton(LABEL_ADD_ALL_IN_DB); + addAllButton.setName(LABEL_ADD_ALL_IN_DB); addAllButton.setToolTipText("Add all tables in the database"); addAllButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent e) { @@ -436,6 +317,7 @@ public void actionPerformed(ActionEvent e) { addAllButton.setEnabled(false); tableButtonPanel.add(addAllButton); JButton addButton = new JButton("Add"); + addButton.setName("Add"); addButton.setToolTipText("Add tables to list"); addButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent e) { @@ -515,7 +397,8 @@ public void actionPerformed(ActionEvent e) { southPanel.add(Box.createVerticalStrut(3)); - JButton scanButton = new JButton("Scan tables"); + JButton scanButton = new JButton(LABEL_SCAN_TABLES); + scanButton.setName(LABEL_SCAN_TABLES); scanButton.setBackground(new Color(151, 220, 141)); scanButton.setToolTipText("Scan the selected tables"); scanButton.addActionListener(new ActionListener() { @@ -564,10 +447,10 @@ public void actionPerformed(ActionEvent e) { targetPanel.setLayout(new GridLayout(0, 2)); targetPanel.setBorder(BorderFactory.createTitledBorder("Target data location")); targetPanel.add(new JLabel("Data type")); - targetType = new JComboBox<>(new String[] {"Delimited text files", "MySQL", "Oracle", "SQL Server", "PostgreSQL", "PDW"}); + targetType = new JComboBox<>(new String[] {DELIMITED_TEXT_FILES, "MySQL", "Oracle", "SQL Server", "PostgreSQL", "PDW"}); targetType.setToolTipText("Select the type of source data available"); targetType.addItemListener(event -> { - targetIsFiles = event.getItem().toString().equals("Delimited text files"); + targetIsFiles = event.getItem().toString().equals(DELIMITED_TEXT_FILES); targetServerField.setEnabled(!targetIsFiles); targetUserField.setEnabled(!targetIsFiles); targetPasswordField.setEnabled(!targetIsFiles); @@ -649,12 +532,13 @@ public void actionPerformed(ActionEvent e) { fakeDataButtonPanel.add(generateRowCount); fakeDataButtonPanel.add(Box.createHorizontalGlue()); - JButton testConnectionButton = new JButton("Test connection"); + JButton testConnectionButton = new JButton(LABEL_TEST_CONNECTION); + testConnectionButton.setName(LABEL_TEST_CONNECTION); testConnectionButton.setBackground(new Color(151, 220, 141)); testConnectionButton.setToolTipText("Test the connection"); testConnectionButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent e) { - testConnection(getTargetDbSettings()); + testConnection(getTargetDbSettings(), null); } }); componentsToDisableWhenRunning.add(testConnectionButton); @@ -681,20 +565,37 @@ public void actionPerformed(ActionEvent e) { private JComponent createConsolePanel() { JTextArea consoleArea = new JTextArea(); + consoleArea.setName("Console"); consoleArea.setToolTipText("General progress information"); consoleArea.setEditable(false); - Console console = new Console(); + this.console = new Console(); console.setTextArea(consoleArea); - System.setOut(new PrintStream(console)); - System.setErr(new PrintStream(console)); + setOutputStreamsToConsole(console); JScrollPane consoleScrollPane = new JScrollPane(consoleArea); consoleScrollPane.setBorder(BorderFactory.createTitledBorder("Console")); - consoleScrollPane.setPreferredSize(new Dimension(800, 200)); + consoleScrollPane.setPreferredSize(new Dimension(800, 180)); consoleScrollPane.setAutoscrolls(true); ObjectExchange.console = console; return consoleScrollPane; } + private void setOutputStreamsToConsole(Console console) { + if (teeOutputStreams) { + System.setOut(new PrintStream(new TeeOutputStream(System.out, new PrintStream(console)))); + System.setErr(new PrintStream(new TeeOutputStream(System.err, new PrintStream(console)))); + } else { + System.setOut(new PrintStream(console)); + System.setErr(new PrintStream(console)); + } + + Thread resetOutputStreams = new Thread(() -> { + System.setOut(new PrintStream(new FileOutputStream(FileDescriptor.out))); + System.setErr(new PrintStream(new FileOutputStream(FileDescriptor.err))); + logger = LoggerFactory.getLogger(WhiteRabbitMain.class); + }); + Runtime.getRuntime().addShutdownHook(resetOutputStreams); + } + private void loadIcons(JFrame f) { List icons = new ArrayList(); icons.add(loadIcon("WhiteRabbit16.png", f)); @@ -719,23 +620,9 @@ private Image loadIcon(String name, JFrame f) { return null; } - private void pickFolder() { - JFileChooser fileChooser = new JFileChooser(new File(folderField.getText())); - fileChooser.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY); - int returnVal = fileChooser.showDialog(frame, "Select folder"); - if (returnVal == JFileChooser.APPROVE_OPTION) { - File selectedDirectory = fileChooser.getSelectedFile(); - if (!selectedDirectory.exists()) { - // When no directory is selected when approving, FileChooser incorrectly appends the current directory to the path. - // Take the opened directory instead. - selectedDirectory = fileChooser.getCurrentDirectory(); - } - folderField.setText(selectedDirectory.getAbsolutePath()); - } - } private void pickScanReportFile() { - JFileChooser fileChooser = new JFileChooser(new File(folderField.getText())); + JFileChooser fileChooser = new JFileChooser(new File(locationsPanel.getFolderField().getText())); fileChooser.setFileSelectionMode(JFileChooser.FILES_ONLY); int returnVal = fileChooser.showDialog(frame, "Select scan report file"); if (returnVal == JFileChooser.APPROVE_OPTION) @@ -750,13 +637,12 @@ private void removeTables() { } private void addAllTables() { - DbSettings sourceDbSettings = getSourceDbSettings(); - if (sourceDbSettings != null) { - RichConnection connection = new RichConnection(sourceDbSettings.server, sourceDbSettings.domain, sourceDbSettings.user, sourceDbSettings.password, - sourceDbSettings.dbType); - for (String table : connection.getTableNames(sourceDbSettings.database)) { + DbSettings dbSettings = getSourceDbSettings(null); + if (dbSettings != null) { + RichConnection connection = new RichConnection(dbSettings); + for (String table : connection.getTableNames(dbSettings.database)) { if (!tables.contains(table)) - tables.add((String) table); + tables.add(table); tableList.setListData(tables); } connection.close(); @@ -764,23 +650,15 @@ private void addAllTables() { } private void pickTables() { - DbSettings sourceDbSettings = getSourceDbSettings(); + DbSettings sourceDbSettings = getSourceDbSettings(null); if (sourceDbSettings != null) { if (sourceDbSettings.sourceType == DbSettings.SourceType.CSV_FILES || sourceDbSettings.sourceType == DbSettings.SourceType.SAS_FILES) { - JFileChooser fileChooser = new JFileChooser(new File(folderField.getText())); - fileChooser.setMultiSelectionEnabled(true); - fileChooser.setFileSelectionMode(JFileChooser.FILES_ONLY); - - if (sourceDbSettings.sourceType == DbSettings.SourceType.CSV_FILES) { - fileChooser.setFileFilter(new FileNameExtensionFilter("Delimited text files", "csv", "txt")); - } else if (sourceDbSettings.sourceType == DbSettings.SourceType.SAS_FILES) { - fileChooser.setFileFilter(new FileNameExtensionFilter("SAS Data Files", "sas7bdat")); - } + JFileChooser fileChooser = getjFileChooser(sourceDbSettings, locationsPanel); int returnVal = fileChooser.showDialog(frame, "Select tables"); if (returnVal == JFileChooser.APPROVE_OPTION) { for (File table : fileChooser.getSelectedFiles()) { - String tableName = DirectoryUtilities.getRelativePath(new File(folderField.getText()), table); + String tableName = DirectoryUtilities.getRelativePath(new File(locationsPanel.getFolderField().getText()), table); if (!tables.contains(tableName)) tables.add(tableName); tableList.setListData(tables); @@ -788,10 +666,9 @@ private void pickTables() { } } else if (sourceDbSettings.sourceType == DbSettings.SourceType.DATABASE) { - RichConnection connection = new RichConnection(sourceDbSettings.server, sourceDbSettings.domain, sourceDbSettings.user, - sourceDbSettings.password, sourceDbSettings.dbType); + RichConnection connection = new RichConnection(sourceDbSettings); String tableNames = StringUtilities.join(connection.getTableNames(sourceDbSettings.database), "\t"); - if (tableNames.length() == 0) { + if (tableNames.isEmpty()) { JOptionPane.showMessageDialog(frame, "No tables found in database " + sourceDbSettings.database, "Error fetching table names", JOptionPane.ERROR_MESSAGE); } else { @@ -809,78 +686,102 @@ private void pickTables() { } } - private DbSettings getSourceDbSettings() { - DbSettings dbSettings = new DbSettings(); - if (sourceType.getSelectedItem().equals("Delimited text files")) { - dbSettings.sourceType = DbSettings.SourceType.CSV_FILES; - if (sourceDelimiterField.getText().length() == 0) { - JOptionPane.showMessageDialog(frame, "Delimiter field cannot be empty for source database", "Error connecting to server", - JOptionPane.ERROR_MESSAGE); - return null; - } - if (sourceDelimiterField.getText().equalsIgnoreCase("tab")) - dbSettings.delimiter = '\t'; - else - dbSettings.delimiter = sourceDelimiterField.getText().charAt(0); - } else if (sourceType.getSelectedItem().equals("SAS7bdat")) { - dbSettings.sourceType = DbSettings.SourceType.SAS_FILES; - } else { - dbSettings.sourceType = DbSettings.SourceType.DATABASE; - dbSettings.user = sourceUserField.getText(); - dbSettings.password = sourcePasswordField.getText(); - dbSettings.server = sourceServerField.getText(); - dbSettings.database = sourceDatabaseField.getText().trim().length() == 0 ? null : sourceDatabaseField.getText(); - if (sourceType.getSelectedItem().toString().equals("MySQL")) - dbSettings.dbType = DbType.MYSQL; - else if (sourceType.getSelectedItem().toString().equals("Oracle")) - dbSettings.dbType = DbType.ORACLE; - else if (sourceType.getSelectedItem().toString().equals("PostgreSQL")) - dbSettings.dbType = DbType.POSTGRESQL; - else if (sourceType.getSelectedItem().toString().equals("BigQuery")) - dbSettings.dbType = DbType.BIGQUERY; - else if (sourceType.getSelectedItem().toString().equals("Redshift")) - dbSettings.dbType = DbType.REDSHIFT; - else if (sourceType.getSelectedItem().toString().equals("SQL Server")) { - dbSettings.dbType = DbType.MSSQL; - if (sourceUserField.getText().length() != 0) { // Not using windows authentication - String[] parts = sourceUserField.getText().split("/"); - if (parts.length == 2) { - dbSettings.user = parts[1]; - dbSettings.domain = parts[0]; - } - } - } else if (sourceType.getSelectedItem().toString().equals("PDW")) { - dbSettings.dbType = DbType.PDW; - if (sourceUserField.getText().length() != 0) { // Not using windows authentication - String[] parts = sourceUserField.getText().split("/"); - if (parts.length == 2) { - dbSettings.user = parts[1]; - dbSettings.domain = parts[0]; + private static JFileChooser getjFileChooser(DbSettings sourceDbSettings, LocationsPanel locationsPanel) { + JFileChooser fileChooser = new JFileChooser(new File(locationsPanel.getFolderField().getText())); + fileChooser.setName("FileChooser"); + fileChooser.setMultiSelectionEnabled(true); + fileChooser.setFileSelectionMode(JFileChooser.FILES_ONLY); + + if (sourceDbSettings.sourceType == DbSettings.SourceType.CSV_FILES) { + fileChooser.setFileFilter(new FileNameExtensionFilter(DELIMITED_TEXT_FILES, "csv", "txt")); + } else if (sourceDbSettings.sourceType == DbSettings.SourceType.SAS_FILES) { + fileChooser.setFileFilter(new FileNameExtensionFilter("SAS Data Files", DbType.SAS7BDAT.name().toLowerCase())); + } + return fileChooser; + } + + private DbSettings getSourceDbSettings(ValidationFeedback feedback) { + DbType dbChoice = locationsPanel.getCurrentDbChoice(); + DbSettings dbSettings; + if (dbChoice != null && dbChoice.supportsStorageHandler()) { + dbSettings = locationsPanel.getCurrentDbChoice().getStorageHandler().getDbSettings(feedback); + return dbSettings; + } else { + String sourceDelimiterField = locationsPanel.getSourceDelimiterField().getText(); + String sourceType = locationsPanel.getSelectedSourceType(); + dbSettings = new DbSettings(); + if (sourceType.equals(DbType.DELIMITED_TEXT_FILES.label())) { + dbSettings.dbType = DbType.DELIMITED_TEXT_FILES; + dbSettings.sourceType = DbSettings.SourceType.CSV_FILES; + if (sourceDelimiterField.isEmpty()) { + JOptionPane.showMessageDialog(frame, "Delimiter field cannot be empty for source database", "Error connecting to server", + JOptionPane.ERROR_MESSAGE); + return null; + } + if (sourceDelimiterField.equalsIgnoreCase("tab")) + dbSettings.delimiter = '\t'; + else + dbSettings.delimiter = locationsPanel.getSourceDelimiterField().getText().charAt(0); + } else if (sourceType.equalsIgnoreCase(DbType.SAS7BDAT.label())) { + dbSettings.sourceType = DbSettings.SourceType.SAS_FILES; + dbSettings.dbType = DbType.SAS7BDAT; + } else { + dbSettings.sourceType = DbSettings.SourceType.DATABASE; + dbSettings.user = locationsPanel.getSourceUserField(); + dbSettings.password = locationsPanel.getSourcePasswordField(); + dbSettings.server = locationsPanel.getSourceServerField(); + String sourceDatabaseField = locationsPanel.getSourceDatabaseField(); + dbSettings.database = sourceDatabaseField.trim().isEmpty() ? null : sourceDatabaseField; + dbSettings.dbType = dbChoice; + if (dbChoice == DbType.SQL_SERVER) { + if (!dbSettings.user.isEmpty()) { // Not using windows authentication + String[] parts = dbSettings.user.split("/"); + if (parts.length == 2) { + dbSettings.user = parts[1]; + dbSettings.domain = parts[0]; + } + } + } else if (dbChoice == DbType.PDW) { + if (!dbSettings.user.isEmpty()) { // Not using windows authentication + String[] parts = dbSettings.user.split("/"); + if (parts.length == 2) { + dbSettings.user = parts[1]; + dbSettings.domain = parts[0]; + } } - } - } else if (sourceType.getSelectedItem().toString().equals("MS Access")) - dbSettings.dbType = DbType.MSACCESS; - else if (sourceType.getSelectedItem().toString().equals("Teradata")) - dbSettings.dbType = DbType.TERADATA; - else if (sourceType.getSelectedItem().toString().equals("Azure")) { - dbSettings.dbType = DbType.AZURE; - dbSettings.database = ""; - } + } else if (dbChoice == DbType.AZURE) { + dbSettings.database = ""; + } + } + return dbSettings; + } + } + + public void runConnectionTest() { + ValidationFeedback feedback = new ValidationFeedback(); + DbSettings dbSettings = getSourceDbSettings(feedback); + if (dbSettings != null) { + testConnection(dbSettings, feedback); + } else { + throw new DBConfigurationException("Source database settings were not initialized"); } - return dbSettings; } - private void testConnection(DbSettings dbSettings) { + private void testConnection(DbSettings dbSettings, ValidationFeedback feedback) { + if (feedbackBlocksContinuation(feedback)) { + return; + } + String folder = locationsPanel.getFolderField().getText(); if (dbSettings.sourceType == DbSettings.SourceType.CSV_FILES || dbSettings.sourceType == DbSettings.SourceType.SAS_FILES) { - if (new File(folderField.getText()).exists()) { - String message = "Folder " + folderField.getText() + " found"; + if (new File(folder).exists()) { + String message = "Folder " + folder + " found"; JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "Working folder found", JOptionPane.INFORMATION_MESSAGE); } else { - String message = "Folder " + folderField.getText() + " not found"; + String message = "Folder " + folder + " not found"; JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "Working folder not found", JOptionPane.ERROR_MESSAGE); } } else { - if (sourceDatabaseField.isEnabled() && (dbSettings.database == null || dbSettings.database.equals(""))) { + if (locationsPanel.isSourceDatabaseFieldEnabled() && (dbSettings.database == null || dbSettings.database.equals(""))) { JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap("Please specify database name", 80), "Error connecting to server", JOptionPane.ERROR_MESSAGE); return; @@ -893,7 +794,7 @@ private void testConnection(DbSettings dbSettings) { RichConnection connection; try { - connection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType); + connection = new RichConnection(dbSettings); } catch (Exception e) { String message = "Could not connect: " + e.getMessage(); JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "Error connecting to server", JOptionPane.ERROR_MESSAGE); @@ -901,7 +802,7 @@ private void testConnection(DbSettings dbSettings) { } try { List tableNames = connection.getTableNames(dbSettings.database); - if (tableNames.size() == 0) + if (tableNames.isEmpty()) throw new RuntimeException("Unable to retrieve table names for database " + dbSettings.database); } catch (Exception e) { String message = "Could not connect to database: " + e.getMessage(); @@ -910,15 +811,57 @@ private void testConnection(DbSettings dbSettings) { } connection.close(); - String message = "Succesfully connected to " + dbSettings.database + " on server " + dbSettings.server; - JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "Connection succesful", JOptionPane.INFORMATION_MESSAGE); + String message = "Successfully connected to " + dbSettings.database + " on server " + dbSettings.server; + JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), LABEL_CONNECTION_SUCCESSFUL, JOptionPane.INFORMATION_MESSAGE); + } + } + + private boolean feedbackBlocksContinuation(ValidationFeedback feedback) { + if (feedback == null || (!feedback.hasWarnings() && !feedback.hasErrors())) { + return false; + } else { + if (feedback.hasErrors()) { + showFeedback(feedback); + return true; + } + if (feedback.hasWarnings()) { + showFeedback(feedback); + return false; + } + } + return false; + } + private void showFeedback(ValidationFeedback feedback) { + if (feedback == null) { + return; } + String message = ""; + String title = ""; + int messageType = JOptionPane.INFORMATION_MESSAGE; + if (feedback.hasErrors()) { + title = TITLE_ERRORS_IN_DATABASE_CONFIGURATION; + message = createMessage(feedback.getErrors().keySet()); + messageType = JOptionPane.ERROR_MESSAGE; + } else if (feedback.hasWarnings()) { + title = TITLE_WARNINGS_ABOUT_DATABASE_CONFIGURATION; + message = createMessage(feedback.getWarnings().keySet()); + messageType = JOptionPane.WARNING_MESSAGE; + } + JOptionPane.showMessageDialog(ObjectExchange.frame, message, title, messageType); + } + + private static String createMessage(Set messages) { + StringBuilder messageBuilder = new StringBuilder(); + for (String message : messages) { + messageBuilder.append(String.format("%s%n", message)); + } + return messageBuilder.toString(); } private DbSettings getTargetDbSettings() { DbSettings dbSettings = new DbSettings(); - if (targetType.getSelectedItem().equals("Delimited text files")) { + if (targetType.getSelectedItem().equals(DELIMITED_TEXT_FILES)) { dbSettings.sourceType = DbSettings.SourceType.CSV_FILES; switch(targetCSVFormat.getSelectedItem().toString()) { @@ -954,7 +897,7 @@ private DbSettings getTargetDbSettings() { dbSettings.dbType = DbType.POSTGRESQL; break; case "SQL Server": - dbSettings.dbType = DbType.MSSQL; + dbSettings.dbType = DbType.SQL_SERVER; if (targetUserField.getText().length() != 0) { // Not using windows authentication String[] parts = targetUserField.getText().split("/"); if (parts.length == 2) { @@ -986,7 +929,7 @@ private DbSettings getTargetDbSettings() { private void scanRun() { if (tables.size() == 0) { - if (sourceIsFiles || sourceIsSas) { + if (locationsPanel.sourceIsFiles() || locationsPanel.sourceIsSas()) { String message = "No files selected for scanning"; JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "No files selected", JOptionPane.ERROR_MESSAGE); return; @@ -1005,7 +948,7 @@ private void scanRun() { int valuesCount = StringUtilities.numericOptionToInt(scanValuesCount.getSelectedItem().toString()); int numStatsSamplerSize = StringUtilities.numericOptionToInt(numericStatsSampleSize.getSelectedItem().toString()); - ScanThread scanThread = new ScanThread( + ScanRunner scanscanRunner = new ScanRunner( rowCount, valuesCount, scanValueScan.isSelected(), @@ -1013,7 +956,7 @@ private void scanRun() { calculateNumericStats.isSelected(), numStatsSamplerSize ); - scanThread.start(); + scanscanRunner.run(); } private void fakeDataRun() { @@ -1022,16 +965,16 @@ private void fakeDataRun() { String message = "File " + filename + " not found"; JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "File not found", JOptionPane.ERROR_MESSAGE); } else { - FakeDataThread thread = new FakeDataThread(); - thread.start(); + FakeDataRunner runner = new FakeDataRunner(); + runner.run(); } } - private class ScanThread extends Thread { + private class ScanRunner implements Runnable { SourceDataScan sourceDataScan = new SourceDataScan(); - public ScanThread(int maxRows, int maxValues, boolean scanValues, int minCellCount, boolean calculateNumericStats, int numericStatsSampleSize) { + public ScanRunner(int maxRows, int maxValues, boolean scanValues, int minCellCount, boolean calculateNumericStats, int numericStatsSampleSize) { sourceDataScan.setSampleSize(maxRows); sourceDataScan.setScanValues(scanValues); sourceDataScan.setMinCellCount(minCellCount); @@ -1044,14 +987,14 @@ public void run() { for (JComponent component : componentsToDisableWhenRunning) component.setEnabled(false); try { - DbSettings dbSettings = getSourceDbSettings(); + DbSettings dbSettings = getSourceDbSettings(null); if (dbSettings != null) { for (String table : tables) { if (dbSettings.sourceType == DbSettings.SourceType.CSV_FILES || dbSettings.sourceType == DbSettings.SourceType.SAS_FILES) - table = folderField.getText() + "/" + table; + table = locationsPanel.getFolderField().getText() + "/" + table; dbSettings.tables.add(table); } - sourceDataScan.process(dbSettings, folderField.getText() + "/" + SourceDataScan.SCAN_REPORT_FILE_NAME); + sourceDataScan.process(dbSettings, locationsPanel.getFolderField().getText() + "/" + SourceDataScan.SCAN_REPORT_FILE_NAME); } } catch (Exception e) { handleError(e); @@ -1060,10 +1003,9 @@ public void run() { component.setEnabled(true); } } - } - private class FakeDataThread extends Thread { + private class FakeDataRunner implements Runnable { public void run() { for (JComponent component : componentsToDisableWhenRunning) { @@ -1077,7 +1019,7 @@ public void run() { dbSettings, Integer.parseInt(generateRowCount.getValue().toString()), scanReportFileField.getText(), - folderField.getText(), + locationsPanel.getFolderField().getText(), doUniformSampling.isSelected() ); } @@ -1164,11 +1106,11 @@ private void doOpenDocumentation() { } private void handleError(Exception e) { - System.err.println("Error: " + e.getMessage()); - String errorReportFilename = ErrorReport.generate(folderField.getText(), e); + logger.error(e.getMessage(), e); + String errorReportFilename = ErrorReport.generate(locationsPanel.getFolderField().getText(), e); String message = "Error: " + e.getLocalizedMessage(); message += "\nAn error report has been generated:\n" + errorReportFilename; - System.out.println(message); + logger.error(message); JOptionPane.showMessageDialog(frame, StringUtilities.wordWrap(message, 80), "Error", JOptionPane.ERROR_MESSAGE); } @@ -1189,4 +1131,11 @@ private JMenuBar createMenuBar() { return menuBar; } + public JFrame getFrame() { + return frame; + } + + public Console getConsole() { + return console; + } } diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/fakeDataGenerator/FakeDataGenerator.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/fakeDataGenerator/FakeDataGenerator.java similarity index 91% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/fakeDataGenerator/FakeDataGenerator.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/fakeDataGenerator/FakeDataGenerator.java index c35f41f2..3b86b797 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/fakeDataGenerator/FakeDataGenerator.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/fakeDataGenerator/FakeDataGenerator.java @@ -15,10 +15,11 @@ * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit.fakeDataGenerator; +package org.ohdsi.whiterabbit.fakeDataGenerator; import java.util.*; +import org.ohdsi.databases.configuration.DbSettings; import org.ohdsi.databases.RichConnection; import org.ohdsi.rabbitInAHat.dataModel.Database; import org.ohdsi.rabbitInAHat.dataModel.Field; @@ -27,9 +28,11 @@ import org.ohdsi.utilities.StringUtilities; import org.ohdsi.utilities.files.Row; import org.ohdsi.utilities.files.WriteCSVFileWithHeader; -import org.ohdsi.whiteRabbit.DbSettings; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; public class FakeDataGenerator { + static Logger logger = LoggerFactory.getLogger(FakeDataGenerator.class); private RichConnection connection; private int maxRowsPerTable = 1000; @@ -49,16 +52,16 @@ public void generateData(DbSettings dbSettings, int maxRowsPerTable, String file this.doUniformSampling = doUniformSampling; StringUtilities.outputWithTime("Starting creation of fake data"); - System.out.println("Loading scan report from " + filename); + logger.info("Loading scan report from {}", filename); Database database = Database.generateModelFromScanReport(filename); if (targetType == DbSettings.SourceType.DATABASE) { - connection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType); + connection = new RichConnection(dbSettings); connection.use(dbSettings.database); for (Table table : database.getTables()) { if (table.getName().toLowerCase().endsWith(".csv")) table.setName(table.getName().substring(0, table.getName().length() - 4)); - System.out.println("Generating table " + table.getName()); + logger.info("Generating table {}", table.getName()); createTable(table); connection.insertIntoTable(generateRows(table).iterator(), table.getName(), false); } @@ -68,7 +71,7 @@ public void generateData(DbSettings dbSettings, int maxRowsPerTable, String file String name = folder + "/" + table.getName(); if (!name.toLowerCase().endsWith(".csv")) name = name + ".csv"; - System.out.println("Generating table " + name); + logger.info("Generating table {}", name); WriteCSVFileWithHeader out = new WriteCSVFileWithHeader(name, dbSettings.csvFormat); for (Row row : generateRows(table)) out.write(row); diff --git a/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/LocationsPanel.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/LocationsPanel.java new file mode 100644 index 00000000..ad5b3cbf --- /dev/null +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/LocationsPanel.java @@ -0,0 +1,354 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.gui; + +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.databases.configuration.DBConfiguration; +import org.ohdsi.whiterabbit.PanelsManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.swing.*; +import javax.swing.event.DocumentEvent; +import javax.swing.event.DocumentListener; +import java.awt.*; +import java.awt.event.ItemEvent; +import java.io.File; +import java.util.Objects; + +import static org.ohdsi.whiterabbit.WhiteRabbitMain.LABEL_TEST_CONNECTION; + +public class LocationsPanel extends JPanel { + + static Logger logger = LoggerFactory.getLogger(LocationsPanel.class); + + public static final String LABEL_LOCATIONS = "Locations"; + public static final String LABEL_SERVER_LOCATION = "Server location"; + public static final String NAME_SERVER_LOCATION = "ServerLocation"; + public static final String LABEL_USER_NAME = "User name"; + public static final String NAME_USER_NAME = "UserName"; + public static final String LABEL_PASSWORD = "Password"; + public static final String NAME_PASSWORD = "PasswordName"; + public static final String LABEL_DATABASE_NAME = "Database name"; + public static final String NAME_DATABASE_NAME = "DatabaseName"; + public static final String LABEL_DELIMITER = "Delimiter"; + public static final String NAME_DELIMITER = "DelimiterName"; + + public static final String TOOLTIP_POSTGRESQL_SERVER = "For PostgreSQL servers this field contains the host name and database name (/)"; + + private final JFrame parentFrame; + private JTextField folderField; + private JComboBox sourceType; + private JTextField sourceDelimiterField; + private JTextField sourceServerField; + private JTextField sourceUserField; + private JTextField sourcePasswordField; + private JTextField sourceDatabaseField; + private DbType currentDbType = null; + + + private SourcePanel sourcePanel; + private boolean sourceIsFiles = true; + private boolean sourceIsSas = false; + + private final transient PanelsManager panelsManager; + + public LocationsPanel(JFrame parentFrame, PanelsManager panelsManager) { + super(); + this.parentFrame = parentFrame; + this.panelsManager = panelsManager; + this.createLocationsPanel(); + } + + private void createLocationsPanel() { + JPanel panel = this; + panel.setName(LABEL_LOCATIONS); + + panel.setLayout(new GridBagLayout()); + GridBagConstraints c = new GridBagConstraints(); + c.fill = GridBagConstraints.BOTH; + c.weightx = 0.5; + c.weighty = 0.8; + + JPanel folderPanel = new JPanel(); + folderPanel.setLayout(new BoxLayout(folderPanel, BoxLayout.X_AXIS)); + folderPanel.setBorder(BorderFactory.createTitledBorder("Working folder")); + folderField = new JTextField(); + folderField.setName("FolderField"); + folderField.setText((new File("").getAbsolutePath())); + folderField.setToolTipText("The folder where all output will be written"); + folderPanel.add(folderField); + JButton pickButton = new JButton("Pick folder"); + pickButton.setToolTipText("Pick a different working folder"); + folderPanel.add(pickButton); + pickButton.addActionListener(e -> pickFolder()); + panelsManager.getComponentsToDisableWhenRunning().add(pickButton); + c.gridx = 0; + c.gridy = 0; + c.gridwidth = 1; + panel.add(folderPanel, c); + + + c.gridx = 0; + c.gridy = 1; + c.gridwidth = 1; + this.sourcePanel = createSourcePanel(); + + // make sure the sourcePanel has usable content by default + createDatabaseFields(DbType.DELIMITED_TEXT_FILES.label()); + sourceType.setSelectedItem(DbType.DELIMITED_TEXT_FILES.label()); + + panel.add(this.sourcePanel, c); + + JPanel testConnectionButtonPanel = new JPanel(); + testConnectionButtonPanel.setLayout(new BoxLayout(testConnectionButtonPanel, BoxLayout.X_AXIS)); + testConnectionButtonPanel.add(Box.createHorizontalGlue()); + + JButton testConnectionButton = new JButton(LABEL_TEST_CONNECTION); + testConnectionButton.setName(LABEL_TEST_CONNECTION); + testConnectionButton.setBackground(new Color(151, 220, 141)); + testConnectionButton.setToolTipText("Test the connection"); + testConnectionButton.addActionListener(e -> this.runConnectionTest()); + panelsManager.getComponentsToDisableWhenRunning().add(testConnectionButton); + testConnectionButtonPanel.add(testConnectionButton); + + c.gridx = 0; + c.gridy = 2; + c.gridwidth = 1; + panel.add(testConnectionButtonPanel, c); + } + + private void runConnectionTest() { + panelsManager.runConnectionTest(); + + } + + private void createDatabaseFields(ItemEvent itemEvent) { + String selectedSourceType = itemEvent.getItem().toString(); + + // remove existing DB related fields in sourcePanel + sourcePanel.clear(); + + currentDbType = DbType.getDbType(selectedSourceType); + if (currentDbType.supportsStorageHandler()) { + createDatabaseFields(); + } else { + createDatabaseFields(selectedSourceType); + } + if (panelsManager.getAddAllButton() != null) { + panelsManager.getAddAllButton().setEnabled(sourceIsDatabase(selectedSourceType)); + } + this.revalidate(); + } + + @FunctionalInterface + public interface SimpleDocumentListener extends DocumentListener { + void update(DocumentEvent e); + + @Override + default void insertUpdate(DocumentEvent e) { + update(e); + } + @Override + default void removeUpdate(DocumentEvent e) { + update(e); + } + @Override + default void changedUpdate(DocumentEvent e) { + update(e); + } + } + + private void createDatabaseFields() { + DBConfiguration currentConfiguration = this.currentDbType.getStorageHandler().getDBConfiguration(); + currentConfiguration.getFields().forEach(f -> { + sourcePanel.addReplacable(new JLabel(f.label)); + JTextField field = new JTextField(f.getValueOrDefault()); + field.setName(f.name); + field.setToolTipText(f.toolTip); + sourcePanel.addReplacable(field); + field.setEnabled(true); + field.getDocument().addDocumentListener((SimpleDocumentListener) e -> { + f.setValue(field.getText()); + }); + }); + } + + private boolean sourceIsFiles(String sourceType) { + return sourceType.equalsIgnoreCase(DbType.DELIMITED_TEXT_FILES.label()); + } + + private boolean sourceIsSas(String sourceType) { + return sourceType.equalsIgnoreCase(DbType.SAS7BDAT.label()); + } + + private boolean sourceIsDatabase(String sourceType) { + return (!sourceIsFiles(sourceType) && !sourceIsSas(sourceType)); + } + + private void createDatabaseFields(String selectedSourceType) { + sourceIsFiles = sourceIsFiles(selectedSourceType); + sourceIsSas = sourceIsSas(selectedSourceType); + boolean sourceIsDatabase = sourceIsDatabase(selectedSourceType); + + sourcePanel.addReplacable(new JLabel(LABEL_SERVER_LOCATION)); + sourceServerField = new JTextField("127.0.0.1"); + sourceServerField.setName(LABEL_SERVER_LOCATION); + sourceServerField.setEnabled(false); + sourcePanel.addReplacable(sourceServerField); + sourcePanel.addReplacable(new JLabel(LABEL_USER_NAME)); + sourceUserField = new JTextField(""); + sourceUserField.setName(LABEL_USER_NAME); + sourceUserField.setEnabled(false); + sourcePanel.addReplacable(sourceUserField); + sourcePanel.addReplacable(new JLabel(LABEL_PASSWORD)); + sourcePasswordField = new JPasswordField(""); + sourcePasswordField.setName(LABEL_PASSWORD); + sourcePasswordField.setEnabled(false); + sourcePanel.addReplacable(sourcePasswordField); + sourcePanel.addReplacable(new JLabel(LABEL_DATABASE_NAME)); + sourceDatabaseField = new JTextField(""); + sourceDatabaseField.setName(LABEL_DATABASE_NAME); + sourceDatabaseField.setEnabled(false); + sourcePanel.addReplacable(sourceDatabaseField); + + sourcePanel.addReplacable(new JLabel(LABEL_DELIMITER)); + JTextField delimiterField = new JTextField(","); + delimiterField.setName(NAME_DELIMITER); + sourceDelimiterField = delimiterField; + sourceDelimiterField.setToolTipText("The delimiter that separates values. Enter 'tab' for tab."); + sourcePanel.addReplacable(sourceDelimiterField); + sourceServerField.setEnabled(sourceIsDatabase); + sourceUserField.setEnabled(sourceIsDatabase); + sourcePasswordField.setEnabled(sourceIsDatabase); + sourceDatabaseField.setEnabled(sourceIsDatabase && !selectedSourceType.equals(DbType.AZURE.label())); + sourceDelimiterField.setEnabled(sourceIsFiles); + + if (sourceIsDatabase) { + if (selectedSourceType.equals(DbType.ORACLE.label())) { + sourceServerField.setToolTipText("For Oracle servers this field contains the SID, servicename, and optionally the port: '/', ':/', '/', or ':/'"); + sourceUserField.setToolTipText("For Oracle servers this field contains the name of the user used to log in"); + sourcePasswordField.setToolTipText("For Oracle servers this field contains the password corresponding to the user"); + sourceDatabaseField.setToolTipText("For Oracle servers this field contains the schema (i.e. 'user' in Oracle terms) containing the source tables"); + } else if (selectedSourceType.equals(DbType.POSTGRESQL.label())) { + sourceServerField.setToolTipText(TOOLTIP_POSTGRESQL_SERVER); + sourceUserField.setToolTipText("The user used to log in to the server"); + sourcePasswordField.setToolTipText("The password used to log in to the server"); + sourceDatabaseField.setToolTipText("For PostgreSQL servers this field contains the schema containing the source tables"); + } else if (selectedSourceType.equals(DbType.BIGQUERY.label())) { + sourceServerField.setToolTipText("GBQ SA & UA: ProjectID"); + sourceUserField.setToolTipText("GBQ SA only: OAuthServiceAccountEMAIL"); + sourcePasswordField.setToolTipText("GBQ SA only: OAuthPvtKeyPath"); + sourceDatabaseField.setToolTipText("GBQ SA & UA: Data Set within ProjectID"); + } else { + if (selectedSourceType.equals(DbType.AZURE.label())) { + sourceServerField.setToolTipText("For Azure, this field contains the host name and database name (;database=)"); + } else { + sourceServerField.setToolTipText("This field contains the name or IP address of the database server"); + } + if (selectedSourceType.equals(DbType.SQL_SERVER.label())) { + sourceUserField.setToolTipText("The user used to log in to the server. Optionally, the domain can be specified as / (e.g. 'MyDomain/Joe')"); + } else { + sourceUserField.setToolTipText("The user used to log in to the server"); + } + sourcePasswordField.setToolTipText("The password used to log in to the server"); + if (selectedSourceType.equals(DbType.AZURE.label())) { + sourceDatabaseField.setToolTipText("For Azure, leave this empty"); + } else { + sourceDatabaseField.setToolTipText("The name of the database containing the source tables"); + } + } + } + } + + private SourcePanel createSourcePanel() { + SourcePanel sourcePanel = new SourcePanel(); + sourcePanel.setLayout(new GridLayout(0, 2)); + sourcePanel.setBorder(BorderFactory.createTitledBorder("Source data location")); + sourcePanel.add(new JLabel("Data type")); + sourceType = new JComboBox<>(DbType.pickList()); + sourceType.setName("SourceType"); + sourceType.setToolTipText("Select the type of source data available"); + sourceType.addItemListener(event -> { + if (event.getStateChange() == ItemEvent.SELECTED) { + createDatabaseFields(event); + } + }); + sourcePanel.add(sourceType); + + return sourcePanel; + } + + public JTextField getFolderField() { + return folderField; + } + + public String getSelectedSourceType() { + return Objects.requireNonNull(sourceType.getSelectedItem()).toString(); + } + + public JTextField getSourceDelimiterField() { + return sourceDelimiterField; + } + + public boolean sourceIsFiles() { + return sourceIsFiles; + } + + public boolean sourceIsSas() { + return sourceIsSas; + } + + public String getSourceServerField() { + return sourceServerField.getText(); + } + + public String getSourceUserField() { + return sourceUserField.getText(); + } + + public String getSourcePasswordField() { + return sourcePasswordField.getText(); + } + public String getSourceDatabaseField() { + return sourceDatabaseField.getText(); + } + + public boolean isSourceDatabaseFieldEnabled() { + return sourceDatabaseField.isEnabled(); + } + + public DbType getCurrentDbChoice() { + return this.currentDbType; + } + + private void pickFolder() { + JFileChooser fileChooser = new JFileChooser(new File(folderField.getText())); + fileChooser.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY); + int returnVal = fileChooser.showDialog(parentFrame, "Select folder"); + if (returnVal == JFileChooser.APPROVE_OPTION) { + File selectedDirectory = fileChooser.getSelectedFile(); + if (!selectedDirectory.exists()) { + // When no directory is selected when approving, FileChooser incorrectly appends the current directory to the path. + // Take the opened directory instead. + selectedDirectory = fileChooser.getCurrentDirectory(); + } + folderField.setText(selectedDirectory.getAbsolutePath()); + } + } +} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/SourcePanel.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/SourcePanel.java new file mode 100644 index 00000000..4dd565b5 --- /dev/null +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/gui/SourcePanel.java @@ -0,0 +1,39 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.gui; + +import javax.swing.*; +import java.util.*; + +public class SourcePanel extends JPanel { + private List clearableComponents = new ArrayList<>(); + + public void addReplacable(JComponent component) { + + this.add(component); + clearableComponents.add(component); + } + + public void clear() { + // remove the components in the reverse order of how they were added, keeps the layout of the JPanel intact + Collections.reverse(clearableComponents); + clearableComponents.forEach(this::remove); + clearableComponents.clear(); + } + +} diff --git a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/SourceDataScan.java b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/scan/SourceDataScan.java similarity index 59% rename from whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/SourceDataScan.java rename to whiterabbit/src/main/java/org/ohdsi/whiterabbit/scan/SourceDataScan.java index 38e64ba3..11e88715 100644 --- a/whiterabbit/src/main/java/org/ohdsi/whiteRabbit/scan/SourceDataScan.java +++ b/whiterabbit/src/main/java/org/ohdsi/whiterabbit/scan/SourceDataScan.java @@ -1,30 +1,26 @@ /******************************************************************************* * Copyright 2019 Observational Health Data Sciences and Informatics - * + *

* This file is part of WhiteRabbit - * + *

* Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at - * + *

* http://www.apache.org/licenses/LICENSE-2.0 - * + *

* Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. ******************************************************************************/ -package org.ohdsi.whiteRabbit.scan; +package org.ohdsi.whiterabbit.scan; import java.io.*; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.rmi.RemoteException; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.time.LocalDate; import java.time.LocalDateTime; import java.util.*; import java.util.stream.Collectors; @@ -38,34 +34,30 @@ import org.apache.poi.ss.usermodel.CellStyle; import org.apache.poi.ss.usermodel.Row; import org.apache.poi.ss.usermodel.Sheet; -import org.apache.poi.xssf.streaming.SXSSFWorkbook; import org.apache.commons.io.FileUtils; -import org.ohdsi.databases.DbType; +import org.apache.poi.xssf.usermodel.XSSFWorkbook; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; import org.ohdsi.databases.RichConnection; -import org.ohdsi.databases.RichConnection.QueryResult; +import org.ohdsi.databases.QueryResult; +import org.ohdsi.databases.*; import org.ohdsi.rabbitInAHat.dataModel.Table; import org.ohdsi.utilities.*; -import org.ohdsi.utilities.collections.CountingSet; -import org.ohdsi.utilities.collections.CountingSet.Count; import org.ohdsi.utilities.collections.Pair; import org.ohdsi.utilities.files.ReadTextFile; -import org.ohdsi.whiteRabbit.DbSettings; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import static java.lang.Long.max; -public class SourceDataScan { - - public static int MAX_VALUES_IN_MEMORY = 100000; - public static int MIN_CELL_COUNT_FOR_CSV = 1000000; - public static int N_FOR_FREE_TEXT_CHECK = 1000; - public static int MIN_AVERAGE_LENGTH_FOR_FREE_TEXT = 100; - +public class SourceDataScan implements ScanParameters { + static Logger logger = LoggerFactory.getLogger(SourceDataScan.class); public final static String SCAN_REPORT_FILE_NAME = "ScanReport.xlsx"; public static final String POI_TMP_DIR_ENVIRONMENT_VARIABLE_NAME = "ORG_OHDSI_WHITERABBIT_POI_TMPDIR"; public static final String POI_TMP_DIR_PROPERTY_NAME = "org.ohdsi.whiterabbit.poi.tmpdir"; - private SXSSFWorkbook workbook; + private XSSFWorkbook workbook; private char delimiter = ','; private int sampleSize; private boolean scanValues = false; @@ -75,7 +67,6 @@ public class SourceDataScan { private int maxValues; private DbSettings.SourceType sourceType; private DbType dbType; - private String database; private Map> tableToFieldInfos; private Map indexedTableNameLookup; @@ -96,6 +87,22 @@ public void setSampleSize(int sampleSize) { this.sampleSize = sampleSize; } + public boolean doCalculateNumericStats() { + return calculateNumericStats; + } + + public int getMaxValues() { + return maxValues; + } + + public boolean doScanValues() { + return scanValues; + } + + public int getNumStatsSamplerSize() { + return numStatsSamplerSize; + } + public void setScanValues(boolean scanValues) { this.scanValues = scanValues; } @@ -104,6 +111,14 @@ public void setMinCellCount(int minCellCount) { this.minCellCount = minCellCount; } + public int getMinCellCount() { + return minCellCount; + } + + public int getSampleSize() { + return sampleSize; + } + public void setMaxValues(int maxValues) { this.maxValues = maxValues; } @@ -116,11 +131,10 @@ public void setNumStatsSamplerSize(int numStatsSamplerSize) { this.numStatsSamplerSize = numStatsSamplerSize; } - public void process(DbSettings dbSettings, String outputFileName) { + public void process(DbSettings dbSettings, String outputFileName) throws IOException { startTimeStamp = LocalDateTime.now(); sourceType = dbSettings.sourceType; dbType = dbSettings.dbType; - database = dbSettings.database; tableToFieldInfos = new HashMap<>(); StringUtilities.outputWithTime("Started new scan of " + dbSettings.tables.size() + " tables..."); @@ -181,7 +195,7 @@ private static Path setupTmpDir(Path tmpDir) { private static void checkWritableTmpDir(String dir) { if (isNotWritable(Paths.get(dir))) { String message = String.format("Directory %s is not writable! (used for tmp files for Apache POI)", dir); - System.out.println(message); + logger.warn(message); throw new RuntimeException(message); } } @@ -214,15 +228,14 @@ private void processDatabase(DbSettings dbSettings) { if (dbSettings.dbType == DbType.BIGQUERY) { dbSettings.domain = dbSettings.database; } - - try (RichConnection connection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { + try (RichConnection connection = new RichConnection(dbSettings)) { connection.setVerbose(false); connection.use(dbSettings.database); tableToFieldInfos = dbSettings.tables.stream() .collect(Collectors.toMap( Table::new, - table -> processDatabaseTable(table, connection) + table -> processDatabaseTable(table, connection, dbSettings.database) )); } } @@ -257,11 +270,11 @@ private void processSasFiles(DbSettings dbSettings) { } } - private void generateReport(String filename) { + private void generateReport(String filename) throws IOException { StringUtilities.outputWithTime("Generating scan report"); removeEmptyTables(); - workbook = new SXSSFWorkbook(100); // keep 100 rows in memory, exceeding rows will be flushed to disk + workbook = new XSSFWorkbook(); int i = 0; indexedTableNameLookup = new HashMap<>(); @@ -465,8 +478,7 @@ private void createMetaSheet() { addRow(metaSheet, "N_FOR_FREE_TEXT_CHECK", SourceDataScan.N_FOR_FREE_TEXT_CHECK); addRow(metaSheet, "MIN_AVERAGE_LENGTH_FOR_FREE_TEXT", SourceDataScan.MIN_AVERAGE_LENGTH_FOR_FREE_TEXT); addRow(metaSheet, "sourceType", this.sourceType.toString()); - addRow(metaSheet, "dbType", this.dbType != null ? this.dbType.getTypeName() : ""); -// addRow(metaSheet, "database", this.database); + addRow(metaSheet, "dbType", this.dbType != null ? this.dbType.name() : ""); addRow(metaSheet, "delimiter", this.delimiter); addRow(metaSheet, "sampleSize", this.sampleSize); addRow(metaSheet, "scanValues", this.scanValues); @@ -479,33 +491,38 @@ private void createMetaSheet() { private void removeEmptyTables() { tableToFieldInfos.entrySet() - .removeIf(stringListEntry -> stringListEntry.getValue().size() == 0); + .removeIf(stringListEntry -> stringListEntry.getValue().isEmpty()); } - private List processDatabaseTable(String table, RichConnection connection) { + private List processDatabaseTable(String table, RichConnection connection, String database) { StringUtilities.outputWithTime("Scanning table " + table); - long rowCount = connection.getTableSize(table); - List fieldInfos = fetchTableStructure(connection, table); + long rowCount; + if (connection.getConnection().hasStorageHandler()) { + rowCount = connection.getConnection().getStorageHandler().getTableSize(table); + } else { + rowCount = connection.getTableSize(table); + } + List fieldInfos = connection.fetchTableStructure(connection, database, table, this); if (scanValues) { int actualCount = 0; QueryResult queryResult = null; try { - queryResult = fetchRowsFromTable(connection, table, rowCount); + queryResult = connection.fetchRowsFromTable(table, rowCount, this); for (org.ohdsi.utilities.files.Row row : queryResult) { for (FieldInfo fieldInfo : fieldInfos) { fieldInfo.processValue(row.get(fieldInfo.name)); } actualCount++; if (sampleSize != -1 && actualCount >= sampleSize) { - System.out.println("Stopped after " + actualCount + " rows"); + logger.info("Stopped after {} rows", actualCount); break; } } for (FieldInfo fieldInfo : fieldInfos) fieldInfo.trim(); } catch (Exception e) { - System.out.println("Error: " + e.getMessage()); + logger.error(e.getMessage(), e); } finally { if (queryResult != null) { queryResult.close(); @@ -516,105 +533,6 @@ private List processDatabaseTable(String table, RichConnection connec return fieldInfos; } - private QueryResult fetchRowsFromTable(RichConnection connection, String table, long rowCount) { - String query = null; - - if (sampleSize == -1) { - if (dbType == DbType.MSACCESS) - query = "SELECT * FROM [" + table + "]"; - else if (dbType == DbType.MSSQL || dbType == DbType.PDW || dbType == DbType.AZURE) - query = "SELECT * FROM [" + table.replaceAll("\\.", "].[") + "]"; - else - query = "SELECT * FROM " + table; - } else { - if (dbType == DbType.MSSQL || dbType == DbType.AZURE) - query = "SELECT * FROM [" + table.replaceAll("\\.", "].[") + "] TABLESAMPLE (" + sampleSize + " ROWS)"; - else if (dbType == DbType.MYSQL) - query = "SELECT * FROM " + table + " ORDER BY RAND() LIMIT " + sampleSize; - else if (dbType == DbType.PDW) - query = "SELECT TOP " + sampleSize + " * FROM [" + table.replaceAll("\\.", "].[") + "] ORDER BY RAND()"; - else if (dbType == DbType.ORACLE) { - if (sampleSize < rowCount) { - double percentage = 100 * sampleSize / (double) rowCount; - if (percentage < 100) - query = "SELECT * FROM " + table + " SAMPLE(" + percentage + ")"; - } else { - query = "SELECT * FROM " + table; - } - } else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) - query = "SELECT * FROM " + table + " ORDER BY RANDOM() LIMIT " + sampleSize; - else if (dbType == DbType.MSACCESS) - query = "SELECT " + "TOP " + sampleSize + " * FROM [" + table + "]"; - else if (dbType == DbType.BIGQUERY) - query = "SELECT * FROM " + table + " ORDER BY RAND() LIMIT " + sampleSize; - } - // System.out.println("SQL: " + query); - return connection.query(query); - - } - - private List fetchTableStructure(RichConnection connection, String table) { - List fieldInfos = new ArrayList<>(); - - if (dbType == DbType.MSACCESS) { - ResultSet rs = connection.getMsAccessFieldNames(table); - try { - while (rs.next()) { - FieldInfo fieldInfo = new FieldInfo(rs.getString("COLUMN_NAME")); - fieldInfo.type = rs.getString("TYPE_NAME"); - fieldInfo.rowCount = connection.getTableSize(table); - fieldInfos.add(fieldInfo); - } - } catch (SQLException e) { - throw new RuntimeException(e.getMessage()); - } - } else { - String query = null; - if (dbType == DbType.ORACLE) - query = "SELECT COLUMN_NAME,DATA_TYPE FROM ALL_TAB_COLUMNS WHERE table_name = '" + table + "' AND owner = '" + database.toUpperCase() + "'"; - else if (dbType == DbType.MSSQL || dbType == DbType.PDW) { - String trimmedDatabase = database; - if (database.startsWith("[") && database.endsWith("]")) - trimmedDatabase = database.substring(1, database.length() - 1); - String[] parts = table.split("\\."); - query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG='" + trimmedDatabase + "' AND TABLE_SCHEMA='" + parts[0] + - "' AND TABLE_NAME='" + parts[1] + "';"; - } else if (dbType == DbType.AZURE) { - String[] parts = table.split("\\."); - query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='" + parts[0] + - "' AND TABLE_NAME='" + parts[1] + "';"; - } else if (dbType == DbType.MYSQL) - query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '" + database + "' AND TABLE_NAME = '" + table - + "';"; - else if (dbType == DbType.POSTGRESQL || dbType == DbType.REDSHIFT) - query = "SELECT COLUMN_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '" + database.toLowerCase() + "' AND TABLE_NAME = '" - + table.toLowerCase() + "' ORDER BY ordinal_position;"; - else if (dbType == DbType.TERADATA) { - query = "SELECT ColumnName, ColumnType FROM dbc.columns WHERE DatabaseName= '" + database.toLowerCase() + "' AND TableName = '" - + table.toLowerCase() + "';"; - } else if (dbType == DbType.BIGQUERY) { - query = "SELECT column_name AS COLUMN_NAME, data_type as DATA_TYPE FROM " + database + ".INFORMATION_SCHEMA.COLUMNS WHERE table_name = \"" + table + "\";"; - } - - for (org.ohdsi.utilities.files.Row row : connection.query(query)) { - row.upperCaseFieldNames(); - FieldInfo fieldInfo; - if (dbType == DbType.TERADATA) { - fieldInfo = new FieldInfo(row.get("COLUMNNAME")); - } else { - fieldInfo = new FieldInfo(row.get("COLUMN_NAME")); - } - if (dbType == DbType.TERADATA) { - fieldInfo.type = row.get("COLUMNTYPE"); - } else { - fieldInfo.type = row.get("DATA_TYPE"); - } - fieldInfo.rowCount = connection.getTableSize(table); - fieldInfos.add(fieldInfo); - } - } - return fieldInfos; - } private List processCsvFile(String filename) { StringUtilities.outputWithTime("Scanning table " + filename); @@ -633,7 +551,7 @@ private List processCsvFile(String filename) { if (lineNr == 1) { for (String cell : row) { - fieldInfos.add(new FieldInfo(cell)); + fieldInfos.add(new FieldInfo(this, cell)); } if (!scanValues) { @@ -660,7 +578,7 @@ private List processSasFile(SasFileReader sasFileReader) throws IOExc SasFileProperties sasFileProperties = sasFileReader.getSasFileProperties(); for (Column column : sasFileReader.getColumns()) { - FieldInfo fieldInfo = new FieldInfo(column.getName()); + FieldInfo fieldInfo = new FieldInfo(this, column.getName()); fieldInfo.label = column.getLabel(); fieldInfo.rowCount = sasFileProperties.getRowCount(); if (!scanValues) { @@ -698,229 +616,6 @@ private List processSasFile(SasFileReader sasFileReader) throws IOExc return fieldInfos; } - private class FieldInfo { - public String type; - public String name; - public String label; - public CountingSet valueCounts = new CountingSet<>(); - public long sumLength = 0; - public int maxLength = 0; - public long nProcessed = 0; - public long emptyCount = 0; - public long uniqueCount = 0; - public long rowCount = -1; - public boolean isInteger = true; - public boolean isReal = true; - public boolean isDate = true; - public boolean isFreeText = false; - public boolean tooManyValues = false; - public UniformSamplingReservoir samplingReservoir; - public Object average; - public Object stdev; - public Object minimum; - public Object maximum; - public Object q1; - public Object q2; - public Object q3; - - public FieldInfo(String name) { - this.name = name; - if (calculateNumericStats) { - this.samplingReservoir = new UniformSamplingReservoir(numStatsSamplerSize); - } - } - - public void trim() { - // Only keep values that are used in scan report - if (valueCounts.size() > maxValues) { - valueCounts.keepTopN(maxValues); - } - - // Calculate numeric stats and dereference sampling reservoir to save memory. - if (calculateNumericStats) { - average = getAverage(); - stdev = getStandardDeviation(); - minimum = getMinimum(); - maximum = getMaximum(); - q1 = getQ1(); - q2 = getQ2(); - q3 = getQ3(); - } - samplingReservoir = null; - } - - public boolean hasValuesTrimmed() { - return tooManyValues; - } - - public Double getFractionEmpty() { - if (nProcessed == 0) - return 1d; - else - return emptyCount / (double) nProcessed; - } - - public String getTypeDescription() { - if (type != null) - return type; - else if (!scanValues) // If not type assigned and not values scanned, do not derive - return ""; - else if (nProcessed == emptyCount) - return DataType.EMPTY.name(); - else if (isFreeText) - return DataType.TEXT.name(); - else if (isDate) - return DataType.DATE.name(); - else if (isInteger) - return DataType.INT.name(); - else if (isReal) - return DataType.REAL.name(); - else - return DataType.VARCHAR.name(); - } - - public Double getFractionUnique() { - if (nProcessed == 0 || uniqueCount == 1) { - return 0d; - } else { - return uniqueCount / (double) nProcessed; - } - - } - - public void processValue(String value) { - nProcessed++; - sumLength += value.length(); - if (value.length() > maxLength) - maxLength = value.length(); - - String trimValue = value.trim(); - if (trimValue.length() == 0) - emptyCount++; - - if (!isFreeText) { - boolean newlyAdded = valueCounts.add(value); - if (newlyAdded) uniqueCount++; - - if (trimValue.length() != 0) { - evaluateDataType(trimValue); - } - - if (nProcessed == N_FOR_FREE_TEXT_CHECK && !isInteger && !isReal && !isDate) { - doFreeTextCheck(); - } - } else { - valueCounts.addAll(StringUtilities.mapToWords(trimValue.toLowerCase())); - } - - // if over this large constant number, then trimmed back to size used in report (maxValues). - if (!tooManyValues && valueCounts.size() > MAX_VALUES_IN_MEMORY) { - tooManyValues = true; - this.trim(); - } - - if (calculateNumericStats && !trimValue.isEmpty()) { - if (isInteger || isReal) { - samplingReservoir.add(Double.parseDouble(trimValue)); - } else if (isDate) { - samplingReservoir.add(DateUtilities.parseDate(trimValue)); - } - } - } - - public List> getSortedValuesWithoutSmallValues() { - List> result = valueCounts.key2count.entrySet().stream() - .filter(e -> e.getValue().count >= minCellCount) - .sorted(Comparator.>comparingInt(e -> e.getValue().count).reversed()) - .limit(maxValues) - .map(e -> new Pair<>(e.getKey(), e.getValue().count)) - .collect(Collectors.toCollection(ArrayList::new)); - - if (result.size() < valueCounts.key2count.size()) { - result.add(new Pair<>("List truncated...", -1)); - } - return result; - } - - private void evaluateDataType(String value) { - if (isReal && !StringUtilities.isNumber(value)) - isReal = false; - if (isInteger && !StringUtilities.isLong(value)) - isInteger = false; - if (isDate && !StringUtilities.isDate(value)) - isDate = false; - } - - private void doFreeTextCheck() { - double averageLength = sumLength / (double) (nProcessed - emptyCount); - if (averageLength >= MIN_AVERAGE_LENGTH_FOR_FREE_TEXT) { - isFreeText = true; - // Reset value count to word count - CountingSet wordCounts = new CountingSet<>(); - for (Map.Entry entry : valueCounts.key2count.entrySet()) - for (String word : StringUtilities.mapToWords(entry.getKey().toLowerCase())) - wordCounts.add(word, entry.getValue().count); - valueCounts = wordCounts; - } - } - - private Object formatNumericValue(double value) { - return formatNumericValue(value, false); - } - - private Object formatNumericValue(double value, boolean dateAsDays) { - if (nProcessed == 0) { - return Double.NaN; - } else if (getTypeDescription().equals(DataType.EMPTY.name())) { - return Double.NaN; - } else if (isInteger || isReal) { - return value; - } else if (isDate && dateAsDays) { - return value; - } else if (isDate) { - return LocalDate.ofEpochDay((long) value).toString(); - } else { - return Double.NaN; - } - } - - private Object getMinimum() { - double min = samplingReservoir.getPopulationMinimum(); - return formatNumericValue(min); - } - - private Object getMaximum() { - double max = samplingReservoir.getPopulationMaximum(); - return formatNumericValue(max); - } - - private Object getAverage() { - double average = samplingReservoir.getPopulationMean(); - return formatNumericValue(average); - } - - private Object getStandardDeviation() { - double stddev = samplingReservoir.getSampleStandardDeviation(); - return formatNumericValue(stddev, true); - } - - private Object getQ1() { - double q1 = samplingReservoir.getSampleQuartiles().get(0); - return formatNumericValue(q1); - } - - private Object getQ2() { - double q2 = samplingReservoir.getSampleQuartiles().get(1); - return formatNumericValue(q2); - } - - private Object getQ3() { - double q3 = samplingReservoir.getSampleQuartiles().get(2); - return formatNumericValue(q3); - } - - } - private Row addRow(Sheet sheet, Object... values) { Row row = sheet.createRow(sheet.getPhysicalNumberOfRows()); for (Object value : values) { diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit.ico b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit.ico similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit.ico rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit.ico diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit128.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit128.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit128.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit128.png diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit16.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit16.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit16.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit16.png diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit256.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit256.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit256.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit256.png diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit32.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit32.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit32.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit32.png diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit48.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit48.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit48.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit48.png diff --git a/whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit64.png b/whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit64.png similarity index 100% rename from whiterabbit/src/main/resources/org/ohdsi/whiteRabbit/WhiteRabbit64.png rename to whiterabbit/src/main/resources/org/ohdsi/whiterabbit/WhiteRabbit64.png diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/GUITestExtension.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/GUITestExtension.java new file mode 100644 index 00000000..12553f70 --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/GUITestExtension.java @@ -0,0 +1,66 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import org.assertj.swing.junit.runner.FailureScreenshotTaker; +import org.assertj.swing.junit.runner.ImageFolderCreator; +import org.junit.jupiter.api.extension.Extension; +import org.junit.jupiter.api.extension.ExtensionContext; +import org.junit.jupiter.api.extension.InvocationInterceptor; +import org.junit.jupiter.api.extension.ReflectiveInvocationContext; + +import java.lang.reflect.Method; + +import static org.assertj.swing.annotation.GUITestFinder.isGUITest; +import static org.assertj.swing.junit.runner.Formatter.testNameFrom; + +/** + * Understands a JUnit 5 extension that takes a screenshot of a failed GUI test. + * The Junit 4 runner is available in {@link org.assertj.swing.junit.runner.GUITestRunner}. + * + * @see assertj-swing #259 + * @author William Bakker + */ +public class GUITestExtension implements Extension, InvocationInterceptor { + //private final FailureScreenshotTaker screenshotTaker; + + public GUITestExtension() { + //screenshotTaker = new FailureScreenshotTaker(new ImageFolderCreator().createImageFolder()); + } + + @Override + public void interceptTestMethod( + Invocation invocation, + ReflectiveInvocationContext invocationContext, + ExtensionContext extensionContext) + throws Throwable { + try { + invocation.proceed(); + } catch (Throwable t) { + //takeScreenshot(invocationContext.getExecutable()); + throw t; + } + } + + private void takeScreenshot(Method method) { + final Class testClass = method.getDeclaringClass(); + if (!(isGUITest(testClass, method))) + return; + //screenshotTaker.saveScreenshot(testNameFrom(testClass, method)); + } +} \ No newline at end of file diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/ScanTestUtils.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/ScanTestUtils.java index 26776c48..19bf8b3b 100644 --- a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/ScanTestUtils.java +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/ScanTestUtils.java @@ -1,127 +1,177 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.whiterabbit.scan; -import org.ohdsi.databases.DbType; -import org.ohdsi.databases.RichConnection; -import org.ohdsi.ooxml.ReadXlsxFileWithHeader; -import org.ohdsi.utilities.files.Row; -import org.ohdsi.utilities.files.RowUtilities; -import org.ohdsi.whiteRabbit.DbSettings; -import org.testcontainers.containers.PostgreSQLContainer; +import org.apache.poi.ss.usermodel.Cell; +import org.apache.poi.xssf.usermodel.XSSFSheet; +import org.apache.poi.xssf.usermodel.XSSFWorkbook; +import org.assertj.swing.timing.Condition; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.whiterabbit.Console; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; +import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; -import java.util.ArrayList; -import java.util.List; +import java.util.*; +import java.util.stream.IntStream; +import static org.assertj.swing.timing.Pause.pause; +import static org.assertj.swing.timing.Timeout.timeout; import static org.junit.jupiter.api.Assertions.*; +import static org.ohdsi.databases.configuration.DbType.*; public class ScanTestUtils { - public static void verifyScanResultsFromXSLX(Path results, DbType dbType) { - assertTrue(Files.exists(results)); - FileInputStream file = null; - try { - file = new FileInputStream(new File(results.toUri())); - } catch (FileNotFoundException e) { - throw new RuntimeException(String.format("File %s was expected to be found, but does not exist.", results), e); - } - - ReadXlsxFileWithHeader sheet = new ReadXlsxFileWithHeader(file); + // Convenience for having the same scan parameters across tests + public static SourceDataScan createSourceDataScan() { + SourceDataScan sourceDataScan = new SourceDataScan(); + sourceDataScan.setMinCellCount(5); + sourceDataScan.setScanValues(true); + sourceDataScan.setMaxValues(1000); + sourceDataScan.setNumStatsSamplerSize(500); + sourceDataScan.setCalculateNumericStats(false); + sourceDataScan.setSampleSize(100000); + + return sourceDataScan; + } - List data = new ArrayList<>(); - int i = 0; - for (Row row : sheet) { - data.add(row); - i++; - } + public static boolean scanResultsSheetMatchesReference(Path scanResults, Path referenceResults, DbType dbType) throws IOException { + Map>> scanSheets = readXlsxAsStringValues(scanResults); + Map>> referenceSheets = readXlsxAsStringValues(referenceResults); - // apparently the order of rows in the generated xslx table is not fixed, - // so they need to be sorted to be able to verify their contents - RowUtilities.sort(data, "Table", "Field"); - assertEquals(42, i); - - // since the table is generated with empty lines between the different tables of the source database, - // a number of empty lines is expected. Verify this, and the first non-empty line - expectRowNIsLike(0, data, dbType, "", "", "", "", "", ""); - expectRowNIsLike(1, data, dbType, "", "", "", "", "", ""); - expectRowNIsLike(2, data, dbType, "cost", "amount_allowed", "", "numeric", "0", "34"); - - // sample some other rows in the available range - expectRowNIsLike(9, data,dbType, "cost", "drg_source_value", "", "character varying", "0", "34"); - expectRowNIsLike(23, data,dbType, "cost", "total_paid", "", "numeric", "0", "34"); - expectRowNIsLike(24, data,dbType, "person", "birth_datetime", "", "timestamp without time zone", "0", "30"); - expectRowNIsLike(41, data,dbType, "person", "year_of_birth", "", "integer", "0", "30"); + return scanValuesMatchReferenceValues(scanSheets, referenceSheets, dbType); } - private static void expectRowNIsLike(int n, List rows, DbType dbType, String... expectedValues) { - assert expectedValues.length == 6; - testColumnValue(n, rows.get(n), "Table", expectedValues[0]); - testColumnValue(n, rows.get(n), "Field", expectedValues[1]); - testColumnValue(n, rows.get(n), "Description", expectedValues[2]); - testColumnValue(n, rows.get(n), "Type", expectedTypeValue(expectedValues[3], dbType)); - testColumnValue(n, rows.get(n), "Max length", expectedValues[4]); - testColumnValue(n, rows.get(n), "N rows", expectedValues[5]); - } + public static boolean isScanReportGeneratedAndMatchesReference(Console console, Path expectedPath, Path referencePath, DbType dbType) throws IOException { + assertNotNull(console); + // wait for the "Scan report generated:" message in the Console text area + pause(new Condition("Label Timeout") { + public boolean test() { + return console.getText().contains("Scan report generated:"); + } - private static void testColumnValue(int i, Row row, String fieldName, String expected) { - if (!expected.equalsIgnoreCase(row.get(fieldName))) { - fail(String.format("In row %d, value '%s' was expected for column '%s', but '%s' was found", - i, expected, fieldName, row.get(fieldName))); - } + }, timeout(10000)); + assertTrue(console.getText().contains(expectedPath.toString())); + + return scanResultsSheetMatchesReference(expectedPath, referencePath, dbType); } - private static String expectedTypeValue(String columnName, DbType dbType) { - /* - * This is very pragmatical and may need to change when tests are added for more databases. - * For now, PostgreSQL is used as the reference, and the expected types need to be adapted to match - * for other database. - */ - if (dbType == DbType.POSTGRESQL || columnName.equals("")) { - return columnName; - } - else if (dbType == DbType.ORACLE){ - switch (columnName) { - case "integer": - return "NUMBER"; - case "numeric": - return "FLOAT"; - case "character varying": - return "VARCHAR2"; - case "timestamp without time zone": - // seems a mismatch in the OMOP CMD v5.2 (Oracle defaults to WITH time zone) - return "TIMESTAMP(6) WITH TIME ZONE"; - default: - throw new RuntimeException("Unsupported column type: " + columnName); + public static boolean scanValuesMatchReferenceValues(Map>> scanSheets, Map>> referenceSheets, DbType dbType) { + assertEquals(scanSheets.size(), referenceSheets.size(), "Number of sheets does not match."); + for (String tabName: new String[]{"Field Overview", "Table Overview", "cost.csv", "person.csv"}) { + if (scanSheets.containsKey(tabName)) { + List> scanSheet = scanSheets.get(tabName); + List> referenceSheet = referenceSheets.get(tabName); + assertEquals(scanSheet.size(), referenceSheet.size(), String.format("Number of rows in sheet %s does not match.", tabName)); + // in WhiteRabbit v0.10.7 and older, the order or tables is not defined, so this can result in differences due to the rows + // being in a different order. By sorting the rows in both sheets, these kind of differences should not play a role. + scanSheet.sort(new RowsComparator()); + referenceSheet.sort(new RowsComparator()); + for (int i = 0; i < scanSheet.size(); ++i) { + final int fi = i; + IntStream.range(0, scanSheet.get(fi).size()) + .parallel() + .forEach(j -> { + final String scanValue = scanSheet.get(fi).get(j); + final String referenceValue = referenceSheet.get(fi).get(j); + if (tabName.equals("Field Overview") && j == 3 && !scanValue.equalsIgnoreCase(referenceValue)) { + assertTrue(matchTypeName(scanValue, referenceValue, dbType), + String.format("Field type '%s' cannot be matched with reference type '%s' for DbType %s", + scanValue, referenceValue, dbType.name())); + } else { + assertTrue(scanValue.equalsIgnoreCase(referenceValue), + String.format("In sheet %s, value '%s' in scan results does not match '%s' in reference", + tabName, scanValue, referenceValue)); + } + }); + } } } - else { - throw new RuntimeException("Unsupported DBType: " + dbType); - } + + return true; } - static DbSettings getTestPostgreSQLSettings(PostgreSQLContainer container) { - DbSettings dbSettings = new DbSettings(); - dbSettings.dbType = DbType.POSTGRESQL; - dbSettings.sourceType = DbSettings.SourceType.DATABASE; - dbSettings.server = container.getJdbcUrl(); - dbSettings.database = "public"; // yes, really - dbSettings.user = container.getUsername(); - dbSettings.password = container.getPassword(); - dbSettings.tables = getTableNamesPostgreSQL(dbSettings); - - return dbSettings; + private static boolean matchTypeName(String type, String reference, DbType dbType) { + if (dbType == ORACLE) { + switch (type) { + case "NUMBER": return reference.equals("integer"); + case "VARCHAR2": return reference.equals("character varying"); + case "FLOAT": return reference.equals("numeric"); + // seems a mismatch in the OMOP CMD v5.2 (Oracle defaults to WITH time zone): + case "TIMESTAMP(6) WITH TIME ZONE": return reference.equals("timestamp without time zone"); + default: throw new RuntimeException(String.format("Unsupported column type '%s' for DbType %s ", type, dbType.name())); + } + } else if (dbType == DbType.SNOWFLAKE) { + switch (type) { + case "NUMBER": return reference.equals("integer") || reference.equals("numeric"); + case "VARCHAR": return reference.equals("character varying"); + case "TIMESTAMPNTZ": return reference.equals("timestamp without time zone"); + default: throw new RuntimeException(String.format("Unsupported column type '%s' for DbType %s ", type, dbType.name())); + } + } else { + throw new RuntimeException("Unsupported DbType: " + dbType.name()); + } } - static List getTableNamesPostgreSQL(DbSettings dbSettings) { - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { - return richConnection.getTableNames("public"); + static class RowsComparator implements Comparator> { + @Override + public int compare(List o1, List o2) { + String firstString_o1 = o1.get(0); + String firstString_o2 = o2.get(0); + return firstString_o1.compareToIgnoreCase(firstString_o2); } } + private static Map>> readXlsxAsStringValues(Path xlsx) throws IOException { + assertTrue(Files.exists(xlsx), String.format("File %s does not exist.", xlsx)); + + Map>> sheets = new HashMap<>(); + FileInputStream file = null; + try { + file = new FileInputStream(new File(xlsx.toUri())); + } catch (FileNotFoundException e) { + throw new RuntimeException(String.format("File %s was expected to be found, but does not exist.", xlsx), e); + } + XSSFWorkbook xssfWorkbook = new XSSFWorkbook(file); + + for (int i = 0; i < xssfWorkbook.getNumberOfSheets(); ++i) { + XSSFSheet xssfSheet = xssfWorkbook.getSheetAt(i); + + List> sheet = new ArrayList<>(); + for (org.apache.poi.ss.usermodel.Row row : xssfSheet) { + List values = new ArrayList<>(); + for (Cell cell: row) { + switch (cell.getCellType()) { + case NUMERIC: values.add(String.valueOf(cell.getNumericCellValue())); break; + case STRING: values.add(cell.getStringCellValue()); break; + default: throw new RuntimeException("Unsupported cell type: " + cell.getCellType().name()); + }; + } + sheet.add(values); + } + sheets.put(xssfSheet.getSheetName(), sheet); + } + return sheets; + } } diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanOracle.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanOracleIT.java similarity index 72% rename from whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanOracle.java rename to whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanOracleIT.java index 3f31dfe5..d05fd37c 100644 --- a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanOracle.java +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanOracleIT.java @@ -1,25 +1,42 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ package org.ohdsi.whiterabbit.scan; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; -import org.ohdsi.databases.DbType; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; import org.ohdsi.databases.RichConnection; -import org.ohdsi.whiteRabbit.DbSettings; -import org.ohdsi.whiteRabbit.scan.SourceDataScan; import org.testcontainers.containers.OracleContainer; import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; import java.io.*; import java.net.URISyntaxException; +import java.net.URL; import java.nio.file.Files; import java.nio.file.Path; +import java.nio.file.Paths; import java.util.*; import static org.junit.jupiter.api.Assertions.*; -@Testcontainers(disabledWithoutDocker = true) -class TestSourceDataScanOracle { +class SourceDataScanOracleIT { private final static String USER_NAME = "test_user"; private final static String SCHEMA_NAME = USER_NAME; @@ -44,11 +61,16 @@ class TestSourceDataScanOracle { .withDatabaseName("testDB") .withInitScript("scan_data/create_data_oracle.sql"); + @BeforeAll + public static void startContainer() { + oracleContainer.start(); + } + @Test public void connectToDatabase() { // this is also implicitly tested by testSourceDataScan(), but having it fail separately helps identify problems quicker DbSettings dbSettings = getTestDbSettings(); - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { + try (RichConnection richConnection = new RichConnection(dbSettings)) { // do nothing, connection will be closed automatically because RichConnection implements interface Closeable } } @@ -61,14 +83,16 @@ public void testGetTableNames() { assertEquals(2, tableNames.size()); } @Test - void testSourceDataScan(@TempDir Path tempDir) throws IOException { + void testSourceDataScan(@TempDir Path tempDir) throws IOException, URISyntaxException { loadData(); Path outFile = tempDir.resolve("scanresult.xslx"); - SourceDataScan sourceDataScan = new SourceDataScan(); + URL referenceScanReport = SourceDataScanOracleIT.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + + SourceDataScan sourceDataScan = ScanTestUtils.createSourceDataScan(); DbSettings dbSettings = getTestDbSettings(); sourceDataScan.process(dbSettings, outFile.toString()); - ScanTestUtils.verifyScanResultsFromXSLX(outFile, dbSettings.dbType); + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(outFile, Paths.get(referenceScanReport.toURI()), DbType.ORACLE)); } private void loadData() { @@ -78,15 +102,16 @@ private void loadData() { private void insertDataFromCsv(String tableName) { DbSettings dbSettings = getTestDbSettings(); - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { + try (RichConnection richConnection = new RichConnection(dbSettings)) { try (BufferedReader reader = new BufferedReader(getResourcePath(tableName))) { String line = null; while ((line = reader.readLine()) != null) { - String[] values = line.split("\t"); - if (line.endsWith("\t")) { + String[] values = line.split(","); + if (line.endsWith(",")) { values = Arrays.copyOf(values, values.length + 1); values[values.length - 1] = ""; } + // Oracle INSERT needs quotes around the values String insertSql = String.format("INSERT INTO %s.%s VALUES('%s');", dbSettings.database, tableName, String.join("','", values)); richConnection.execute(insertSql); } @@ -97,7 +122,7 @@ private void insertDataFromCsv(String tableName) { } private InputStreamReader getResourcePath(String tableName) throws URISyntaxException, IOException { - String resourceName = String.format("scan_data/%s.csv", tableName); + String resourceName = String.format("scan_data/%s-no-header.csv", tableName); ClassLoader classLoader = getClass().getClassLoader(); File file = new File(Objects.requireNonNull(classLoader.getResource(resourceName)).toURI()); @@ -105,7 +130,7 @@ private InputStreamReader getResourcePath(String tableName) throws URISyntaxExce } private List getTableNames(DbSettings dbSettings) { - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { + try (RichConnection richConnection = new RichConnection(dbSettings)) { return richConnection.getTableNames(SCHEMA_NAME); } } diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLGuiIT.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLGuiIT.java new file mode 100644 index 00000000..b205f108 --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLGuiIT.java @@ -0,0 +1,121 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import com.github.caciocavallosilano.cacio.ctc.junit.CacioTest; +import org.assertj.swing.annotation.GUITest; +import org.assertj.swing.core.GenericTypeMatcher; +import org.assertj.swing.edt.GuiActionRunner; +import org.assertj.swing.finder.WindowFinder; +import org.assertj.swing.fixture.DialogFixture; +import org.assertj.swing.fixture.FrameFixture; +import org.junit.jupiter.api.*; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.whiterabbit.Console; +import org.ohdsi.whiterabbit.WhiteRabbitMain; +import org.ohdsi.whiterabbit.gui.LocationsPanel; +import org.testcontainers.containers.PostgreSQLContainer; +import org.testcontainers.junit.jupiter.Container; + +import javax.swing.*; +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.ohdsi.databases.configuration.DbType.POSTGRESQL; +import static org.ohdsi.whiterabbit.scan.SourceDataScanPostgreSQLIT.createPostgreSQLContainer; + +@ExtendWith(GUITestExtension.class) +@CacioTest +class SourceDataScanPostgreSQLGuiIT { + + private static FrameFixture window; + private static Console console; + + private final static int WIDTH = 1920; + private final static int HEIGHT = 1080; + @BeforeAll + public static void setupOnce() { + System.setProperty("cacio.managed.screensize", String.format("%sx%s", WIDTH, HEIGHT)); + } + + @BeforeEach + public void onSetUp() { + String[] args = {}; + WhiteRabbitMain whiteRabbitMain = GuiActionRunner.execute(() -> new WhiteRabbitMain(true, args)); + console = whiteRabbitMain.getConsole(); + window = new FrameFixture(whiteRabbitMain.getFrame()); + window.show(); // shows the frame to test + } + + @Container + public static PostgreSQLContainer postgreSQL = createPostgreSQLContainer(); + + @ExtendWith(GUITestExtension.class) + @Test + void testConnectionAndSourceDataScan(@TempDir Path tempDir) throws IOException, URISyntaxException { + URL referenceScanReport = TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + Path personCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/person-no-header.csv").toURI()); + Path costCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/cost-no-header.csv").toURI()); + Files.copy(personCsv, tempDir.resolve("person.csv")); + Files.copy(costCsv, tempDir.resolve("cost.csv")); + window.tabbedPane(WhiteRabbitMain.NAME_TABBED_PANE).selectTab(WhiteRabbitMain.LABEL_LOCATIONS); + window.comboBox("SourceType").selectItem(DbType.POSTGRESQL.label()); + window.textBox("FolderField").setText(tempDir.toAbsolutePath().toString()); + // verify one tooltip text, assume that all other tooltip texts will be fine too (fingers crossed) + assertEquals(LocationsPanel.TOOLTIP_POSTGRESQL_SERVER, window.textBox(LocationsPanel.LABEL_SERVER_LOCATION).target().getToolTipText()); + window.textBox(LocationsPanel.LABEL_SERVER_LOCATION).setText(String.format("%s:%s/%s", + postgreSQL.getHost(), + postgreSQL.getFirstMappedPort(), + postgreSQL.getDatabaseName())); + window.textBox(LocationsPanel.LABEL_USER_NAME).setText(postgreSQL.getUsername()); + window.textBox(LocationsPanel.LABEL_PASSWORD).setText(postgreSQL.getPassword()); + window.textBox(LocationsPanel.LABEL_DATABASE_NAME).setText("public"); + + // use the "Test connection" button + window.button(WhiteRabbitMain.LABEL_TEST_CONNECTION).click(); + GenericTypeMatcher matcher = new GenericTypeMatcher(JDialog.class, true) { + protected boolean isMatching(JDialog frame) { + return WhiteRabbitMain.LABEL_CONNECTION_SUCCESSFUL.equals(frame.getTitle()); + } + }; + DialogFixture frame = WindowFinder.findDialog(matcher).using(window.robot()); + frame.button().click(); + + // switch to the scan panel, add all tables found and run the scan + window.tabbedPane(WhiteRabbitMain.NAME_TABBED_PANE).selectTab(WhiteRabbitMain.LABEL_SCAN).click(); + window.button(WhiteRabbitMain.LABEL_ADD_ALL_IN_DB).click(); + window.button(WhiteRabbitMain.LABEL_SCAN_TABLES).click(); + + // verify the generated scan report against the reference + assertTrue(ScanTestUtils.isScanReportGeneratedAndMatchesReference( + console, + tempDir.resolve("ScanReport.xlsx"), + Paths.get(referenceScanReport.toURI()), + POSTGRESQL)); + + //window.close(); + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLIT.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLIT.java new file mode 100644 index 00000000..2892e3a8 --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanPostgreSQLIT.java @@ -0,0 +1,107 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.configuration.DbSettings; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.databases.RichConnection; +import org.testcontainers.containers.BindMode; +import org.testcontainers.containers.PostgreSQLContainer; +import org.testcontainers.junit.jupiter.Container; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; + + +class SourceDataScanPostgreSQLIT { + + @Container + public static PostgreSQLContainer postgreSQL = createPostgreSQLContainer(); + + @Test + public void connectToDatabase() { + // this is also implicitly tested by testSourceDataScan(), but having it fail separately helps identify problems quicker + DbSettings dbSettings = getTestDbSettings(); + try (RichConnection richConnection = new RichConnection(dbSettings)) { + // do nothing, connection will be closed automatically because RichConnection implements interface Closeable + } + } + + @Test + public void testGetTableNames() { + // this is also implicitly tested by testSourceDataScan(), but having it fail separately helps identify problems quicker + DbSettings dbSettings = getTestDbSettings(); + List tableNames = getTableNames(dbSettings); + assertEquals(2, tableNames.size()); + } + + public static PostgreSQLContainer createPostgreSQLContainer() { + PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer<>("postgres:13.1") + .withUsername("test") + .withPassword("test") + .withDatabaseName("test") + .withClasspathResourceMapping( + "scan_data", + "/scan_data", + BindMode.READ_ONLY) + .withInitScript("scan_data/create_data_postgresql.sql"); + + postgreSQLContainer.start(); + + return postgreSQLContainer; + } + + @Test + void testSourceDataScan(@TempDir Path tempDir) throws IOException, URISyntaxException { + Path outFile = tempDir.resolve("scanresult.xslx"); + URL referenceScanReport = SourceDataScanPostgreSQLIT.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + + SourceDataScan sourceDataScan = ScanTestUtils.createSourceDataScan(); + DbSettings dbSettings = getTestDbSettings(); + + sourceDataScan.process(dbSettings, outFile.toString()); + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(outFile, Paths.get(referenceScanReport.toURI()), DbType.POSTGRESQL)); + } + + private List getTableNames(DbSettings dbSettings) { + try (RichConnection richConnection = new RichConnection(dbSettings)) { + return richConnection.getTableNames("public"); + } + } + + private DbSettings getTestDbSettings() { + DbSettings dbSettings = new DbSettings(); + dbSettings.dbType = DbType.POSTGRESQL; + dbSettings.sourceType = DbSettings.SourceType.DATABASE; + dbSettings.server = postgreSQL.getJdbcUrl(); + dbSettings.database = "public"; // always for PostgreSQL + dbSettings.user = postgreSQL.getUsername(); + dbSettings.password = postgreSQL.getPassword(); + dbSettings.tables = getTableNames(dbSettings); + + return dbSettings; + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeGuiIT.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeGuiIT.java new file mode 100644 index 00000000..880c8c2f --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeGuiIT.java @@ -0,0 +1,142 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import com.github.caciocavallosilano.cacio.ctc.junit.CacioTest; +import org.assertj.swing.annotation.GUITest; +import org.assertj.swing.core.GenericTypeMatcher; +import org.assertj.swing.edt.GuiActionRunner; +import org.assertj.swing.finder.WindowFinder; +import org.assertj.swing.fixture.DialogFixture; +import org.assertj.swing.fixture.FrameFixture; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.SnowflakeHandler.SnowflakeConfiguration; +import org.ohdsi.databases.SnowflakeTestUtils; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.whiterabbit.Console; +import org.ohdsi.whiterabbit.WhiteRabbitMain; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.junit.jupiter.Container; + +import javax.swing.*; +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.ohdsi.databases.configuration.DbType.SNOWFLAKE; +import static org.ohdsi.whiterabbit.scan.SourceDataScanSnowflakeIT.*; + +@ExtendWith(GUITestExtension.class) +@CacioTest +class SourceDataScanSnowflakeGuiIT { + + private static FrameFixture window; + private static Console console; + + private final static int WIDTH = 1920; + private final static int HEIGHT = 1080; + @BeforeAll + public static void setupOnce() { + System.setProperty("cacio.managed.screensize", String.format("%sx%s", WIDTH, HEIGHT)); + } + + @Container + public static GenericContainer testContainer; + + @BeforeEach + public void onSetUp() { + try { + testContainer = createPythonContainer(); + prepareTestData(testContainer); + } catch (IOException | InterruptedException e) { + throw new RuntimeException("Creating python container failed."); + } + String[] args = {}; + WhiteRabbitMain whiteRabbitMain = GuiActionRunner.execute(() -> new WhiteRabbitMain(true, args)); + console = whiteRabbitMain.getConsole(); + window = new FrameFixture(whiteRabbitMain.getFrame()); + window.show(); // shows the frame to test + } + + @ExtendWith(GUITestExtension.class) + @Test + void testConnectionAndSourceDataScan(@TempDir Path tempDir) throws IOException, URISyntaxException { + Assumptions.assumeTrue(new SnowflakeTestUtils.SnowflakeSystemPropertiesFileChecker(), "Snowflake system properties file not available"); + URL referenceScanReport = TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + Path personCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/person-no-header.csv").toURI()); + Path costCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/cost-no-header.csv").toURI()); + Files.copy(personCsv, tempDir.resolve("person.csv")); + Files.copy(costCsv, tempDir.resolve("cost.csv")); + window.tabbedPane(WhiteRabbitMain.NAME_TABBED_PANE).selectTab(WhiteRabbitMain.LABEL_LOCATIONS); + window.comboBox("SourceType").selectItem(DbType.SNOWFLAKE.label()); + window.textBox("FolderField").setText(tempDir.toAbsolutePath().toString()); + + // first use the test connection button, and expect a popup that informs us that several required fields are empty + // use the "Test connection" button + window.button(WhiteRabbitMain.LABEL_TEST_CONNECTION).click(); + GenericTypeMatcher matcher = new GenericTypeMatcher(JDialog.class, true) { + protected boolean isMatching(JDialog frame) { + return WhiteRabbitMain.TITLE_ERRORS_IN_DATABASE_CONFIGURATION.equals(frame.getTitle()); + } + }; + DialogFixture frame = WindowFinder.findDialog(matcher).using(window.robot()); + frame.button().click(); // close the popup + + // fill in all the required values and try again + assertEquals(SnowflakeConfiguration.TOOLTIP_SNOWFLAKE_ACCOUNT, window.textBox(SnowflakeConfiguration.SNOWFLAKE_ACCOUNT).target().getToolTipText()); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_ACCOUNT).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_ACCOUNT")); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_USER).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_USER")); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_PASSWORD).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_PASSWORD")); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_WAREHOUSE).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_WAREHOUSE")); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_DATABASE).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_DATABASE")); + window.textBox(SnowflakeConfiguration.SNOWFLAKE_SCHEMA).setText(SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_SCHEMA")); + + // use the "Test connection" button + window.button(WhiteRabbitMain.LABEL_TEST_CONNECTION).click(); + matcher = new GenericTypeMatcher(JDialog.class, true) { + protected boolean isMatching(JDialog frame) { + return WhiteRabbitMain.LABEL_CONNECTION_SUCCESSFUL.equals(frame.getTitle()); + } + }; + frame = WindowFinder.findDialog(matcher).using(window.robot()); + frame.button().click(); + + // switch to the scan panel, add all tables found and run the scan + window.tabbedPane(WhiteRabbitMain.NAME_TABBED_PANE).selectTab(WhiteRabbitMain.LABEL_SCAN).click(); + window.button(WhiteRabbitMain.LABEL_ADD_ALL_IN_DB).click(); + window.button(WhiteRabbitMain.LABEL_SCAN_TABLES).click(); + + // verify the generated scan report against the reference + assertTrue(ScanTestUtils.isScanReportGeneratedAndMatchesReference( + console, + tempDir.resolve("ScanReport.xlsx"), + Paths.get(referenceScanReport.toURI()), + SNOWFLAKE)); + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeIT.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeIT.java new file mode 100644 index 00000000..acfe3dec --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/SourceDataScanSnowflakeIT.java @@ -0,0 +1,159 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import org.apache.commons.lang.StringUtils; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.databases.SnowflakeTestUtils; +import org.ohdsi.whiterabbit.WhiteRabbitMain; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.testcontainers.containers.BindMode; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.utility.DockerImageName; + +import java.io.File; +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.*; + +public class SourceDataScanSnowflakeIT { + + public final static String SNOWFLAKE_ACCOUNT_ENVIRONMENT_VARIABLE = "SNOWFLAKE_WR_TEST_ACCOUNT"; + static Logger logger = LoggerFactory.getLogger(SourceDataScanSnowflakeIT.class); + + final static String CONTAINER_DATA_PATH = "/scan_data"; + @Container + public static GenericContainer testContainer; + + @BeforeEach + public void setUp() { + try { + testContainer = createPythonContainer(); + prepareTestData(testContainer); + } catch (IOException | InterruptedException e) { + throw new RuntimeException("Creating python container failed."); + } + } + + //@Test + void testWarnWhenRunningWithoutSnowflakeConfigured() { + String snowflakeWrTestAccunt = System.getenv(SNOWFLAKE_ACCOUNT_ENVIRONMENT_VARIABLE); + assertFalse(StringUtils.isEmpty(snowflakeWrTestAccunt) && StringUtils.isEmpty(System.getProperty("ohdsi.org.whiterabbit.skip_snowflake_tests")), + String.format("\nTest class %s is being run without a Snowflake test instance configured.\n" + + "This is NOT a valid verification run.", SourceDataScanSnowflakeIT.class.getName())); + } + + @Test + //@EnabledIfEnvironmentVariable(named = SNOWFLAKE_ACCOUNT_ENVIRONMENT_VARIABLE, matches = ".+") + void testProcessSnowflakeFromIni(@TempDir Path tempDir) throws URISyntaxException, IOException { + Assumptions.assumeTrue(new SnowflakeTestUtils.SnowflakeSystemPropertiesFileChecker(), "Snowflake system properties file not available"); + Charset charset = StandardCharsets.UTF_8; + Path iniFile = tempDir.resolve("snowflake.ini"); + URL iniTemplate = SourceDataScanSnowflakeIT.class.getClassLoader().getResource("scan_data/snowflake.ini.template"); + URL referenceScanReport = SourceDataScanSnowflakeIT.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + assert iniTemplate != null; + String content = new String(Files.readAllBytes(Paths.get(iniTemplate.toURI())), charset); + content = content.replaceAll("%WORKING_FOLDER%", tempDir.toString()) + .replaceAll("%SNOWFLAKE_ACCOUNT%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_ACCOUNT")) + .replaceAll("%SNOWFLAKE_USER%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_USER")) + .replaceAll("%SNOWFLAKE_PASSWORD%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_PASSWORD")) + .replaceAll("%SNOWFLAKE_WAREHOUSE%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_WAREHOUSE")) + .replaceAll("%SNOWFLAKE_DATABASE%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_DATABASE")) + .replaceAll("%SNOWFLAKE_SCHEMA%", SnowflakeTestUtils.getPropertyOrFail("SNOWFLAKE_WR_TEST_SCHEMA")); + Files.write(iniFile, content.getBytes(charset)); + WhiteRabbitMain wrMain = new WhiteRabbitMain(true, new String[]{"-ini", iniFile.toAbsolutePath().toString()}); + assert referenceScanReport != null; + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(tempDir.resolve("ScanReport.xlsx"), Paths.get(referenceScanReport.toURI()), DbType.SNOWFLAKE)); + } + + static void prepareTestData(GenericContainer container) throws IOException, InterruptedException { + SnowflakeTestUtils.SnowflakeSystemPropertiesFileChecker checker = new SnowflakeTestUtils.SnowflakeSystemPropertiesFileChecker(); + if (checker.getAsBoolean()) { + prepareTestData(container, new SnowflakeTestUtils.PropertyReader()); + } + } + + static void prepareTestData(GenericContainer container, SnowflakeTestUtils.ReaderInterface reader) throws IOException, InterruptedException { + // snowsql is used for initializing the database + + // add some packages needed for the installation of snowsql + execAndVerifyCommand(container, "/bin/sh", "-c", "apt update; apt -y install wget unzip"); + // download snowsql + execAndVerifyCommand(container, "/bin/bash", "-c", + "wget -q https://sfc-repo.snowflakecomputing.com/snowsql/bootstrap/1.2/linux_x86_64/snowsql-1.2.29-linux_x86_64.bash;"); + // install snowsql + execAndVerifyCommand(container, "/bin/bash", "-c", + "echo -e \"/tmp\\nN\" | bash snowsql-1.2.29-linux_x86_64.bash "); + + // run the sql script needed to initialize the test data + execAndVerifyCommand(container, "/bin/bash", "-c", + String.format("(cd %s; SNOWSQL_PWD='%s' /tmp/snowsql -a %s -u %s -d %s -s %s -f %s/create_data_snowflake.sql)", + CONTAINER_DATA_PATH, + reader.getOrFail("SNOWFLAKE_WR_TEST_PASSWORD"), + reader.getOrFail(SNOWFLAKE_ACCOUNT_ENVIRONMENT_VARIABLE), + reader.getOrFail("SNOWFLAKE_WR_TEST_USER"), + reader.getOrFail("SNOWFLAKE_WR_TEST_DATABASE"), + reader.getOrFail("SNOWFLAKE_WR_TEST_SCHEMA"), + CONTAINER_DATA_PATH + )); + } + + public static GenericContainer createPythonContainer() throws IOException, InterruptedException { + GenericContainer testContainer = new GenericContainer<>(DockerImageName.parse("ubuntu:22.04")) + .withCommand("/bin/sh", "-c", "tail -f /dev/null") // keeps the container running until it is explicitly stopped + .withClasspathResourceMapping( + "scan_data", + CONTAINER_DATA_PATH, + BindMode.READ_ONLY); + + testContainer.start(); + + return testContainer; + } + + private static void execAndVerifyCommand(GenericContainer container, String... command) throws IOException, InterruptedException { + execAndVerifyCommand(container, 0, command); + } + private static void execAndVerifyCommand(GenericContainer container, int expectedExitValue, String... command) throws IOException, InterruptedException { + org.testcontainers.containers.Container.ExecResult result; + + result = container.execInContainer(command); + if (result.getExitCode() != expectedExitValue) { + logger.error("stdout: {}", result.getStdout()); + logger.error("stderr: {}", result.getStderr()); + // hide the password, if present, so it won't appear in logs (pragmatic) + String message = ("Command failed: " + String.join(" ", command)) + .replace(SnowflakeTestUtils.getEnvOrFail("SNOWFLAKE_WR_TEST_PASSWORD"), "xxxxx"); + assertEquals(expectedExitValue, result.getExitCode(), message); + } + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScan.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScan.java deleted file mode 100644 index c36e99ad..00000000 --- a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScan.java +++ /dev/null @@ -1,200 +0,0 @@ -package org.ohdsi.whiterabbit.scan; - -import org.apache.commons.io.FileUtils; -import org.junit.jupiter.api.Tag; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; -import org.ohdsi.databases.DbType; -import org.ohdsi.databases.RichConnection; -import org.ohdsi.ooxml.ReadXlsxFileWithHeader; -import org.ohdsi.utilities.files.Row; -import org.ohdsi.utilities.files.RowUtilities; -import org.ohdsi.whiteRabbit.DbSettings; -import org.ohdsi.whiteRabbit.scan.SourceDataScan; -import org.testcontainers.containers.BindMode; -import org.testcontainers.containers.PostgreSQLContainer; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; - -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.lang.reflect.Field; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.*; - -import static org.junit.jupiter.api.Assertions.*; - - -@Testcontainers -@Tag("DockerRequired") -class TestSourceDataScan { - - @Container - public static PostgreSQLContainer postgreSQL; - - static { - /* - * Since the database is only read, setting it up once suffices. - * - * Note that the init script is read locally, but accesses the CSV files from - * the resource mapped into the container. - * - * The data used in this test are actually OMOP data. One reason for this is convenience: the DDL - * for this data is know and could simply be copied instead of composed. - * Also, for the technical correctness of WhiteRabbit (does it open the database, get the table - * names and scan those tables), the actual nature of the source data does not matter. - */ - try { - postgreSQL = new PostgreSQLContainer<>("postgres:13.1") - .withUsername("test") - .withPassword("test") - .withDatabaseName("test") - .withClasspathResourceMapping( - "scan_data", - "/scan_data", - BindMode.READ_ONLY) - .withInitScript("scan_data/create_data_postgresql.sql"); - - postgreSQL.start(); - - } finally { - if (postgreSQL != null) { - postgreSQL.stop(); - } - } - } - - void testProcess(Path tempDir) throws IOException { - Path outFile = tempDir.resolve(SourceDataScan.SCAN_REPORT_FILE_NAME); - SourceDataScan sourceDataScan = new SourceDataScan(); - DbSettings dbSettings = ScanTestUtils.getTestPostgreSQLSettings(postgreSQL); - - sourceDataScan.process(dbSettings, outFile.toString()); - ScanTestUtils.verifyScanResultsFromXSLX(outFile, dbSettings.dbType); - } - - @Test - void testApachePoiTmpFileProblemWithAutomaticResolution(@TempDir Path tempDir) throws IOException, ReflectiveOperationException { - // intends to verify solution of this bug: https://github.com/OHDSI/WhiteRabbit/issues/293 - - /* - * This tests a fix that assumes that the bug referenced here occurs in a multi-user situation where the - * first user running the scan, and causing /tmp/poifiles to created, does so by creating it read-only - * for everyone else. This directory is not automatically cleaned up, so every following user on the same - * system running the scan encounters the problem that /tmp/poifiles already exists and is read-only, - * causing a crash when the Apacho poi library attemps to create the xslx file. - * - * The class SourceDataScan has been extended with a static method, called implicitly once through a static{} - * block, to create a TempDir strategy that will create a unique directory for each instance/run of WhiteRabbit. - * This effectively solves the assumed error situation. - * - * This test does not execute a multi-user situation, but emulates it by leaving the tmp directory in a - * read-only state after the first scan, and then confirming that a second scan fails. After that, - * a new unique tmp dir is enforced by invoking SourceDataScan.setUniqueTempDirStrategyForApachePoi(), - * and a new scan now runs successfully. - */ - - // Make sure the scenarios are tested without a user configured tmp dir, so set environment variable and - // system property to an empty value - System.setProperty(SourceDataScan.POI_TMP_DIR_PROPERTY_NAME, ""); - updateEnv(SourceDataScan.POI_TMP_DIR_ENVIRONMENT_VARIABLE_NAME, ""); - Path defaultTmpPath = SourceDataScan.getDefaultPoiTmpPath(tempDir); - - if (!Files.exists(defaultTmpPath)) { - Files.createDirectory(defaultTmpPath); - } else { - if (Files.exists(defaultTmpPath.resolve(SourceDataScan.SCAN_REPORT_FILE_NAME))) { - Files.delete(defaultTmpPath.resolve(SourceDataScan.SCAN_REPORT_FILE_NAME)); - } - } - - // process should pass without problem, and afterwards the default tmp dir should exist - testProcess(defaultTmpPath); - assertTrue(Files.exists(defaultTmpPath)); - - // provoke the problem situation. make the default tmp dir readonly, try to process again - assertTrue(Files.deleteIfExists(defaultTmpPath.resolve(SourceDataScan.SCAN_REPORT_FILE_NAME))); // or Apache Poi will happily reuse it - assertTrue(defaultTmpPath.toFile().setReadOnly()); - RuntimeException thrown = assertThrows(RuntimeException.class, () -> { - testProcess(defaultTmpPath); - }); - assertTrue(thrown.getMessage().contains("Permission denied")); - - // invoke the static method to set a new tmp dir, process again (should succeed) and verify that - // the new tmpdir is indeed different from the default - String myTmpDir = SourceDataScan.setUniqueTempDirStrategyForApachePoi(); - testProcess(Paths.get(myTmpDir)); - assertNotEquals(defaultTmpPath.toFile().getAbsolutePath(), myTmpDir); - - // we might have left behind an unworkable situation; attempt to solve that - if (Files.exists(defaultTmpPath) && !Files.isWritable(defaultTmpPath)) { - assertTrue(defaultTmpPath.toFile().setWritable(true)); - } - } - - @Test - void testApachePoiTmpFileProblemWithUserConfiguredResolution(@TempDir Path tempDir) throws IOException, ReflectiveOperationException { - // 1. Verify that the poi tmp dir property is used, if set - Path tmpDirFromProperty = tempDir.resolve("setByProperty"); - System.setProperty(SourceDataScan.POI_TMP_DIR_PROPERTY_NAME, tmpDirFromProperty.toFile().getAbsolutePath()); - Files.createDirectories(tmpDirFromProperty); - - SourceDataScan.setUniqueTempDirStrategyForApachePoi(); // need to reset to pick up the property - testProcess(tmpDirFromProperty); - assertTrue(Files.exists(tmpDirFromProperty)); - - cleanTmpDir(tmpDirFromProperty); - - // 2. Verify that the poi tmp dir environment variable is used, if set, and overrules the property set above - Path tmpDirFromEnvironmentVariable = tempDir.resolve("setByEnvVar"); - updateEnv(SourceDataScan.POI_TMP_DIR_ENVIRONMENT_VARIABLE_NAME, tmpDirFromEnvironmentVariable.toFile().getAbsolutePath()); - Files.createDirectories(tmpDirFromEnvironmentVariable); - - SourceDataScan.setUniqueTempDirStrategyForApachePoi(); // need to reset to pick up the env. var. - testProcess(tmpDirFromEnvironmentVariable); - assertFalse(Files.exists(tmpDirFromProperty)); - assertTrue(Files.exists(tmpDirFromEnvironmentVariable)); - cleanTmpDir(tmpDirFromEnvironmentVariable); - } - - @SuppressWarnings({ "unchecked" }) - private static void updateEnv(String name, String val) throws ReflectiveOperationException { - Map env = System.getenv(); - Field field = env.getClass().getDeclaredField("m"); - field.setAccessible(true); - ((Map) field.get(env)).put(name, val); - } - private List getTableNames(DbSettings dbSettings) { - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { - return richConnection.getTableNames("public"); - } - } - - private static void cleanTmpDir(Path path) { - if (Files.exists(path)) { - if (!Files.isWritable(path)) { - assertTrue(path.toFile().setWritable(true), - String.format("This test cannot run properly if %s exists but is not writeable. Either remove it or make it writeable", - path.toFile().getAbsolutePath())); - } - assertTrue(deleteDir(path.toFile())); - } - } - private static boolean deleteDir(File file) { - if (Files.exists(file.toPath())) { - File[] contents = file.listFiles(); - if (contents != null) { - for (File f : contents) { - if (!Files.isSymbolicLink(f.toPath())) { - deleteDir(f); - } - } - } - return file.delete(); - } - return true; - } -} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvGui.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvGui.java new file mode 100644 index 00000000..20c5d188 --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvGui.java @@ -0,0 +1,86 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import com.github.caciocavallosilano.cacio.ctc.junit.CacioTest; +import org.assertj.swing.edt.GuiActionRunner; +import org.assertj.swing.fixture.FrameFixture; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.*; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.whiterabbit.Console; +import org.ohdsi.whiterabbit.WhiteRabbitMain; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertTrue; +import org.junit.jupiter.api.extension.ExtendWith; +import org.ohdsi.whiterabbit.gui.LocationsPanel; + +@ExtendWith(GUITestExtension.class) +@CacioTest +public class TestSourceDataScanCsvGui { + private static FrameFixture window; + private static Console console; + + private final static int WIDTH = 1920; + private final static int HEIGHT = 1080; + @BeforeAll + public static void setupOnce() { + System.setProperty("cacio.managed.screensize", String.format("%sx%s", WIDTH, HEIGHT)); + } + + @BeforeEach + public void onSetUp() { + String[] args = {}; + WhiteRabbitMain whiteRabbitMain = GuiActionRunner.execute(() -> new WhiteRabbitMain(true, args)); + console = whiteRabbitMain.getConsole(); + window = new FrameFixture(whiteRabbitMain.getFrame()); + window.show(); // shows the frame to test + } + + @Test + void testSourceDataScanFromGui(@TempDir Path tempDir) throws IOException, URISyntaxException { + URL referenceScanReport = TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-csv.xlsx"); + Path personCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/person-header.csv").toURI()); + Path costCsv = Paths.get(TestSourceDataScanCsvGui.class.getClassLoader().getResource("scan_data/cost-header.csv").toURI()); + Files.copy(personCsv, tempDir.resolve("person.csv")); + Files.copy(costCsv, tempDir.resolve("cost.csv")); + window.tabbedPane("TabbedPane").selectTab(WhiteRabbitMain.LABEL_LOCATIONS); + window.comboBox("SourceType").selectItem(DbType.DELIMITED_TEXT_FILES.label()); + window.textBox(LocationsPanel.NAME_DELIMITER).setText(","); + window.textBox("FolderField").setText(tempDir.toAbsolutePath().toString()); + window.tabbedPane("TabbedPane").selectTab("Scan"); + window.button("Add").click(); + window.fileChooser("FileChooser").fileNameTextBox().setText("\"cost.csv\" \"person.csv\""); + window.fileChooser("FileChooser").approveButton().click(); + window.button(WhiteRabbitMain.LABEL_SCAN_TABLES).click(); + + assertTrue(ScanTestUtils.isScanReportGeneratedAndMatchesReference( + console, + tempDir.resolve("ScanReport.xlsx"), + Paths.get(referenceScanReport.toURI()), + DbType.DELIMITED_TEXT_FILES)); + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvIniFile.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvIniFile.java new file mode 100644 index 00000000..0e6f21c9 --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanCsvIniFile.java @@ -0,0 +1,74 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import org.junit.jupiter.api.*; +import org.junit.jupiter.api.io.TempDir; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.whiterabbit.WhiteRabbitMain; +import org.opentest4j.AssertionFailedError; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class TestSourceDataScanCsvIniFile { + @Test + void testSourceDataScanFromIniFile(@TempDir Path tempDir) throws URISyntaxException, IOException { + Charset charset = StandardCharsets.UTF_8; + Path iniFile = tempDir.resolve("tsv.ini"); + URL iniTemplate = TestSourceDataScanCsvIniFile.class.getClassLoader().getResource("scan_data/tsv.ini.template"); + URL referenceScanReport = TestSourceDataScanCsvIniFile.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-csv.xlsx"); + Path personCsv = Paths.get(TestSourceDataScanCsvIniFile.class.getClassLoader().getResource("scan_data/person-header.csv").toURI()); + Path costCsv = Paths.get(TestSourceDataScanCsvIniFile.class.getClassLoader().getResource("scan_data/cost-header.csv").toURI()); + assertNotNull(iniTemplate); + String content = new String(Files.readAllBytes(Paths.get(iniTemplate.toURI())), charset); + content = content.replaceAll("%WORKING_FOLDER%", tempDir.toString()); + Files.write(iniFile, content.getBytes(charset)); + Files.copy(personCsv, tempDir.resolve("person.csv")); + Files.copy(costCsv, tempDir.resolve("cost.csv")); + WhiteRabbitMain wrMain = new WhiteRabbitMain(false, new String[]{"-ini", iniFile.toAbsolutePath().toString()}); + assertNotNull(referenceScanReport); + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(tempDir.resolve("ScanReport.xlsx"), Paths.get(referenceScanReport.toURI()), DbType.DELIMITED_TEXT_FILES)); + } + + @Test + // minimal test to verify comparing ScanReports: test the tester :-) (and no, this test strictly speaking does not belong here, it should be in its own class) + void testCompareSheets() { + // conform that ScanTestUtils.compareSheets does know how to compare scan results (same, different) + Map>> sheets1 = Collections.singletonMap("Field Overview", Collections.singletonList(Arrays.asList("one", "two", "three"))); + Map>> sheets2 = Collections.singletonMap("Field Overview", Collections.singletonList(Arrays.asList("one", "two", "three"))); + Map>> sheets3 = Collections.singletonMap("Field Overview", Collections.singletonList(Arrays.asList("two", "three", "four"))); + AssertionFailedError thrown = Assertions.assertThrows(AssertionFailedError.class, () -> { + ScanTestUtils.scanValuesMatchReferenceValues(sheets1, sheets3, DbType.POSTGRESQL); + }, "AssertionFailedError was expected"); + ScanTestUtils.scanValuesMatchReferenceValues(sheets1, sheets2, DbType.POSTGRESQL); + } +} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanPostgreSQL.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanPostgreSQL.java deleted file mode 100644 index 0678d308..00000000 --- a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/TestSourceDataScanPostgreSQL.java +++ /dev/null @@ -1,83 +0,0 @@ -package org.ohdsi.whiterabbit.scan; - -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; -import org.ohdsi.databases.RichConnection; -import org.ohdsi.whiteRabbit.DbSettings; -import org.ohdsi.whiteRabbit.scan.SourceDataScan; -import org.testcontainers.containers.BindMode; -import org.testcontainers.containers.PostgreSQLContainer; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; - -import java.io.IOException; -import java.nio.file.Path; -import java.util.List; - -import static org.junit.jupiter.api.Assertions.*; - - -@Testcontainers(disabledWithoutDocker = true) -class TestSourceDataScanPostgreSQL { - - @Container - public static PostgreSQLContainer postgreSQL; - - static { - /* - * Since the database is only read, setting it up once suffices. - * - * Note that the init script is read locally, but accesses the CSV files from - * the resource mapped into the container. - * - * The data used in this test are actually OMOP data. One reason for this is convenience: the DDL - * for this data is know and could simply be copied instead of composed. - * Also, for the technical correctness of WhiteRabbit (does it open the database, get the table - * names and scan those tables), the actual nature of the source data does not matter. - */ - try { - postgreSQL = new PostgreSQLContainer<>("postgres:13.1") - .withUsername("test") - .withPassword("test") - .withDatabaseName("test") - .withClasspathResourceMapping( - "scan_data", - "/scan_data", - BindMode.READ_ONLY) - .withInitScript("scan_data/create_data_postgresql.sql"); - - postgreSQL.start(); - - } finally { - if (postgreSQL != null) { - postgreSQL.stop(); - } - } - } - - @Test - public void connectToDatabase() { - // this is also implicitly tested by testSourceDataScan(), but having it fail separately helps identify problems quicker - DbSettings dbSettings = ScanTestUtils.getTestPostgreSQLSettings(postgreSQL); - try (RichConnection richConnection = new RichConnection(dbSettings.server, dbSettings.domain, dbSettings.user, dbSettings.password, dbSettings.dbType)) { - // do nothing, connection will be closed automatically because RichConnection implements interface Closeable - } - } - - @Test - public void testGetTableNames() { - // this is also implicitly tested by testSourceDataScan(), but having it fail separately helps identify problems quicker - DbSettings dbSettings = ScanTestUtils.getTestPostgreSQLSettings(postgreSQL); - List tableNames = ScanTestUtils.getTableNamesPostgreSQL(dbSettings); - assertEquals(2, tableNames.size()); - } - @Test - void testSourceDataScan(@TempDir Path tempDir) throws IOException { - Path outFile = tempDir.resolve("scanresult.xslx"); - SourceDataScan sourceDataScan = new SourceDataScan(); - DbSettings dbSettings = ScanTestUtils.getTestPostgreSQLSettings(postgreSQL); - - sourceDataScan.process(dbSettings, outFile.toString()); - ScanTestUtils.verifyScanResultsFromXSLX(outFile, dbSettings.dbType); - } -} diff --git a/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/VerifyDistributionIT.java b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/VerifyDistributionIT.java new file mode 100644 index 00000000..dbad9b2f --- /dev/null +++ b/whiterabbit/src/test/java/org/ohdsi/whiterabbit/scan/VerifyDistributionIT.java @@ -0,0 +1,223 @@ +/******************************************************************************* + * Copyright 2023 Observational Health Data Sciences and Informatics & The Hyve + * + * This file is part of WhiteRabbit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + ******************************************************************************/ +package org.ohdsi.whiterabbit.scan; + +import org.apache.commons.lang.StringUtils; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.junit.jupiter.api.io.TempDir; +import org.junit.runners.Parameterized; +import org.ohdsi.databases.DBConnector; +import org.ohdsi.databases.SnowflakeTestUtils; +import org.ohdsi.databases.configuration.DbType; +import org.ohdsi.utilities.files.IniFile; +import org.ohdsi.whiterabbit.WhiteRabbitMain; +import org.testcontainers.containers.BindMode; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.Container.ExecResult; +import org.testcontainers.utility.DockerImageName; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.*; +import java.util.function.BooleanSupplier; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.*; +import static org.ohdsi.whiterabbit.scan.SourceDataScanSnowflakeIT.createPythonContainer; +import static org.ohdsi.whiterabbit.scan.SourceDataScanSnowflakeIT.prepareTestData; + +/** + * Intent: "deploy" the distributed application in a docker container (TestContainer) containing a Java runtime + * of a specified version, and runs a test of WhiteRabbit that aim to verify that the distribution is complete, + * i.e. no dependencies are missing. A data for a scan on csv files is used to run whiterabbit. + * + * Note that this does not test any of the JDBC driver dependencies, unless these databases are actually used. + */ +public class VerifyDistributionIT { + + @TempDir + static Path tempDir; + + private static final String WORKDIR_IN_CONTAINER = "/whiterabbit"; + private static final String APPDIR_IN_CONTAINER = "/app"; + + @Test + void testDistributionWithJava8() throws IOException, URISyntaxException, InterruptedException { + testWhiteRabbitInContainer("eclipse-temurin:8", "openjdk version \"1.8."); + } + + @Test + void testDistributionWithJava11() throws IOException, URISyntaxException, InterruptedException { + testWhiteRabbitInContainer("eclipse-temurin:11", "openjdk version \"11.0."); + } + @Test + void testDistributionWithJava17() throws IOException, URISyntaxException, InterruptedException { + testWhiteRabbitInContainer("eclipse-temurin:17", "openjdk version \"17.0."); + } + + @Test + void verifyAllJDBCDriversLoadable() throws IOException, InterruptedException { + try (GenericContainer javaContainer = createJavaContainer("eclipse-temurin:11")) { + javaContainer.start(); + ExecResult execResult = javaContainer.execInContainer("sh", "-c", + String.format("cd %s/repo; java -classpath '*' org.ohdsi.databases.DBConnector", APPDIR_IN_CONTAINER)); + if (execResult.getExitCode() != 0) { + System.out.println("stdout:" + execResult.getStdout()); + System.out.println("stderr:" + execResult.getStderr()); + } + assertTrue(execResult.getStdout().contains(DBConnector.ALL_JDBC_DRIVERS_LOADABLE), "Not all supported JDBC drivers could be loaded"); + javaContainer.execInContainer("sh", "-c", "rm /app/repo/snowflake*"); // sabotage, confirms that test breaks if driver missing + execResult = javaContainer.execInContainer("sh", "-c", + String.format("cd %s/repo; java -classpath '*' org.ohdsi.databases.DBConnector", APPDIR_IN_CONTAINER)); + assertFalse(execResult.getStdout().contains(DBConnector.ALL_JDBC_DRIVERS_LOADABLE), "Not all supported JDBC drivers could be loaded"); + } + } + + //@Test // useful while developing/debugging, leaving in place to test again after Snowflake JDBC driver update + void verifySnowflakeFailureInJava17() throws IOException, URISyntaxException, InterruptedException { + /* + * There is an issue with Snowflake JDBC that causes a failure in Java 16 and later + * (see https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later) + * A flag can be passed to the JVM to work around this: --add-opens=java.base/java.nio=ALL-UNNAMED + * + * The whiteRabbit script in the distribution passes this flag. + * + * The tests below verify that: + * - the flag does not cause problems when running with Java 8 (1.8) or 11 + * - without the flag, a failure occurs when running with Java 17 + * - passing the flag fixes the failure with Java 17 + * + * As the flag is in the distributed script, it needs to be edited out of the script. + * + * Note that we only test with the LTS versions of Java. This leaves Java 16 untested and unfixed. + * + * Once a fix is available in a newer version of the Snowflake JDBC jar, and it is used in WhiteRabbit, + * The test that now confirms the issue by expecting an Assertion error should start to fail. + * Then it is time to remove the flag (it is in the pom.xml for the whiterabbit module), and remove these tests, + * or normalize them to simply verify that all works well. + */ + String patchingFlag = "--add-opens=java.base/java.nio=ALL-UNNAMED"; + String javaOpts = String.format("JAVA_OPTS='%s'", patchingFlag); + + // verify that the flag as set in the whiteRabbit script does not have an adversary effect when running with Java 11 + // note that this flag is not supported by Java 8 (1.8) + runDistributionWithSnowflake("eclipse-temurin:11",javaOpts); + + // verify that the failure occurs when running with Java 17, without the flag + AssertionError ignoredError = Assertions.assertThrows(org.opentest4j.AssertionFailedError.class, () -> { + runDistributionWithSnowflake("eclipse-temurin:17",""); + }); + + // finally, verify that passing the flag fixes the failure when running wuth Java 17 + runDistributionWithSnowflake("eclipse-temurin:17",javaOpts); + } + + void runDistributionWithSnowflake(String javaImageName, String javaOpts) throws IOException, InterruptedException, URISyntaxException { + // test only run when there are settings available for Snowflake; otherwise it should be skipped + Assumptions.assumeTrue(new SnowflakeTestUtils.SnowflakeSystemPropertiesFileChecker(), "Snowflake system properties file not available"); + SnowflakeTestUtils.PropertyReader reader = new SnowflakeTestUtils.PropertyReader(); + try (GenericContainer testContainer = createPythonContainer()) { + prepareTestData(testContainer, reader); + testContainer.stop(); + + try (GenericContainer javaContainer = createJavaContainer(javaImageName)) { + javaContainer.start(); + Charset charset = StandardCharsets.UTF_8; + Path iniFile = tempDir.resolve("snowflake.ini"); + URL iniTemplate = VerifyDistributionIT.class.getClassLoader().getResource("scan_data/snowflake.ini.template"); + URL referenceScanReport = SourceDataScanSnowflakeIT.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-sql.xlsx"); + assert iniTemplate != null; + String content = new String(Files.readAllBytes(Paths.get(iniTemplate.toURI())), charset); + content = content.replaceAll("%WORKING_FOLDER%", WORKDIR_IN_CONTAINER) + .replaceAll("%SNOWFLAKE_ACCOUNT%", reader.getOrFail("SNOWFLAKE_WR_TEST_ACCOUNT")) + .replaceAll("%SNOWFLAKE_USER%", reader.getOrFail("SNOWFLAKE_WR_TEST_USER")) + .replaceAll("%SNOWFLAKE_PASSWORD%", reader.getOrFail("SNOWFLAKE_WR_TEST_PASSWORD")) + .replaceAll("%SNOWFLAKE_WAREHOUSE%", reader.getOrFail("SNOWFLAKE_WR_TEST_WAREHOUSE")) + .replaceAll("%SNOWFLAKE_DATABASE%", reader.getOrFail("SNOWFLAKE_WR_TEST_DATABASE")) + .replaceAll("%SNOWFLAKE_SCHEMA%", reader.getOrFail("SNOWFLAKE_WR_TEST_SCHEMA")); + Files.write(iniFile, content.getBytes(charset)); + // verify that the distribution of whiterabbit has been generated and is available inside the container + ExecResult execResult = javaContainer.execInContainer("sh", "-c", String.format("ls %s", APPDIR_IN_CONTAINER)); + assertTrue(execResult.getStdout().contains("repo"), "WhiteRabbit distribution is not accessible inside container"); + + // run whiterabbit and verify the result + execResult = javaContainer.execInContainer("sh", "-c", + String.format("%s /app/bin/whiteRabbit -ini %s/snowflake.ini", javaOpts, WORKDIR_IN_CONTAINER)); + assertTrue(execResult.getStdout().contains("Started new scan of 2 tables...")); + assertTrue(execResult.getStdout().contains("Scanning table PERSON")); + assertTrue(execResult.getStdout().contains("Scanning table COST")); + assertTrue(execResult.getStdout().contains("Scan report generated: /whiterabbit/ScanReport.xlsx")); + + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(tempDir.resolve("ScanReport.xlsx"), Paths.get(referenceScanReport.toURI()), DbType.SNOWFLAKE)); + } + } + } + + private void testWhiteRabbitInContainer(String imageName, String expectedVersion) throws IOException, InterruptedException, URISyntaxException { + try (GenericContainer javaContainer = createJavaContainer(imageName)) { + javaContainer.start(); + + Charset charset = StandardCharsets.UTF_8; + Path iniFile = tempDir.resolve("tsv.ini"); + URL iniTemplate = VerifyDistributionIT.class.getClassLoader().getResource("scan_data/tsv.ini.template"); + URL referenceScanReport = VerifyDistributionIT.class.getClassLoader().getResource("scan_data/ScanReport-reference-v0.10.7-csv.xlsx"); + Path personCsv = Paths.get(VerifyDistributionIT.class.getClassLoader().getResource("scan_data/person-header.csv").toURI()); + Path costCsv = Paths.get(VerifyDistributionIT.class.getClassLoader().getResource("scan_data/cost-header.csv").toURI()); + assertNotNull(iniTemplate); + String content = new String(Files.readAllBytes(Paths.get(iniTemplate.toURI())), charset); + content = content.replaceAll("%WORKING_FOLDER%", WORKDIR_IN_CONTAINER); + Files.write(iniFile, content.getBytes(charset)); + Files.copy(personCsv, tempDir.resolve("person.csv"), StandardCopyOption.REPLACE_EXISTING); + Files.copy(costCsv, tempDir.resolve("cost.csv"), StandardCopyOption.REPLACE_EXISTING); + + // verify that the default java version in the container is actually 1.8 + ExecResult execResult = javaContainer.execInContainer("sh", "-c", "java -version"); + assertTrue(execResult.getStderr().startsWith(expectedVersion), "default java version in container should match version " + expectedVersion); + + // verify that the distribution of whiterabbit has been generated and is available inside the container + execResult = javaContainer.execInContainer("sh", "-c", String.format("ls %s", APPDIR_IN_CONTAINER)); + assertTrue(execResult.getStdout().contains("repo"), "WhiteRabbit distribution is not accessible inside container"); + + // run whiterabbit and verify the result + execResult = javaContainer.execInContainer("sh", "-c", String.format("/app/bin/whiteRabbit -ini %s/tsv.ini", WORKDIR_IN_CONTAINER)); + assertTrue(execResult.getStdout().contains("Started new scan of 2 tables...")); + assertTrue(execResult.getStdout().contains("Scanning table /whiterabbit/person.csv")); + assertTrue(execResult.getStdout().contains("Scanning table /whiterabbit/cost.csv")); + assertTrue(execResult.getStdout().contains("Scan report generated: /whiterabbit/ScanReport.xlsx")); + + assertTrue(ScanTestUtils.scanResultsSheetMatchesReference(tempDir.resolve("ScanReport.xlsx"), Paths.get(referenceScanReport.toURI()), DbType.DELIMITED_TEXT_FILES)); + + javaContainer.stop(); + } + } + + private GenericContainer createJavaContainer(String imageName) { + return new GenericContainer<>( + DockerImageName.parse(imageName)) + .withCommand("sh", "-c", "tail -f /dev/null") + .withFileSystemBind(Paths.get("../dist").toAbsolutePath().toString(), APPDIR_IN_CONTAINER) + .withFileSystemBind(tempDir.toString(), WORKDIR_IN_CONTAINER, BindMode.READ_WRITE); + } +} diff --git a/whiterabbit/src/test/resources/scan_data/README.md b/whiterabbit/src/test/resources/scan_data/README.md new file mode 100644 index 00000000..19c7fe13 --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/README.md @@ -0,0 +1,6 @@ +The ScanReport-reference-v0.10.7-{csv,sql}.xlsx files in this directory were generated using the last version +of WhiteRabbit that did not have any unit or integration tests, and serve as the reference for smoke/regression +tests. + +Not that the order in which files/tables are generated into these xlsx files was (is) not entirely predictable, +so some sorting is done in the tests to match the version under test. \ No newline at end of file diff --git a/whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-csv.xlsx b/whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-csv.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..b8d0d44164f2b1df797a3711df58973b31089a5f GIT binary patch literal 9379 zcmbVyWmsIxvi9JEI|K+2++lDD?h*(Z+}+(ZxCaPsf#5zk!Cixe0D~t;umJ)DCvYMA zes|~O+vof^HM3@BJbIEDTC8 zgJ%ab8yK#T2H#ute3#8Zib)wn=)-mB%l2j^@WZg(lGJf*YTe>$H5xuXSz004rB0D$tp^#MEd+ zKe~U~!iZDOu;60bK*tc_FX8Z^h|L#B9#lX`*47W<>Yi<+o4L z>3&Xl`x%pEMOVeWk?ca=syauqo<>%kLheH;?yM1o}+^r{v_`^QhrX<#zgT-uMW5 zQ_?zKvmVFLCR(*TABQK)6C3~Sl`#4I9`*AMH3<}K+~*(PKc#PWF^;zkvLpOKJ8xcG z_kzN^kEWQYmy16p&B%}}>gyDjOm|$O^wWw<1o^7#Cz_V%gh(qgY6&JSYk0R8C;=GP zaF5;j>gNebd}wU*!}af7js1IfOdVa!ANymUup-|N!jd*D+U(3JW~m5milIjhUJqu9 z_M{+MGC7sNIk-MAcCX4?rMa4TN^965RL<@#vyD_hB3_?OT^g1ca!7pQ1S#64K7MUG zB2!RHv6x`5i#itS+~PU-9u@AI+NQy>m!}Z3hXcUoqrPmJIK}SAu5^r5Qw=vZz;uKa zEw~oc+f9S{)WX^IH~Nm6nvY z`?+3U&6W10J5O~-WY~{%=ux$$gMancm*WR?k2cg6VIM+x7%t$4>)+cD@4<#%c5JRz z#xCY&YHlvp4wkNuW)zAQ!{~j61vu&qhOV&sfCiJ#jn+|rz=T$Z6*4CyY@BXwI&-`w z+IC$291$JbkEj%aCt+Hy`+y`HANw=;CMUenUp8F>&MC~M4F_srB zXBj<9&Vl#9w!P|hVqbnDR|=TZph^1=5bAVn$!VUpuAB8PK?vW2Ot zUDz$gecj2h4oj8rZTNJQlZ2TarD+FOsB=J%BbBh>2|4y9*6XZYywf|kPOUTaM_Wpt zTH#|p*pl`?W^v-bZ28wD_q#=ZO?Vxj#~#g!FlAV4@iMy*6RUg)U3`yV6iMrtIzAJU zYU%akh@Ckv(I!f2#KtVn??`*x;({Gx9i8fIm7z2m2flSMYt=d;L0PM_}> z_j8|qQzjDllWy2fs z1?RvjeWb72YrY)_1=F(%s!YpxIcX^rr3$s~uFuy^2PA&q$M+-S*&4>=_=d;QuhIHS zU}|bEmBwe;0`}MZ1(T#U*}REo8)-xk_&;*IDYG1^JV_sbrn{5XXOp^S4OQOZQ_&Hc z+#o;N4c?QZ;`@VDEaCrC3_kfg23@Vp&D~rdvfjfN`(rTXWUsiaOJju{o#NqL*rpR5 zr`6>wdP%ga+cP#U*lASLYZHgqh*6q==N-wBhdIRkTWPLP1pSHgUka&`$F=<2y78Ft zt-TT+`PA0gN_6ha6IaJ4r|J2#tLx*M;!W4`z1vOZ*4nwPt$t19%L~t+jjnC)kV5<$ znwFdb-yYcM#0pwo@wg2v`Ef&zTu26Grad^P`o4OB(cd-%29nU}W~Gut#tu&P?>qJ| z4;|mUHpIPYy)d{Ko%A2AbKJ^$bFAM55hN7;Y0k^z(&Tk?62dYYlIa()oGfx`VE3-C zZ_t0jfZV=Lr)^y~KDy@WUM2vjujSVx=zFj=pxos}no<*8wTE9T*D9xalQtG}pcK7) zc&BjLP}>Zr>1TEIM$9b`=QbMtwBNx*{Cf~QIaGM`Z4T2p7R3#K7iuqKKkrC?r*_%V zT)R73g*^MT>~06Y&E_S2z&3uzx@=UZwvCKnuTWZq$_GLpk2Y7w&=ab=j}n=;zTl2D zCs*+2Z>;^8LguCNNCTQ5p0ZrKG*~6rYbL)+U6s5c+)MbXF%~Llmk=rNN*OvN(y1IV z%Ftdpo#yM$*-qMi84)USt)!26e~6j&_3XPTdpjE{6h}SgYPdw}v)ZMuin%GP-3vM% z*~U6+yHoSzqQn*cZeSFHG(7=`V|5#JL)iZ7qh3?L2Mo$pSv-7AOE zF5hLn^b*I-oeJ#$@L^|yL-&Y4`MY6L4Y(AAjMbnjCz#iTc$9dk@>Zj{iN%6>4477n zd7+F9HbJFmD1c0Tz_&?flxg zp;Okxei`}6?b*-mh@p#jau%I3d4uj&!3z2CS>oKs458Ghl*pX;KC7X77D}QE3(7cU zjqD6YAOTN=Wo%*1;FtOE1>)S!jEr)7X;0nai|3!_YUj6pz*l~?HDn0trKXm|i=`XG z7STpI55u`d)$k;b9C7J?nc9x*5;k5BQs>2l8;w^?*K)J3M0*k&9nZ^|{KbO4XF^%= zHr6IdfQ1psW+>Uxy3`>TG5M+?6vuYMq1#B$q`fO2K12c;Bf`eSnSOQSnG1;K+lQy+ zS>rI~X*@cW^*jMcXwvRHTl+7-g2`XJOKFQE*7Y$tkSaMRFO*keG3hHPa37WraIhiH zwd0p2xOP}Hm(R)_OtA)dHe%v<)v6ty_|8Q*e3R0n`L5#;!!amda zNvs3^jjw}x*EVy*dWy4JJtsN$n)Ev?R5=1koMlF z_5j@ip40&ZsR+SlYwDGZ*FH(F#<*b@9i8`iEgPmRGzS@U6z7nwI>!lI^rlZ1?cy@E zQVDQAd<@GnZ@@Uk_g=vaqo&0}t06y2Uy5*Laf75dqo{X0uQn912^aw+hQMsOLx>xz zndzwoLKq9Fi+O~&(FW5mb9GRK+|9FlKNptF`Mp%!Gy-`GHA;^&#vn&2rF&4M|5$XJ z20c;qu|s-3FHWeRf}4=(HvP~&5>!+UIS^5Nv6{WK4*@+lQNfn#qHP7YjfS=jn2^nj z<=rZZgCh@JPos5#F{qlYGj2|tM4zZ+irc#2!RomDPX0tSaXh7Q-((4=F0BkU51jM2q34Cu@=D=A8;%jTbC6fiS&eVocv*|;JsjtPX3l5w%L8}P2A z=f*K0VS+>K6X(2bg{4X-k_K_Uc{wg->sdNhaYCO%2+&jo~NzfVmOY>)uaRPag(R zyWnJ>+p%EY4kVt%GYh1DXqRio?vq{`F{3E%M@#0=&P1d2-liPG)N0I_P~8RoB?0#v_UE}FO1H{D8Ql8a=x-jhETS@F#d{^ZC2fVs?nQDkvE7Fl8H zJq}ACw2mVt_!V1zk0tz&0~1{ny9jiN`Gaam151A@T9jf_7x@-m)FNah?uYN2O)5#t zi@@;QgB}5Mw+T&^k3>78pXHnR@6WUhR*rAh^*uly+X4Wy{(07I()CL*pr$)WmM{aG ziz2QzL{i?KY1v9_?yc316wXkyNrL<-WyLIDk=KkMGdobvS;Y+ZxH&ik)#TfaL)^mE zrftqCL~Q_}Is5t!Tp2Dce0FJsVJO;2npBiUFFb3gki0pV30DvA9L+>N4UBD(zmV^H z*}zr!eo;$s)E+ZK)Mig4vYZb^m<4(63uA>ewI^rYq(a?Mtc43_^w|=M^i7S2@+3X) z*r=+Q;iNm$eTi|Faeo(EN-!cn8~4Loo1W*BE2lvDa)3j};{dMbn5ny1rJjBlcX!?zy|&mlQ@I(h*pmdu6D_TKLUf_X+>B zhz@*o_45ts+Mi4VvMEP* za)L1(9IIm&ZUelzz|=A8>zbP*{+~h`3YD9yMDNB^$W(>50yb-urOvDhy&4Pe#^xpseiSNzxyC{eFtuLF=~K^t(C{v@ zl!nJWBqS{gDsI=M!u$Ex!ReVu%NsT%Y(DQKf54#V0N*fTTokRs59Yz>`uO~T}q}tv4$dbx_C6vvmCgF7|(CUher?#xrh5c!-iGYBv!JS(Wr&MA-XIP?)#FLwkQ8A?zYns@9lvhF$%l~kLU@#t5q&~yQG<&7&u^fsNA=q!UE<(*EZ@XQix=m)Ld$|OGfp81)szt8 z{#@6jJzzd!aEBCF5q?P=Z145+6M`BhpI%pi^!(KX*BaGmfGAiWen|rC5C2>gECK&q z9Q+1&j%=c6>GL$O7|QZhE%*X#OMec3Kx7p2_XwU zpn|{zlTt#kgZ(AJFM%GhM#zIvz`dl+_`&13RX(C%S{tYoJA=IM&IGGARX5ix&b`2PW{CO${Zf)6`H07) zeC4WXfukL{;qB=CuPylYK{JGzhm9WUr~kzq!1XIJWa_J}%i=vG2F*g3E_eh|Thw#` zs%P|r9ES3`%53hQdI+4m>M+F}-9yFqGoq-!sLTkL zsoTd$kzIsX%{0)xSrK7_9#x4Dv+3QwXo{6nI9B=8(5yNb$gi|+zc8W5C}@#*Z81JD zk+P?1mrYIZ@$4eRBi~i0D);$w-bm=HK1>hhQCOCqImIj`yOLQ*$V^ZN^*do*?L*vc zk@}drOSYTB zVj~8g$lhlrrVO8{;WTuc1apTqzVJRRGpQ%%H`uoy#+*s0HVQ1UK{wmkX=#2rV9LfAM|2)acz+8C^(;a#|bnTLo#$ zKtXkEs9rv)uKcQ5!-cSl`-`623-dV#c}8<*4jVu6eI&VT_Zh2t9nO_KRfdeW)8B4# zK156K>I}SqU!KoOk>AaZ754NL)KJ(DrwBA&>{ex^EBGii7e7;Jd4&)cBOjF$YcZL0 zkE-OwDA?rV(Widnq!5)GTYNE4{KY_FH#fF(5;o=#+r6`3!ZSi`b@ywsr%fp*5XV|B zQ973?!>~-7`x!Lrg$h^Q!<7YTfh00y!;BY0`!+q6H z9b4EJwwWSTL>yJ#cMSyH4XIZ%lm)~MB@w|Ug9kN2xOjM+4$k%!Z1fh(UltJ!ls~rg zHI9GCOLunM4`44f^2Dv{C^HJgZRpwU%pcXOYB90~@Jz%xO3QN_S zPZf9CNHLFz{sDu_ET{t>!qe>mXhH>R0+!$)!rcaNQ1osLfCmuV4Dhuyv0V(;o3tPP z-@i}4j}e3w!wt%Tg+e_Yj4ht-y)C=(JyA;=!i4#}tu;EMnj3Oay;a(Ax+ zpa7!?fG;Z8llDBF)fE$72mpQPJ>Uii{`J>rM%#ck0EDI+$AnT`T{1ETn21fmLn$c} zdKrmDfo(!9o-7&p7MO@b@rqJX!R$v;Qw$Y|l2YK4qGmARe`@U;f;u3 zME^s1q{8rDWJ2!8OcxpgntPZQ*f4@Dh~u8m?I;*w1SClj>QUV(G-RnlDp_|qr!M}*A-L`SfidQT_Y3%BsusL*eC^U<~%+RZyLCfFI+x~!>FXBF^ zmzVE(!W-nKVD4dq5U3dUL_~g{I758c)3N>aa}ozb!$JWFlgfH%tfIS1@OLWq5c-}l z3hHl3g@~!)O8Cm49sZ6x1aZO%r?}+`m{wx1)S06;xcSqcthFK1P(Z2v{^F`5O`aov*skdN9pjP$9!cWFrX%bS;%P`cC4n!!-7xVGVs$P&m#G8FDA6%s) zH@zcV!q98<&qq5Rb;z1;4UGjc)4Jw($u)K^?_`7{yX?NF0Sv%!FNd!isRgi-izcXY zS0?&O_k$@)wq!As8T^AGM)Gv^2f+z7MbriSDWSHt%5xV=4r3_TW_GaK=F*hcnrn_n z_xF^aT}0veQ_#!d=EhfXCdXTy-m$zDPf*yp zEGihET!pXKnG1d9DDRZ|l!JNXBmr0Ht!z1r*jN~&oilA~m*ZF&mYK!?Ma{H|e-WQo z$)Zw`^@_u{NR=p0v)5Pg1ifA+s&#tk^rvEzfPxeAdj&nD*&szHGx3 z87RE!+!e**kY zbU#x4e@pzsg5@{j|4+r=Ddk5V{BJ3LXzQP>_@COplc|ri^WWnAP!9c{_WvWH|5X2- z(R^fF|CWx29P&^6>z@d}(`=8-=ilOq@E5}W@t}XI{>~~r@^F6(HPSED-#Mf|m4ByJ z9*L*F#R>RJ`QOk`f8zYUd+3uX+8uX55?o%{{RB91l9lm literal 0 HcmV?d00001 diff --git a/whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-sql.xlsx b/whiterabbit/src/test/resources/scan_data/ScanReport-reference-v0.10.7-sql.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..ea091dfe5d036171e5c5e80666068518c9fce42f GIT binary patch literal 9384 zcmbVybzGF));2M84J|F*A)qwUNF&`PFmyN4-Q6vnf4M@L6^6=|)D@S70Ae;Z0WJ9$_+ zd6;VXx>&gzbNM(rjHiz|z2gDO1~DGD#^U@asZlhWeE&w4?FM145Qx=lNr4%(>KZJ| zno$MR&`IR;J`khUZ(y=7jUx`P%C8?y#(jpUBUlaVVxj?O9TyTka%FG#BC)fe%89RC z7?$Q}oEgq-X1zoker41D{aFE8T*ffU0HM=BzAqQaOI!#32TPB>Iw0kJNEaJ<>lYfy z?a4;nb_e!Ud3$bvQJ5TCUnk1L=^e=*e8AtNEpAd0O!U#-W1$eWHl>k}0GHx~`fA06 zOH$#2G}oS-Q9`(he_o2VQ!%+5j|E$yKhgbzwBIHw4q0raUIKRgj?ltY6FTXavOW$` z_MH;YCA~t%5GJ~QWrDr~rkM{wy7x_GykGY^yypr?|H(H% zcneGC7wT@#F7DjsE-v?Bs*>1`2@g}5!$8@Q?&V!cnW^+p^g@{A(}!iyC!ZXo-Y7m@ z&Hua(;Yq6O7=OK%z}hKMzciOX^z2D|;+Bp$&D<`7NVldmCO)AE-vP!7vUmJKaam@j)G=Q_@yqumq{va$UZ5z}|9qee!XZUlme)+Sc)Ln; zeU2H3`xE)TJ6~UV&!@nRZG&I`eyH((@6HQnH>>;pI3`0C2YHC#({7_%WlA_Kyfusk zIdm8DKP{}IzM2a2Bw_Acq&kgW>5uBqB?sqQ0OSlFXEzMv?Td@N zXOS^ae^4JnP7&v=!bp`hJJ4^4CXNeKpUUnluXUveIHhux z>WRgDCb^So6D<|PGQt5_vKwSBPpzHmzI4TBQ7{AGFrmJ4lM9?jH;D>D%d})nGS{oN z%{{I*dfSXouhu4TK6bodw)I-6PsLNieMeG8odv_^+NC<>aHFZK%kKOwtIc!kcq_*2flrr4`1S97NCfwx zj{~>6jk%kZrKX3QtrN)o-i>1MlDO|40uc^-LSf5XemukJXJ%_yKM=zkBFZ?@QPxj3 zH(Yts$i6u*eU6F^A4F9RLzRV15%@g*?kAq8n_W4;qLHDLP#a>xkLzE0Rdf|L6#V+} zDsad*Rwpt&rqj2Qz;RzsC*5d-Yc0lZ>3iKNN+}Mu;0@eTG<28Vc_tsYT`~W0^l|>z zat~hzt6wG;r1q=cPxV7lS)v$vrRq=N>QE{sV%M*4XkEt*+7h1Svnf<&R5IHt?x;Rg z^}5}-a~K=-B4r#Uizg;`_v=+)L9FxOY#iT#Hg)-zb*?diozh*l5h#yPn}zoC_lh2X zwb5ds2?vNbDy0c)6ErGg4$i@IPqNc3ZBrd`){tydz7{KPa#a5ap3pUBAQ({EAKFJ| z#BZB>yurV~(3Vb2mbWZBYtt18C>en<3%T!P@FWOm8Q4h!W?c~R0

6P>yB_m0Wr0 zqH%Pk5}&Jv@!$ybY9h7d?7nByL{~v)&0^9ZkD%LYcQwVz*p-dTy^7eM9LyUQ9+&1=j`_Jl zo7E$W-BCKm0aYK*2(g#ixlJ?Alf!t~x~5r*F$avI!gJMItyz)2;ajMsPep?iy$O{Z z5J{5Qv+MHJ^*QQw;lLo3#PRtekGimJgH)OGE$u-=O24W(^Qe;1XCOXMmu<9fe-m?L zpkO6N&V;T(}t@lgO(?;B8M;oVtF_FKkMa6p&vosT*_S zT=zv)<{MgJiT2kF5!{}qWq(*pp9@M6u9_t}yNc5a7*Q`yK8T5a2%r;Df&d(A~z$%EKL=^59>O?jx~0AL>>r2aG&CAxb|0 z^I{8^J5&-lH-l>ywb&{Ri6fh2>H(PXFhR8u!EeZyS4lQaw3m@e<;0c~0;A>E zub^q`Mdg0B^&g+NExqr!T?)9=TS|cJ?!b-&S~S~rwmO{>YvQQ!-epy4UzW#Uo~BcI z|D5|NTjK_vv1Vts=DRc8xKajLBj^2Ow2Qa36#ur&4Ax0G;;c2+uU_?dG26-CetL7b zu&%nC*rIQ9p5z*MNAjH>Q!mp_7l??*@}=AEq(amN92MPo?lNS8_u20z{Q9X?ki3Yt z zLu#jb3XKYLB;IMcM0U;`Uh*39bnR$AyBw+1!_gefR4a;m>w+I*BzV{2r$hWI42e~E zB&&qJ4o3~u1E3Zqp!8KTO09{!qR(RrtAIZl_ggR5?!ix_@5RqB@r*t^*_*(rc?8PX znlYkfRj+QrL%p)iIWG1AD~m~#2lrDji%SPMCwRbNt_~w~Wi_FGEq&6wl zqx?waZ9sGW_jn(0z7LN@R__|$4>A?*_==GD;=3kW2knx^F+kvV)#ctvu;?Q%4WM|K%+6oBk6F#SpoFGw( z6%G(AVk2QZ-ii+rya3^L*a~q-zRSU(It_ls!aI^h%Lf{T`0{q?Pl$~`^mzxcvmt9z z*7(m*FMCsjEPq^L%RaYE=9acLKU_l0K~ako7>js+2~ovYL$j$O+JRZ~7BO3-a!d&p zs%QqY^)%&f~?ugkm@1Pcx1gJEjj73hjGra`l&eu;=` z)X!FXHgCCgn=eE1N*3Na=gZzWt-K(N_Y9xGPa)c`)XirukMa3n2{{V(HZ6yiE5uVl zX%vqKO#vw64$$n`rWB4C;94)4kRYPs)&>e?zXA`&3To8{Bbqiis}|;8T6z3@iPgQ? z&-pA&CLX%+G;>JO<7)Gp^;M%7(s=2&h-ZuGlS&YT|H8*dd10~|O@$OS!1aPAhJNcY zE*)Cfn1;}rdJZzG<<(Cc)sO`z-QMJnhBRg@7`yDB6Mg!_+_tH3JL99=P!R3iiVC87K*QFUR9iYWX+_=h7qWA@A%bLg$1jg&HLE4de-GJo|!i)4?55 z{J33qCNqo|yKYr*qm?ffV)*#`rPqnm*ljX9QZ6uXbTR6zMMAp4b_Q;N`||SWHRH*0 z6vFN;wgT+~_<4o;*tlVMaK)FX$dn2?G6bTaG`#{ZBnnVGMudZwc6MN&D;th&a|vJ6 zn3G#n*b_0nh`6?~LC~|}EK8S>x^2qo+g0pIF!>?iY%-e~(^p#`+)y-|WF?IoDBkxh zmC|i%WZcJ%4^3!=i?GLBJZ@bu5u{ zR?J5TVlOVcd}3{iNi8y|i#0-MC@r(p9^JHJmjAITVPtAr_elHL%C19Dw25MOeUk9* zE#Zx(?gpgi3kc888mVZ2E8-(8-lI@5_Fb4s7jAOx0ESbqllcU49S=jl zp4NdX^p3n(UsmA#REVq_*+Cxsu-k(|N!?ec9z6xEfhg_gaztP}Gc%ox!0JS%#?3l| z5b>;VMW~UkDDiehPYQ`TQ$^g|B2%*TAv_;xm;fJ)KN#fPCXRmXi}vn_Al$`eeP6?; z8?B#NA73H+T&qsisILhT8#9vs)P3R920?4Y1JPqk;%FL0j1MKYP3bG2@xzN9Bsw(r z(>DM-d@i59WKsLi9h@yv#Zvru+mMj8D6DYl6PNk&?_^xSYZt_nFhnZ_YC5B1mvBK@ zg$HWKp#ym0ugbQ?)@dM5IU}U+?wd|N{S^#bmNpphrr(2ULG2$#xn$0QCjw#f%(qHk zffAH(CjgR72-a@HAYc9O%rPF)86kww2BaqDjR08z!tRx-fh3ZGWxyz7%p!e2|3^A_ z&d{tZ507?X1VrWw^Mj5eMrc1RyuR=;mk|udhc}}!FDtSfN%#=b(ejx!@#P2Dmhf=6E#% zwL(Vvi3%8F%K{aLg!(UzMIN<9Ih2Z9jaSqyZa2+#tgI$Dt&A%t9c#^15Og>*UFuj3 zWpRs~mZ_PPR{35b=1zl*X(pPM59eN9h7?={{W^jXHrb0|!azVslm9PjE#CWDD?&@f zxs(UkeXgt5!_8D?YJfQ7MfjQ8K)ciN5Et7F#lcc7?QGRsxhCA{p?}`jZUH{$4cZ=G zXNp$Dz4S?pA{1IsvAZUVG2`{7(3R`B^Ay;}9=14xFcSUR_?`XWK_!subx8&n50#CS z;tRu=^hL$D6QLY_&!N#Q*xa)08C$kSRV2;MBOK`%caPWU68zU{L+dF9A?LW2yrwI|sC@Mb@Bq-J8H)0DsRnm+IU07J4ojI-c9LlltT$`Edxtp>Yl8)mw>pSiUo;T$kC<0a!rwAYOM;Zp9ikwI6p@A?l{}9?>hPv{>^3iQ6Wp&X zKS?JdzdN7@wFy7fD^QDHnY2pi35T){WiMQQQt`1o)49f)YtM)x56^YW+rH9HwK8Jg>PEVe$ zI-?m+E*yjlxXW+soOlVJxf`%29NxjC_HtqvzC526uhw#mlchcnvzcyYPFjA-4LhuV zO3rO`t=t-~pmg+ns<};LI7mo!&2eEug-z5t<)`)d&_u?rhC@CB3FP!V%&WxRpuSK< zL?9ZbGl1vC@d1gme^w<=)uD1`K5RO;o1sYDQ2&7N+f#5{&uVim+3t1H{(<{p#3r~U zFVbz$)Q@A{p}eCUtaw@)2+e5dpy)T{DCv1K1LD3at1x4I68-L>#S7NY49MDst)hh^ z+Ft}Ht1ZAZLMD5TBY4xv4Q4@=b~vVw!?@ht=lSBWZTER?3zqncKuOXxbDV_U85GQ> zFIta%Hk#&LlDc_I0$$j#$vk0tdeWi1m0+8^F@>YMGD}QRK_(~xPG z>nWR8i;|Bi({4XM3zoiEcTJ-%6F;)N)zt8iEw8%28sCp5{+N1)rRu{b+Un;upmpV<6jK;qaXwV> z#YAbRFuvy_QrrQ)XHV&bca-MJ&ewEryDC0(0$YU?xkC0F(`qe`Z(`GC%a0DdLl-5~ z-w(FRc1#xRtz5u>w@WoM>@$%k*Ovego9+zN+_C+sO3UtPTw2`83D&YLNACtsBX8KX zhIH{~$G~b4%gKYMbHZ0GpOJZLpWoIEkJM4a@9S?2?2(l1bD!3e@z(p^Hk0%?Jdi>i~F3yJ<+34Ovz3+)^pPCG8YWpFoO{z;z~ zLMv9IE&>#JUcAo)8HUq`i{ORc*oNQ_vanx_H2UZ`^1pwdeHSN+BuN-tfCPhiJDK4E zI1#)s8pF4vfi(DZ3{vqj(W&UJ1hg7-gNorcF+gs7IwmQIOmr@~D#laUX9H_>c z1pH*-oK>UUlpt}>32~Oi4G7$QLxLOquXTYZ@&7N#SsnQ9Iar?D|4(gS_+Y# zFc}B~u}=k%tTFo8UNAxv)W1d7A9x&=Z-QBkxwUk=X32`iGpIGhHv>0btIM9+l=-xj z-Hy$EH2G1c)W|;;hJvT3reMr4*RJhb4w3^99*|O%N0ib~y}l;#Kta)Fyy~0?uPia6yr7E;5wiRPS1@UFp?aNN9F0+S0lm1-D<3%nw9N1N0;| z+=6Z=f_iG759RaUw||he{8VU0RZo^@%nLVq1fJ)Kncof%h_K*1&0P32wAd^{lV&4c z!1IPMcby>@@MGuWt0rhV!?7b0`S}5V;R8;EjU_tL)~!kn)@c+(%qPoO56|chZ+VSTRxY-M zTUaZxSlhN8aPw}xcN^$-8zdbTAD>XQ(g0D&)($L#ypT_7uTjR$&I(Q zJ;0r|axd|(daVO9*IdN=<>H67J2ya_9~JmBmKwsHvm zR@p1nI0y7k`Dm?tRw%Y4DnIQKmfS!$Pd-@ap1O-(f*RT_+o%G7N;NwA+)SZp;t3k{ zmdedtPVleNe+ZOlGzLJW8v_6$5{=RT5vj%`^fL?#6_6igPzA=YKl<=X^lkA9UIj+4 zLxT8oVEiTUQ$W1c?E2WCAhyu7g}8UZQjh%x&=|Sjgnk|~v|F-K8(=8i=m~&IG`69) z#+VtRx8gK?ySSo~stPkZ#D^h=?$gavhwjtQBZgAZ&Etm#$TX^GvTKnr>dXN&??yY8y?G*=q`|OWKHGl@GqL?5sBR zh2~~xaaZh26b`FYwfc@Q=Mr>y*?_|t-L68cAzo6{&4rJ4DK4XM45)4cQa7_ z7m4BVePT$+ggU{?;QPb?oHJ~6f5@*`Wuv&L?Y@ml?)f?yOVxtd#Zp9}Q>`~rE28y@ zL4(61zSXDm*SV83_Jw5f5W$k6`EU{*93uvP#vv~Ucnya&*L55iubSTdooe`9@(~J6 zo(S!kG<0he_gl1~M-tYhwX}X>_|lvD@pJ6bX2lPDoyfs2zC{yN2q-Ef#u3=?bdlwW z^E*0gp)_EK9}O?Ew0}(RNzXIgnFm-^-EevtxYA^&tlIYc{00F)}{j%_JgYbz#XSQm}OVy1d?&!pZPvX)(qdw|6$;^>eSjB0>j@ z+LQwFg~=H~nn=uAB}5XyY;gj#zTMa}<=8ChWfFntlg&|ANaFmDcK~8k(rAo)@@Eg& zv~xpE&^$UwfhGvdp%aa(RH20^eSNNI4SUTo`!g2)xr-SMIIPj4vl`ioSzNm1wcq+0 zpohZ6D!RS1GjpE{uhfnfi^NdPukG`M8lPA80T2lQxj-&Z@VS z8wOF;T|*xhqQpVB55Snjt2dCjHE#Rsa@3DIA&i24&=Y0?h4Byw1U&y7Zk0?lv&S=jU$F~KoMp{W0fQX~Iq@rdrEEYv( znmJ_myu{Sw&urqpMLR0=CxF-hVz~+9=5X#u@yN?v|7tv z^8B!Uv-sqZm+###4)4wPiS}hUlTi)dEvhRaA`v0{{qV%^ME?E56aOmz@F>Nf0Kc={ z_k91~k_g|p{Ko$OsrWmud{2h|Ej4gk|D?wM)c&1Oz2~0)7GHQW_&@Fc$3*|B{yVLC zPrLpt-S8sgpGeq05q{^|?&;6J#T(@>g#RN$|5W{*UAiaZ{uTzbU#h=zNq;K;PO;n* zP=AXf`Y+{wLr49I^ZW6^Jt6bAl*7XW?tyzF{y%!=Pn6#`sQ-$B509pQK>2m>@F&Xe z#ofQ6oWcj^A5ea6ZU031eI@u;lsmG2MEO+~{)zJYWcgQ=2h{(F@@oxIS3-e534?%u P1Apnl15kwa{_cMOINBW} literal 0 HcmV?d00001 diff --git a/whiterabbit/src/test/resources/scan_data/cost-header.csv b/whiterabbit/src/test/resources/scan_data/cost-header.csv new file mode 100644 index 00000000..f6825044 --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/cost-header.csv @@ -0,0 +1,35 @@ +cost_id,cost_event_id,cost_domain_id,cost_type_concept_id,currency_concept_id,total_charge,total_cost,total_paid,paid_by_payer,paid_by_patient,paid_patient_copay,paid_patient_coinsurance,paid_patient_deductible,paid_by_primary,paid_ingredient_cost,paid_dispensing_fee,payer_plan_period_id,amount_allowed,revenue_code_concept_id,reveue_code_source_value,drg_concept_id,drg_source_value +10791,1,Drug,0,44818668,,,180,,0,,0,,,,,,,0,,0, +10792,2,Drug,0,44818668,,,70,,70,,70,,,,,,,0,,0, +10793,3,Drug,0,44818668,,,60,,0,,0,,,,,,,0,,0, +10794,4,Drug,0,44818668,,,130,,40,,40,,,,,,,0,,0, +10795,6,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10796,8,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10797,10,Drug,0,44818668,,,120,,0,,0,,,,,,,0,,0, +10798,11,Drug,0,44818668,,,40,,10,,10,,,,,,,0,,0, +10799,12,Drug,0,44818668,,,110,,40,,40,,,,,,,0,,0, +10800,14,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10801,18,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10802,19,Drug,0,44818668,,,10,,0,,0,,,,,,,0,,0, +10803,21,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10804,25,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10805,27,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10806,28,Drug,0,44818668,,,0,,10,,10,,,,,,,0,,0, +10807,29,Drug,0,44818668,,,30,,10,,10,,,,,,,0,,0, +10808,31,Drug,0,44818668,,,350,,0,,0,,,,,,,0,,0, +10809,33,Drug,0,44818668,,,10,,10,,10,,,,,,,0,,0, +10810,35,Drug,0,44818668,,,570,,80,,80,,,,,,,0,,0, +10811,37,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10812,38,Drug,0,44818668,,,150,,0,,0,,,,,,,0,,0, +10813,41,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10814,42,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10815,45,Drug,0,44818668,,,70,,0,,0,,,,,,,0,,0, +10816,51,Drug,0,44818668,,,80,,0,,0,,,,,,,0,,0, +10817,52,Drug,0,44818668,,,120,,0,,0,,,,,,,0,,0, +10818,53,Drug,0,44818668,,,70,,70,,70,,,,,,,0,,0, +10819,55,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10820,56,Drug,0,44818668,,,70,,170,,170,,,,,,,0,,0, +10821,58,Drug,0,44818668,,,70,,0,,0,,,,,,,0,,0, +10822,61,Drug,0,44818668,,,160,,0,,0,,,,,,,0,,0, +10823,62,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10824,63,Drug,0,44818668,,,350,,10,,10,,,,,,,0,,0, diff --git a/whiterabbit/src/test/resources/scan_data/cost-no-header.csv b/whiterabbit/src/test/resources/scan_data/cost-no-header.csv new file mode 100644 index 00000000..fa8fa46a --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/cost-no-header.csv @@ -0,0 +1,34 @@ +10791,1,Drug,0,44818668,,,180,,0,,0,,,,,,,0,,0, +10792,2,Drug,0,44818668,,,70,,70,,70,,,,,,,0,,0, +10793,3,Drug,0,44818668,,,60,,0,,0,,,,,,,0,,0, +10794,4,Drug,0,44818668,,,130,,40,,40,,,,,,,0,,0, +10795,6,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10796,8,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10797,10,Drug,0,44818668,,,120,,0,,0,,,,,,,0,,0, +10798,11,Drug,0,44818668,,,40,,10,,10,,,,,,,0,,0, +10799,12,Drug,0,44818668,,,110,,40,,40,,,,,,,0,,0, +10800,14,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10801,18,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10802,19,Drug,0,44818668,,,10,,0,,0,,,,,,,0,,0, +10803,21,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10804,25,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10805,27,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10806,28,Drug,0,44818668,,,0,,10,,10,,,,,,,0,,0, +10807,29,Drug,0,44818668,,,30,,10,,10,,,,,,,0,,0, +10808,31,Drug,0,44818668,,,350,,0,,0,,,,,,,0,,0, +10809,33,Drug,0,44818668,,,10,,10,,10,,,,,,,0,,0, +10810,35,Drug,0,44818668,,,570,,80,,80,,,,,,,0,,0, +10811,37,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10812,38,Drug,0,44818668,,,150,,0,,0,,,,,,,0,,0, +10813,41,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10814,42,Drug,0,44818668,,,20,,0,,0,,,,,,,0,,0, +10815,45,Drug,0,44818668,,,70,,0,,0,,,,,,,0,,0, +10816,51,Drug,0,44818668,,,80,,0,,0,,,,,,,0,,0, +10817,52,Drug,0,44818668,,,120,,0,,0,,,,,,,0,,0, +10818,53,Drug,0,44818668,,,70,,70,,70,,,,,,,0,,0, +10819,55,Drug,0,44818668,,,0,,0,,0,,,,,,,0,,0, +10820,56,Drug,0,44818668,,,70,,170,,170,,,,,,,0,,0, +10821,58,Drug,0,44818668,,,70,,0,,0,,,,,,,0,,0, +10822,61,Drug,0,44818668,,,160,,0,,0,,,,,,,0,,0, +10823,62,Drug,0,44818668,,,30,,0,,0,,,,,,,0,,0, +10824,63,Drug,0,44818668,,,350,,10,,10,,,,,,,0,,0, diff --git a/whiterabbit/src/test/resources/scan_data/cost.csv b/whiterabbit/src/test/resources/scan_data/cost.csv deleted file mode 100644 index 7904c62e..00000000 --- a/whiterabbit/src/test/resources/scan_data/cost.csv +++ /dev/null @@ -1,34 +0,0 @@ -10791 1 Drug 0 44818668 180 0 0 0 0 -10792 2 Drug 0 44818668 70 70 70 0 0 -10793 3 Drug 0 44818668 60 0 0 0 0 -10794 4 Drug 0 44818668 130 40 40 0 0 -10795 6 Drug 0 44818668 30 0 0 0 0 -10796 8 Drug 0 44818668 20 0 0 0 0 -10797 10 Drug 0 44818668 120 0 0 0 0 -10798 11 Drug 0 44818668 40 10 10 0 0 -10799 12 Drug 0 44818668 110 40 40 0 0 -10800 14 Drug 0 44818668 30 0 0 0 0 -10801 18 Drug 0 44818668 0 0 0 0 0 -10802 19 Drug 0 44818668 10 0 0 0 0 -10803 21 Drug 0 44818668 30 0 0 0 0 -10804 25 Drug 0 44818668 20 0 0 0 0 -10805 27 Drug 0 44818668 20 0 0 0 0 -10806 28 Drug 0 44818668 0 10 10 0 0 -10807 29 Drug 0 44818668 30 10 10 0 0 -10808 31 Drug 0 44818668 350 0 0 0 0 -10809 33 Drug 0 44818668 10 10 10 0 0 -10810 35 Drug 0 44818668 570 80 80 0 0 -10811 37 Drug 0 44818668 0 0 0 0 0 -10812 38 Drug 0 44818668 150 0 0 0 0 -10813 41 Drug 0 44818668 0 0 0 0 0 -10814 42 Drug 0 44818668 20 0 0 0 0 -10815 45 Drug 0 44818668 70 0 0 0 0 -10816 51 Drug 0 44818668 80 0 0 0 0 -10817 52 Drug 0 44818668 120 0 0 0 0 -10818 53 Drug 0 44818668 70 70 70 0 0 -10819 55 Drug 0 44818668 0 0 0 0 0 -10820 56 Drug 0 44818668 70 170 170 0 0 -10821 58 Drug 0 44818668 70 0 0 0 0 -10822 61 Drug 0 44818668 160 0 0 0 0 -10823 62 Drug 0 44818668 30 0 0 0 0 -10824 63 Drug 0 44818668 350 10 10 0 0 diff --git a/whiterabbit/src/test/resources/scan_data/create_data_postgresql.sql b/whiterabbit/src/test/resources/scan_data/create_data_postgresql.sql index 23cd38f3..f7c7d66e 100644 --- a/whiterabbit/src/test/resources/scan_data/create_data_postgresql.sql +++ b/whiterabbit/src/test/resources/scan_data/create_data_postgresql.sql @@ -51,5 +51,5 @@ CREATE TABLE cost ; -COPY COST FROM '/scan_data/cost.csv' DELIMITER E'\t' CSV ENCODING 'UTF8'; -COPY PERSON FROM '/scan_data/person.csv' DELIMITER E'\t' CSV ENCODING 'UTF8'; +COPY COST FROM '/scan_data/cost-no-header.csv' DELIMITER ',' CSV ENCODING 'UTF8'; +COPY PERSON FROM '/scan_data/person-no-header.csv' DELIMITER ',' CSV ENCODING 'UTF8'; diff --git a/whiterabbit/src/test/resources/scan_data/create_data_snowflake.sql b/whiterabbit/src/test/resources/scan_data/create_data_snowflake.sql new file mode 100644 index 00000000..3b53cf1e --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/create_data_snowflake.sql @@ -0,0 +1,32 @@ +// +// To be able to use the configured snowflake test environment, make sure that the role and grant +// statements below have been exectuded, using the correct snowflake username for <> +// +//create role if not exists testrole; +//grant usage on database test to role testrole; +//grant usage on schema test.wr_test to role testrole; +//grant ALL PRIVILEGES on schema test.wr_test to role testrole; +//grant role testrole to user <>; + +//use schema test.wr_test; + +DROP TABLE IF EXISTS wr_test.person; +DROP TABLE IF EXISTS wr_test.cost; + +CREATE TABLE wr_test.cost (cost_id BIGINT, cost_event_id BIGINT, cost_domain_id STRING, cost_type_concept_id BIGINT, currency_concept_id BIGINT, total_charge NUMERIC, total_cost NUMERIC, total_paid NUMERIC, paid_by_payer NUMERIC, paid_by_patient NUMERIC, paid_patient_copay NUMERIC, paid_patient_coinsurance NUMERIC, paid_patient_deductible NUMERIC, paid_by_primary NUMERIC, paid_ingredient_cost NUMERIC, paid_dispensing_fee NUMERIC, payer_plan_period_id BIGINT, amount_allowed NUMERIC, revenue_code_concept_id BIGINT, reveue_code_source_value STRING, drg_concept_id BIGINT, drg_source_value STRING); + +CREATE TABLE wr_test.person (person_id BIGINT, gender_concept_id BIGINT, year_of_birth BIGINT, month_of_birth BIGINT, day_of_birth BIGINT, birth_datetime TIMESTAMP, race_concept_id BIGINT, ethnicity_concept_id BIGINT, location_id BIGINT, provider_id BIGINT, care_site_id BIGINT, person_source_value STRING, gender_source_value STRING, gender_source_concept_id BIGINT, race_source_value STRING, race_source_concept_id BIGINT, ethnicity_source_value STRING, ethnicity_source_concept_id BIGINT); + +REMOVE @~ pattern=".*csv.gz"; + +put file:///scan_data/cost-no-header.csv @~; + +put file:///scan_data/person-no-header.csv @~; + +CREATE OR REPLACE FILE FORMAT my_csv_format TYPE = 'csv' FIELD_DELIMITER = ','; + +COPY INTO cost from @~/cost-no-header.csv.gz FILE_FORMAT = (FORMAT_NAME = 'my_csv_format'); + +COPY INTO person from @~/person-no-header.csv.gz FILE_FORMAT = (FORMAT_NAME = 'my_csv_format'); + +REMOVE @~ pattern=".*csv.gz"; \ No newline at end of file diff --git a/whiterabbit/src/test/resources/scan_data/person-header.csv b/whiterabbit/src/test/resources/scan_data/person-header.csv new file mode 100644 index 00000000..2661396a --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/person-header.csv @@ -0,0 +1,31 @@ +person_id,gender_concept_id,year_of_birth,month_of_birth,day_of_birth,birth_datetime,race_concept_id,ethnicity_concept_id,location_id,provider_id,care_site_id,person_source_value,gender_source_value,gender_source_concept_id,race_source_value,race_source_concept_id,ethnicity_source_value,ethnicity_source_concept_id +1,8507,1923,5,1,,8527,38003564,1,,,00013D2EFD8E45D1,1,,1,,1, +2,8507,1943,1,1,,8527,38003564,2,,,00016F745862898F,1,,1,,1, +3,8532,1936,9,1,,8527,38003564,3,,,0001FDD721E223DC,2,,1,,1, +4,8507,1941,6,1,,0,38003563,4,,,00021CA6FF03E670,1,,5,,5, +5,8507,1936,8,1,,8527,38003564,5,,,00024B3D2352D2D0,1,,1,,1, +6,8507,1943,10,1,,8516,38003564,6,,,0002DAE1C81CC70D,1,,2,,2, +7,8507,1922,7,1,,8527,38003564,7,,,0002F28CE057345B,1,,1,,1, +8,8507,1935,9,1,,8527,38003564,8,,,000308435E3E5B76,1,,1,,1, +9,8532,1976,9,1,,8527,38003564,9,,,000345A39D4157C9,2,,1,,1, +10,8532,1938,10,1,,8516,38003564,10,,,00036A21B65B0206,2,,2,,2, +11,8532,1934,2,1,,8527,38003564,11,,,000489E7EAAD463F,2,,1,,1, +12,8507,1929,6,1,,8527,38003564,12,,,00048EF1F4791C68,1,,1,,1, +13,8532,1936,7,1,,8527,38003564,13,,,0004F0ABD505251D,2,,1,,1, +14,8507,1934,5,1,,8527,38003564,14,,,00052705243EA128,1,,1,,1, +15,8532,1936,3,1,,8527,38003564,15,,,00070B63745BE497,2,,1,,1, +16,8507,1934,1,1,,8527,38003564,16,,,0007E57CC13CE880,1,,1,,1, +17,8532,1919,9,1,,8516,38003564,17,,,0007F12A492FD25D,2,,2,,2, +18,8532,1919,10,1,,8516,38003564,18,,,000A005BA0BED3EA,2,,2,,2, +19,8532,1942,7,1,,8527,38003564,19,,,000B4662348C35B4,2,,1,,1, +20,8507,1938,4,1,,8527,38003564,20,,,000B97BA2314E971,1,,1,,1, +21,8507,1932,8,1,,8516,38003564,21,,,000C7486B11E7030,1,,2,,2, +23,8507,1932,7,1,,8527,38003564,23,,,000DDD364C46E2C6,1,,1,,1, +25,8507,1965,4,1,,8527,38003564,25,,,00108066CA1FACCE,1,,1,,1, +26,8532,1939,12,1,,8527,38003564,26,,,0010D6F80D245D62,2,,1,,1, +27,8532,1940,4,1,,8527,38003564,27,,,0011714C14B52EEB,2,,1,,1, +28,8507,1937,10,1,,8527,38003564,28,,,0011CB1FE23E91AF,1,,1,,1, +29,8507,1938,4,1,,8527,38003564,29,,,0012AFEEC379A69D,1,,1,,1, +30,8532,1959,11,1,,8527,38003564,30,,,00131C35661B2926,2,,1,,1, +31,8532,1922,10,1,,8527,38003564,31,,,00139C345A104F72,2,,1,,1, +32,8532,1953,12,1,,8527,38003564,32,,,0013E139F1F37264,2,,1,,1, diff --git a/whiterabbit/src/test/resources/scan_data/person-no-header.csv b/whiterabbit/src/test/resources/scan_data/person-no-header.csv new file mode 100644 index 00000000..63d4629b --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/person-no-header.csv @@ -0,0 +1,30 @@ +1,8507,1923,5,1,,8527,38003564,1,,,00013D2EFD8E45D1,1,,1,,1, +2,8507,1943,1,1,,8527,38003564,2,,,00016F745862898F,1,,1,,1, +3,8532,1936,9,1,,8527,38003564,3,,,0001FDD721E223DC,2,,1,,1, +4,8507,1941,6,1,,0,38003563,4,,,00021CA6FF03E670,1,,5,,5, +5,8507,1936,8,1,,8527,38003564,5,,,00024B3D2352D2D0,1,,1,,1, +6,8507,1943,10,1,,8516,38003564,6,,,0002DAE1C81CC70D,1,,2,,2, +7,8507,1922,7,1,,8527,38003564,7,,,0002F28CE057345B,1,,1,,1, +8,8507,1935,9,1,,8527,38003564,8,,,000308435E3E5B76,1,,1,,1, +9,8532,1976,9,1,,8527,38003564,9,,,000345A39D4157C9,2,,1,,1, +10,8532,1938,10,1,,8516,38003564,10,,,00036A21B65B0206,2,,2,,2, +11,8532,1934,2,1,,8527,38003564,11,,,000489E7EAAD463F,2,,1,,1, +12,8507,1929,6,1,,8527,38003564,12,,,00048EF1F4791C68,1,,1,,1, +13,8532,1936,7,1,,8527,38003564,13,,,0004F0ABD505251D,2,,1,,1, +14,8507,1934,5,1,,8527,38003564,14,,,00052705243EA128,1,,1,,1, +15,8532,1936,3,1,,8527,38003564,15,,,00070B63745BE497,2,,1,,1, +16,8507,1934,1,1,,8527,38003564,16,,,0007E57CC13CE880,1,,1,,1, +17,8532,1919,9,1,,8516,38003564,17,,,0007F12A492FD25D,2,,2,,2, +18,8532,1919,10,1,,8516,38003564,18,,,000A005BA0BED3EA,2,,2,,2, +19,8532,1942,7,1,,8527,38003564,19,,,000B4662348C35B4,2,,1,,1, +20,8507,1938,4,1,,8527,38003564,20,,,000B97BA2314E971,1,,1,,1, +21,8507,1932,8,1,,8516,38003564,21,,,000C7486B11E7030,1,,2,,2, +23,8507,1932,7,1,,8527,38003564,23,,,000DDD364C46E2C6,1,,1,,1, +25,8507,1965,4,1,,8527,38003564,25,,,00108066CA1FACCE,1,,1,,1, +26,8532,1939,12,1,,8527,38003564,26,,,0010D6F80D245D62,2,,1,,1, +27,8532,1940,4,1,,8527,38003564,27,,,0011714C14B52EEB,2,,1,,1, +28,8507,1937,10,1,,8527,38003564,28,,,0011CB1FE23E91AF,1,,1,,1, +29,8507,1938,4,1,,8527,38003564,29,,,0012AFEEC379A69D,1,,1,,1, +30,8532,1959,11,1,,8527,38003564,30,,,00131C35661B2926,2,,1,,1, +31,8532,1922,10,1,,8527,38003564,31,,,00139C345A104F72,2,,1,,1, +32,8532,1953,12,1,,8527,38003564,32,,,0013E139F1F37264,2,,1,,1, diff --git a/whiterabbit/src/test/resources/scan_data/person.csv b/whiterabbit/src/test/resources/scan_data/person.csv deleted file mode 100644 index e10b61b0..00000000 --- a/whiterabbit/src/test/resources/scan_data/person.csv +++ /dev/null @@ -1,30 +0,0 @@ -1 8507 1923 5 1 8527 38003564 1 00013D2EFD8E45D1 1 1 1 -2 8507 1943 1 1 8527 38003564 2 00016F745862898F 1 1 1 -3 8532 1936 9 1 8527 38003564 3 0001FDD721E223DC 2 1 1 -4 8507 1941 6 1 0 38003563 4 00021CA6FF03E670 1 5 5 -5 8507 1936 8 1 8527 38003564 5 00024B3D2352D2D0 1 1 1 -6 8507 1943 10 1 8516 38003564 6 0002DAE1C81CC70D 1 2 2 -7 8507 1922 7 1 8527 38003564 7 0002F28CE057345B 1 1 1 -8 8507 1935 9 1 8527 38003564 8 000308435E3E5B76 1 1 1 -9 8532 1976 9 1 8527 38003564 9 000345A39D4157C9 2 1 1 -10 8532 1938 10 1 8516 38003564 10 00036A21B65B0206 2 2 2 -11 8532 1934 2 1 8527 38003564 11 000489E7EAAD463F 2 1 1 -12 8507 1929 6 1 8527 38003564 12 00048EF1F4791C68 1 1 1 -13 8532 1936 7 1 8527 38003564 13 0004F0ABD505251D 2 1 1 -14 8507 1934 5 1 8527 38003564 14 00052705243EA128 1 1 1 -15 8532 1936 3 1 8527 38003564 15 00070B63745BE497 2 1 1 -16 8507 1934 1 1 8527 38003564 16 0007E57CC13CE880 1 1 1 -17 8532 1919 9 1 8516 38003564 17 0007F12A492FD25D 2 2 2 -18 8532 1919 10 1 8516 38003564 18 000A005BA0BED3EA 2 2 2 -19 8532 1942 7 1 8527 38003564 19 000B4662348C35B4 2 1 1 -20 8507 1938 4 1 8527 38003564 20 000B97BA2314E971 1 1 1 -21 8507 1932 8 1 8516 38003564 21 000C7486B11E7030 1 2 2 -23 8507 1932 7 1 8527 38003564 23 000DDD364C46E2C6 1 1 1 -25 8507 1965 4 1 8527 38003564 25 00108066CA1FACCE 1 1 1 -26 8532 1939 12 1 8527 38003564 26 0010D6F80D245D62 2 1 1 -27 8532 1940 4 1 8527 38003564 27 0011714C14B52EEB 2 1 1 -28 8507 1937 10 1 8527 38003564 28 0011CB1FE23E91AF 1 1 1 -29 8507 1938 4 1 8527 38003564 29 0012AFEEC379A69D 1 1 1 -30 8532 1959 11 1 8527 38003564 30 00131C35661B2926 2 1 1 -31 8532 1922 10 1 8527 38003564 31 00139C345A104F72 2 1 1 -32 8532 1953 12 1 8527 38003564 32 0013E139F1F37264 2 1 1 diff --git a/whiterabbit/src/test/resources/scan_data/snowflake.ini.template b/whiterabbit/src/test/resources/scan_data/snowflake.ini.template new file mode 100644 index 00000000..ab12cd68 --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/snowflake.ini.template @@ -0,0 +1,16 @@ +# Usage: dist/bin/whiteRabbit -ini +WORKING_FOLDER = %WORKING_FOLDER% # Path to the folder where all output will be written +DATA_TYPE = Snowflake # "Delimited text files", "MySQL", "Oracle", "SQL Server", "PostgreSQL", "MS Access", "Redshift", "BigQuery", "Azure", "Teradata", "SAS7bdat" +SNOWFLAKE_ACCOUNT = %SNOWFLAKE_ACCOUNT% # Name or address of the server. For Postgres, add the database name +SNOWFLAKE_USER = %SNOWFLAKE_USER% # User name for the database +SNOWFLAKE_PASSWORD = %SNOWFLAKE_PASSWORD% # Password for the database +SNOWFLAKE_WAREHOUSE = %SNOWFLAKE_WAREHOUSE% # Name of the data schema used +SNOWFLAKE_DATABASE = %SNOWFLAKE_DATABASE% +SNOWFLAKE_SCHEMA = %SNOWFLAKE_SCHEMA% +TABLES_TO_SCAN = * # Comma-delimited list of table names to scan. Use "*" (asterix) to include all tables in the database +SCAN_FIELD_VALUES = yes # Include the frequency of field values in the scan report? "yes" or "no" +MIN_CELL_COUNT = 5 # Minimum frequency for a field value to be included in the report +MAX_DISTINCT_VALUES = 1000 # Maximum number of distinct values per field to be reported +ROWS_PER_TABLE = 100000 # Maximum number of rows per table to be scanned for field values +CALCULATE_NUMERIC_STATS = no # Include average, standard deviation and quartiles in the scan report? "yes" or "no" +NUMERIC_STATS_SAMPLER_SIZE = 500 # Maximum number of rows used to calculate numeric statistics diff --git a/whiterabbit/src/test/resources/scan_data/tsv.ini.template b/whiterabbit/src/test/resources/scan_data/tsv.ini.template new file mode 100644 index 00000000..2e287355 --- /dev/null +++ b/whiterabbit/src/test/resources/scan_data/tsv.ini.template @@ -0,0 +1,14 @@ +WORKING_FOLDER = %WORKING_FOLDER% # Path to the folder where all output will be written +DATA_TYPE = Delimited text files # "Delimited text files", "MySQL", "Oracle", "SQL Server", "PostgreSQL", "MS Access", "Redshift", "BigQuery", "Azure", "Teradata", "SAS7bdat" +SERVER_LOCATION = 127.0.0.1/data_base_name # Name or address of the server. For Postgres, add the database name +USER_NAME = joe # User name for the database +PASSWORD = supersecret # Password for the database +DATABASE_NAME = schema_name # Name of the data schema used +DELIMITER = , # The delimiter that separates values +TABLES_TO_SCAN = * # Comma-delimited list of table names to scan. Use "*" (asterix) to include all tables in the database +SCAN_FIELD_VALUES = yes # Include the frequency of field values in the scan report? "yes" or "no" +MIN_CELL_COUNT = 5 # Minimum frequency for a field value to be included in the report +MAX_DISTINCT_VALUES = 1000 # Maximum number of distinct values per field to be reported +ROWS_PER_TABLE = 100000 # Maximum number of rows per table to be scanned for field values +CALCULATE_NUMERIC_STATS = no # Include average, standard deviation and quartiles in the scan report? "yes" or "no" +NUMERIC_STATS_SAMPLER_SIZE = 500 # Maximum number of rows used to calculate numeric statistics