Merge/merging release 2.26.X into develop ver3 #1997

AdrianOlosutean · 2021-12-07T16:21:42Z

No description provided.

…1873) * 1868 statistics with missing counts and datasets missing proprties

* 1843 Home page with properties, side panel with missing counts and summary page for properties with tab containing datasets missing that particular property

* #1603 serde tests for CR and MT DataFrameFilters (mongo-bson-based serde tests for CR and MT DataFrameFilters, mongo-bson-based serde tests extended for CR with a blank mappingTableFilter)

…mport

Project config and management updates * poc issue template * CODEOWNERS update * developers update * Badges to README.md

1881 HyperConformance enceladus_info_version from payload

#1887 defaultTimestampTimeZone can be source type specific * `DefaultsByFormat` extends the `Defaults` trait, being able to read defaults from configuration files * `DefaultsByFormat` offers further granularity by first checking the format specific setting only then taking the global one * Basic `GlobalDefaults` are not configuration dependent anymore * Standardization now user `DefaultsByFormat` for its defaults, where rawFormat is used for format parameter * Switched to configuration path to be `enceladus.defaultTimestampTimeZone.default` and `enceladus.defaultTimestampTimeZone.[rawFormat]` respectively * `defaultTimestampTimeZone` is still supported/read as an obsolete fallback Co-authored-by: Daniel K <dk1844@gmail.com>

#1887 defaultTimestampTimeZone can be source type specific * rename of the configuration prefix from `enceladus.` to `standardization.`

* Upgrade of Atum to 3.6.0 * Writing the default time zones for timestamps and dates into _INFO file

…1912) * #1894 Spline S3 support via custom persistence factory `HadoopFsPersistenceFactory`. Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com>

…obrix 2.3.0

…les.

* #1927 - testing setup: set both spline _LINEAGE and atum _INFO to hdfs file permissions 733 -> the result on EMR HDFS was 711 (due to 022 umask there) -> evidence of working * #1927 - cleanup of test settings of 733 fs permissions * #1927 Atum final version 3.7.0 used instead of the snapshot (same code) * #1927 comment change * #1927 - default 644 FS permissions for both _INFO and _LINEAGE files.

* 1937 limit output file size

* Handling 0 partition case in repartitioning * .gitignore duplicate entry fixed

…ase-2.26.1-into-develop-ver3 # Conflicts: # dao/pom.xml # data-model/pom.xml # examples/pom.xml # menas/pom.xml # menas/src/main/scala/za/co/absa/enceladus/menas/controllers/LandingPageController.scala # menas/ui/components/dataset/conformanceRule/ConformanceRuleDialog.js # menas/ui/components/dataset/conformanceRule/ConformanceRuleForm.js # menas/ui/components/mappingTable/filterEdit/FilterEdit.js # menas/ui/components/mappingTable/filterEdit/filterTreeEdit.fragment.xml # menas/ui/css/style.css # menas/ui/index.html # menas/ui/npm-shrinkwrap.json # menas/ui/package.json # menas/ui/service/EntityDialog.js # menas/ui/service/FilterTreeUtils.js # menas/ui/service/RestDAO.js # migrations-cli/pom.xml # migrations/pom.xml # plugins-api/pom.xml # plugins-builtin/pom.xml # pom.xml # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/StatisticsController.scala # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/DatasetService.scala # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/PropertyDefinitionService.scala # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/StatisticsService.scala # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/utils/implicits/package.scala # rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/DatasetApiIntegrationSuite.scala # rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/repositories/DatasetRepositoryIntegrationSuite.scala # rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/repositories/StatisticsIntegrationSuite.scala # rest-api/src/test/scala/za/co/absa/enceladus/rest_api/services/DatasetServiceTest.scala # spark-jobs/pom.xml # spark-jobs/src/main/resources/reference.conf # spark-jobs/src/main/resources/spline.properties.template # spark-jobs/src/main/scala/za/co/absa/enceladus/common/CommonJobExecution.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/ConformanceExecution.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/HyperConformance.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/HyperConformanceAttributes.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/config/FilterFromConfig.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/streaming/InfoVersionFactory.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/standardization/StandardizationExecution.scala # spark-jobs/src/main/scala/za/co/absa/enceladus/standardization/interpreter/StandardizationInterpreter.scala # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/fixtures/StreamingFixture.scala # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/streaming/HyperConformanceIntegrationSuite.scala # utils/pom.xml

dk1844

LGTM (just read the code)

* Menas gets log4j via Spring, explicitly imported to ensure 2.15 is used * Upgraded Hyperdrive in SparkJobs which already uses log4j 2.15

…ase-2.26.1-into-develop-ver3 # Conflicts: # menas/pom.xml

sonarqubecloud · 2021-12-13T08:59:32Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
3.3% Duplication

dk1844

Still LGTM (just read the code)

benedeki · 2022-01-10T12:03:27Z

Just merge it, I am sure it will work 🤞 🤣

benedeki and others added 30 commits August 21, 2021 10:20

Update for next development version 2.24.0-SNAPSHOT

bc787c4

Suppress download noise in license check

6e8055a

Suppress compiler warning of obsolete Java (#1892)

a4998ad

1868 statistics with missing counts and datasets missing proprties (#…

c96f779

…1873) * 1868 statistics with missing counts and datasets missing proprties

1843 Summary page for properties (#1880)

758ac7d

* 1843 Home page with properties, side panel with missing counts and summary page for properties with tab containing datasets missing that particular property

Feature/1603 mapping table filtering general (#1879)

7c341bd

* #1603 serde tests for CR and MT DataFrameFilters (mongo-bson-based serde tests for CR and MT DataFrameFilters, mongo-bson-based serde tests extended for CR with a blank mappingTableFilter)

#1909 Increase the limit of columns shown in menas column selection

162e088

1903 Add validation for complex default values in mapping tables on i…

2614495

…mport

Project config and management updates (#1908)

c3bfd7e

Project config and management updates * poc issue template * CODEOWNERS update * developers update * Badges to README.md

1881 HyperConformance enceladus_info_version from payload (#1896)

0f5f4fe

1881 HyperConformance enceladus_info_version from payload

#1887 defaultTimestampTimeZone can be source type specific (#1916)

761c410

#1887 defaultTimestampTimeZone can be source type specific * rename of the configuration prefix from `enceladus.` to `standardization.`

#172 Save original timezone information in metadata file (#1900)

bdb40a1

* Upgrade of Atum to 3.6.0 * Writing the default time zones for timestamps and dates into _INFO file

#1894 HadoopFsPersistenceFactory - adding Spline S3 write support (#…

92d15d4

…1912) * #1894 Spline S3 support via custom persistence factory `HadoopFsPersistenceFactory`. Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com>

Update versions for release v2.24.0

c721c78

Merge branch 'develop'

27c6750

Update for next development version 2.25.0-SNAPSHOT

eecab71

#1926 Add executor extra java opts to helper scripts

f42e88b

#1931 Add switch for running kinit in helper scripts

62c08f6

#1882 Update Cobrix dependency to v.2.3.0

5d03ecc

#1882 Remove explicit "collapse_root" since it is the default since C…

cf44db8

…obrix 2.3.0

#1882 Update Cobrix to 2.4.1 and update Cobol test suite for ASCII fi…

3c9f7b1

…les.

#1882 Bump up Cobrix version to 2.4.2.

74247a8

1937 limit output file size (#1941)

7c5baa8

* 1937 limit output file size

1937 limit output file size

a91910b

1937 renamings + constants

d7bb8a5

1937 more conditions

8685d05

1937 rename params

9a2c771

1937 feedback + script params

adca1a4

benedeki and others added 11 commits November 23, 2021 12:34

Merge branch 'develop'

d4fe8cc

Update for next development version 2.27.0-SNAPSHOT

5ecf807

Update versions for hotfix v2.26.1

4498678

#1987: Repartitioning prevents processing of zero size files (#1990)

56c2c79

* Handling 0 partition case in repartitioning * .gitignore duplicate entry fixed

Merge branch 'hotfix/2.26.1'

16a4746

Update to hotfix version @{version}

e5b25e3

Merge branch 'hotfix/2.26.1' into develop

86dae90

Update for next development version 2.27.0-SNAPSHOT

c3aa5f9

#1987: Zero partitions test explanation (#1994)

fbfde4c

merge 2.26.1

0002376

AdrianOlosutean requested review from benedeki and dk1844 as code owners December 7, 2021 16:21

dk1844 previously approved these changes Dec 9, 2021

View reviewed changes

benedeki and others added 7 commits December 12, 2021 09:06

Update versions for hotfix v2.26.2

c82964e

#1998: Ensure log4j v2 at least 2.15.0 (#2000)

39a27f5

* Menas gets log4j via Spring, explicitly imported to ensure 2.15 is used * Upgraded Hyperdrive in SparkJobs which already uses log4j 2.15

Update to hotfix version @{version}

67d0900

Merge branch 'hotfix/2.26.2' into develop

cc87c43

Update for next development version 2.27.0-SNAPSHOT

0b58604

Merge remote-tracking branch 'origin/develop' into merge/merging-rele…

ad4e24a

…ase-2.26.1-into-develop-ver3 # Conflicts: # menas/pom.xml

log4j

ea50478

AdrianOlosutean dismissed dk1844’s stale review via ea50478 December 13, 2021 08:58

dk1844 approved these changes Dec 13, 2021

View reviewed changes

AdrianOlosutean changed the title ~~Merge/merging release 2.26.1 into develop ver3~~ Merge/merging release 2.26.X into develop ver3 Dec 21, 2021

AdrianOlosutean self-assigned this Dec 23, 2021

benedeki added the PR:no testing needed Only for PR - PR doesn't need to be tested by a tester (person) label Jan 10, 2022

AdrianOlosutean merged commit d9a72fc into develop-ver-3.0 Jan 10, 2022

AdrianOlosutean deleted the merge/merging-release-2.26.1-into-develop-ver3 branch January 10, 2022 12:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge/merging release 2.26.X into develop ver3 #1997

Merge/merging release 2.26.X into develop ver3 #1997

AdrianOlosutean commented Dec 7, 2021

dk1844 left a comment

sonarqubecloud bot commented Dec 13, 2021

dk1844 left a comment

benedeki commented Jan 10, 2022

Merge/merging release 2.26.X into develop ver3 #1997

Merge/merging release 2.26.X into develop ver3 #1997

Conversation

AdrianOlosutean commented Dec 7, 2021

dk1844 left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 13, 2021

dk1844 left a comment

Choose a reason for hiding this comment

benedeki commented Jan 10, 2022