Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multibuild support for SBT #121

Closed
dk1844 opened this issue Nov 11, 2021 · 0 comments · Fixed by #124
Closed

Multibuild support for SBT #121

dk1844 opened this issue Nov 11, 2021 · 0 comments · Fixed by #124

Comments

@dk1844
Copy link
Collaborator

dk1844 commented Nov 11, 2021

As a continued effort for SBT in Atum (#21), the possibility to properly multi-build Atum in SBT should be done.

Main focus of this task is:

  • correctly pair Scala version with Spark version and related dependencies, E.g.:
    • Scala 2.11 + Spark 2.4 + Json4s 3.5.3 ...
    • Scala 2.12 + Spark 3.1 + Json4s 3.7.0-M5 ...
dk1844 added a commit that referenced this issue Nov 22, 2021
@dk1844
#121 multiversion build (scala, spark, json4s). Hadoop2 still used
b642582
dk1844 added a commit that referenced this issue Nov 22, 2021
@dk1844
#121 build-sbt 1st attempt
8aa62c6
dk1844 added a commit that referenced this issue Nov 23, 2021
@dk1844
#121 hadoop3 used for Spark3/Scala2.12, sbt build excludes scala 2.12…
ba88beb 
… + Spark 2.

versions update: spark 2.4.7->8, scala 2.12.12->15
dk1844 added a commit that referenced this issue Nov 23, 2021
@dk1844
#121 hadoop2 only (same as mvn build)
58cbeb5
dk1844 added a commit that referenced this issue Nov 24, 2021
@dk1844
#121 sbt-build: hdfs-minicluster: hadoop3-build (junit4 added for tes…
4484858 
…t scope)
dk1844 added a commit that referenced this issue Nov 24, 2021
@dk1844
#121 licence check update
f317f43
dk1844 added a commit that referenced this issue Nov 25, 2021
@dk1844
#121 sbt github autobuild - examples included, too.
15b6d61
dk1844 added a commit that referenced this issue Nov 29, 2021
@dk1844
Feature/121 sbt cross comptilation (#124)
4a0d6ea 
* #121 multiversion build (scala, spark, json4s).
* #121 hadoop3 used for Spark3/Scala2.12
 - versions update: spark 2.4.7->8, scala 2.12.12->15
* #121 sbt github autobuild
@dk1844 dk1844 linked a pull request Nov 29, 2021 that will close this issue
Feature/121 sbt cross comptilation #124
Merged
@dk1844 dk1844 closed this as completed Nov 29, 2021
@dk1844 dk1844 mentioned this issue Nov 29, 2021
Closed
Zejnilovic added a commit that referenced this issue Apr 13, 2022
@Zejnilovic @benedeki
@dk1844 @lokm01
Merge master in release 4.0.0
0360819 
* #88: Add some files to configure the project (#89)
* #88: Add some files configure the project
* Git configuration
* Scalastyle support
* Ensured no Scalastyle errors
* Added CRLF for Windows *.bat and *.cmd files to .editorconfig
* Added Spark 3.1 build into the `build-all.sh` script
* Created Windows `build-all.cmd` script
* Upgrade from Spark 3.1.1 to 3.1.2 (fixes several issues of the previous version)
* `--no-transfer-progress` added to build.yml
* #97 Aggregate control type strategy (#107)
* #97 AggregateControlTypeStrategy suggested API for ControlMeasureBuilder usage
* #97 ControlMeasureBuilder.withAggregateColumn(s) implementations. Todo needs testing, cleanup & documentation update
MeasurementProcessor split into object/class to offer generic processing methods to be reusable.
* #97 ControlMeasureBuilder.withAggregateColumn(s) unit tests added (regression guard)
* #97 ControlMeasureBuilder.withAggregateColumn(s) in README.md, original only-default `cmBuilder.calculateMeasurement` removed
* [maven-release-plugin] prepare release v3.6.0
* [maven-release-plugin] prepare for next development iteration
* #97 readme update - ControlMeasureBuilder API (#110)
* #97 readme update (related to #97, too)
* #97 maven central version badge added
* Feature/113 info permissions config (#114)
* #113 atum info file permissions for hdfs loaded from `atum.hdfs.info.file.permissions` config value
  - tests use MiniDfsCluster to assert controlled correct behavior
  - test update (custom MiniDfsCluster with umask 000 allows max permissions)
  - HdfsFileUtils.DefaultFilePermissions is now publicly exposed; the user is expected to call compose the default and configured value it on his own by e.g.:
`HdfsFileUtils.getInfoFilePermissionsFromConfig().getOrElse(HdfsFileUtils.DefaultFilePermissions)`
* #77 Fix parameter handling bug in CreateInfoFileToolCSV (#78)

* #121 sbt cross comptilation
* #121 multiversion build (scala, spark, json4s).
* #121 hadoop3 used for Spark3/Scala2.12
* #121 sbt github autobuild
* #125 publish, pgp/gpg plugin added; sbt-sonatype howto referenced; model, parent prefixed by `atum-` to conform to the mvn publish, cleanup
* Upgrade dependencies and remove MiniDFSCluster
* GH Action fix
* Remove pom.xml files
* Add licence header and header check
* Fix examples

Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com>
Co-authored-by: Daniel K <dk1844@gmail.com>
Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/121 sbt cross comptilation AbsaOSS/atum
1 participant
@dk1844