Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML-26] Build for different spark version by -Pprofile #39

Merged
merged 10 commits into from
Apr 30, 2021

Conversation

bobjiang82
Copy link
Contributor

What changes were proposed in this pull request?

For supporting different Spark versions.

(Please fill in changes proposed in this fix)

  • build for different spark version by profile (e.g. -Pspark-3.0.2)
  • use spark-3.0.0 by default if profile is not specified
  • implemented KMeans 3.0.2

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?
https://github.com/oap-project/oap-mllib/issues

Then could you also rename pull request title and commit log in the following format?

[ML-${ISSUES_ID}] ${detailed message}

See also:

@xwu99 xwu99 linked an issue Mar 18, 2021 that may be closed by this pull request
@bobjiang82 bobjiang82 changed the title Build for different spark version by -Pprofile [ML-26] Build for different spark version by -Pprofile Mar 19, 2021
@github-actions
Copy link

#26

mllib-dal/pom.xml Show resolved Hide resolved
@bobjiang82 bobjiang82 requested a review from xwu99 March 29, 2021 02:50
mllib-dal/pom.xml Outdated Show resolved Hide resolved
@carsonwang
Copy link
Collaborator

@xwu99 , any further comments on this?

@carsonwang
Copy link
Collaborator

@bobjiang82 , can we also add support for 3.0.1? OAP will probably have a minor release for 3.0.1 so all components will need to support it.

@xwu99
Copy link
Collaborator

xwu99 commented Apr 19, 2021

@bobjiang82 Could you merge changes from master as I have some updates in #44. And could you pls also add 3.0.1 support as Carson mentioned. I also have added unit test and cluster test in CI. You need to check all profiles passed CI.

reorganize folders for build by profile

update KMeans DAL for profile spark-3.0.2

update KMeans DAL for profile spark-3.1.1; update PCA DAL for spark-3.0.2 and spark-3.1.1.

Update oap-mllib-ci.yml

Add tests for building by profile

Update pom.xml

remove duplicated <spark.version>3.0.0</spark.version> as it is set in the default activated profile

updated README.md
@bobjiang82
Copy link
Contributor Author

Rebased and added spark v3.0.1 support.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
dev/ci-build.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
mllib-dal/build.sh Outdated Show resolved Hide resolved
dev/ci-build.sh Outdated Show resolved Hide resolved
dev/ci-test.sh Outdated Show resolved Hide resolved
dev/ci-build.sh Show resolved Hide resolved
@xwu99 xwu99 merged commit a2df040 into oap-project:master Apr 30, 2021
@xwu99
Copy link
Collaborator

xwu99 commented Apr 30, 2021

@bobjiang82 Merged! Thanks for your work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[PIP] Support Spark 3.0.1 / 3.0.2 and upcoming 3.1.1
3 participants