Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: Add basic CI test pipelines #18

Merged
merged 2 commits into from
Feb 14, 2024
Merged

build: Add basic CI test pipelines #18

merged 2 commits into from
Feb 14, 2024

Conversation

sunchao
Copy link
Member

@sunchao sunchao commented Feb 13, 2024 •
edited
Loading

This adds some basic CI test pipelines for the project. Basically run tests in the Rust and Java/Scala side within the repo.

Closes #7

@sunchao
Copy link
Member Author

sunchao commented Feb 13, 2024

Need to find a way to test this. Also this only run tests in Linux for Spark 3.4 atm, and we need more combinations.

@sunchao sunchao force-pushed the add-ci branch 7 times, most recently from 9a6cd49 to 1d82548 Compare February 14, 2024 05:11
@sunchao sunchao marked this pull request as ready for review February 14, 2024 05:38
@sunchao
Copy link
Member Author

sunchao commented Feb 14, 2024

@viirya @andygrove this is ready now. Tested here.

sunchao
sunchao commented Feb 14, 2024
View reviewed changes
core/src/errors.rs
@@ -651,6 +651,7 @@ mod tests {
/// See [`object_panic_exception`] for a test which involves generating a panic and verifying
/// that the resulting stack trace includes the offending call.
#[test]
#[ignore]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the golden file for this test is not added yet, so ignore for now

sunchao
sunchao commented Feb 14, 2024
View reviewed changes
spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala
@@ -297,7 +297,7 @@ class CometTPCDSV1_4_PlanStabilitySuite extends CometPlanStabilitySuite {
new File(baseResourcePath, "approved-plans-v1_4").getAbsolutePath

tpcdsQueries.foreach { q =>
test(s"check simplified (tpcds-v1.4/$q)") {
ignore(s"check simplified (tpcds-v1.4/$q)") {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

depends on #10

viirya
viirya reviewed Feb 14, 2024
View reviewed changes
.github/workflows/pr_build.yml Outdated
run: |
cd core
# This is required to run some JNI related tests on the Rust side
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib:$JAVA_HOME/lib/server cargo test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RUST_BACKTRACE=1?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea added

viirya
viirya reviewed Feb 14, 2024
View reviewed changes
.github/workflows/pr_build.yml

- name: Run tests
run: |
SPARK_HOME=`pwd` ./mvnw clean install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we use ./mvnw verify in Makefile, do we need to do that?

Copy link
Member

@viirya viirya Feb 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need BOSON_CONF_DIR?

Copy link
Member Author

@sunchao sunchao Feb 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install covers verify: https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#a-build-lifecycle-is-made-up-of-phases

COMET_CONF_DIR is optional and by default, Rust test outputs are directed to stdout.

@viirya
Copy link
Member

viirya commented Feb 14, 2024

There are a log message like this:

Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar (245 kB at 2.3 MB/s)
Progress (1): 1.0/2.3 MB
Progress (1): 1.1/2.3 MB
Progress (1): 1.1/2.3 MB
...

It is really distracting for looking at the logs. Can we silent it?

@sunchao
Copy link
Member Author

sunchao commented Feb 14, 2024

It is really distracting for looking at the logs. Can we silent it?

Hmm perhaps we can cache the Maven dependencies so these will only show up in the first time. Let me check

@sunchao sunchao merged commit 311ef6b into apache:main Feb 14, 2024
@sunchao
Copy link
Member Author

sunchao commented Feb 14, 2024

Thanks! merged. Let me check if the Maven cache did work. I'll create followups if it doesn't.

@sunchao sunchao deleted the add-ci branch February 14, 2024 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andygrove andygrove andygrove approved these changes

@viirya viirya viirya approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create CI/CD pipelines
3 participants
@sunchao @viirya @andygrove