[SPARK-35483][INFRA] Add docker-integration-tests to run-tests.py and GA. #32631

sarutak · 2021-05-22T06:34:12Z

What changes were proposed in this pull request?

This PR proposes to add docker-integratin-tests to run-tests.py and GA.
doker-integration-tests can't run if docker is not installed so it run only if docker-integration-tests is specified with --module.

Why are the changes needed?

CI for docker-integration-tests is absent for now.

Does this PR introduce any user-facing change?

GA.

How was this patch tested?

…integration-test-ga

sarutak · 2021-05-22T06:35:47Z

cc: @maropu
As I comment in JIRA, I've been working on this issue.
This seems different from your solution in #32620 . What do you think?

sarutak · 2021-05-22T07:08:50Z

cc: @dongjoon-hyun too.

dev/sparktestsupport/modules.py

.github/workflows/build_and_test.yml

dongjoon-hyun

BTW, this is not a blocker for Apache Spark 3.1.2. If you don't mind, let's hold on this for a while.

HyukjinKwon · 2021-05-23T05:21:29Z

I think we can merge it into master first, and backport after the RC of 3.1.2.

dev/run-tests.py

sarutak · 2021-05-23T05:35:40Z

@dongjoon-hyun @HyukjinKwon

I think we can merge it into master first, and backport after the RC of 3.1.2.

BTW, this is not a blocker for Apache Spark 3.1.2. If you don't mind, let's hold on this for a while.

I agree. I think this is not a blocker and better to merge into master first.

…integration-test-ga

… image for docker integration tests ### What changes were proposed in this pull request? This PR adds a feature for removing pulled container image after every docker integration test finish. This feature is enabled by the new propoerty `spark.tes.docker.removePulledImage`. ### Why are the changes needed? For idempotent. I'm trying to add docker integration tests to GA in SPARK-35483 (#32631) but I noticed that `jdbc.OracleIntegrationSuite` consistently fails(https://github.com/sarutak/spark/runs/2646707235?check_suite_focus=true). I investigated the reason and I found it's short of the storage capacity of the host on GA. ``` ORACLE PASSWORD FOR SYS AND SYSTEM: oracle The location '/opt/oracle' specified for database files has insufficient space. Database creation needs at least '4.5GB' disk space. Specify a different database file destination that has enough space in the configuration file '/etc/sysconfig/oracle-xe-18c.conf'. mv: cannot stat '/opt/oracle/product/18c/dbhomeXE/dbs/spfileXE.ora': No such file or directory mv: cannot stat '/opt/oracle/product/18c/dbhomeXE/dbs/orapwXE': No such file or directory ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this information is not available for the current user ID . You can set ORACLE_BASE manually if it is required. Resetting ORACLE_BASE to its previous value or ORACLE_HOME The Oracle base remains unchanged with value /opt/oracle ##################################### ########### E R R O R ############### DATABASE SETUP WAS NOT SUCCESSFUL! Please check output for further info! ########### E R R O R ############### ##################################### The following output is now a tail of the alert.log: tail: cannot open '/opt/oracle/diag/rdbms/*/*/trace/alert*.log' for reading: No such file or directory tail: no files remaining ``` With this feature, pulled container image is removed and keep the capacity for `jdbc.OracleIntegrationSuite` in GA. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed the following things. * A container image which is absent in the local repository is removed after test finished if `spark.test.container.removePulledImage` is `true`. * A container image which is present in the local repository is not removed after the finished even if `spark.test.container.removePulledImage` is `true`. * A container image is not removed regardless of presence of the container image in the local repository even if `spark.test.container.removePulledImage` is `true`. Closes #32652 from sarutak/docker-image-rm. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…integration-test-ga.tmp2

dev/sparktestsupport/modules.py

HyukjinKwon

I skimmed - looks good otherwise

dev/sparktestsupport/modules.py

dev/run-tests.py

...r-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerIntegrationFunSuite.scala

…testBody)

SparkQA · 2021-05-27T16:05:14Z

Test build #139020 has finished for PR 32631 at commit 692d95d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-05-27T22:56:18Z

Merged to master.

I know this GA and Docker integration things are sort of tricky to handle .. Thanks @sarutak for working on this!

dongjoon-hyun · 2021-05-28T00:42:32Z

Thank you, @sarutak and @HyukjinKwon . Ya, this is really nice to have.

HyukjinKwon · 2021-05-28T05:28:13Z

.github/workflows/build_and_test.yml

+      run: |
+        git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+        git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+        git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit"


Oh, we should add echo "::set-output name=APACHE_SPARK_REF::$apache_spark_ref" after this line because we're running tests with run-tests.py.

I will revert this for now .. seems like it breaks other tests.

@sarutak would you mind opening a Pr again for this?

Ah, O.K. I'll do it. Thanks for letting me know.

HyukjinKwon · 2021-05-28T05:30:19Z

sorry for reverting quickly - I reverted first as the issue is sort of minor but it takes a while to test related to this 😢

### What changes were proposed in this pull request? This PR proposes to add `docker-integratin-tests` to `run-tests.py` and GA. Once #32631 was merged but there was a lack of consideration. Diff between this change and 692d95d merged in #32631 is as follows. ``` if: github.repository != 'apache/spark' id: sync-branch run: | + apache_spark_ref=`git rev-parse HEAD` git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/} git -c user.name='Apache Spark Test Account' -c user.email='sparktestaccgmail.com' merge --no-commit --progress --squash FETCH_HEAD git -c user.name='Apache Spark Test Account' -c user.email='sparktestaccgmail.com' commit -m "Merged commit" + echo "::set-output name=APACHE_SPARK_REF::$apache_spark_ref" - name: Cache Scala, SBT and Maven uses: actions/cachev2 with: ``` ### Why are the changes needed? CI for `docker-integration-tests` is absent for now. ### Does this PR introduce _any_ user-facing change? GA. ### How was this patch tested? Closes #32691 from sarutak/docker-integration-test-ga-take2. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

sarutak added 3 commits May 22, 2021 14:55

Add docker-integration-tests to GA.

6308cc0

Merge branch 'master' of https://github.com/apache/spark into docker-…

62afdb2

…integration-test-ga

Minor fix.

5d17d60

github-actions bot added BUILD INFRA SQL labels May 22, 2021

maropu reviewed May 22, 2021

View reviewed changes

dev/sparktestsupport/modules.py Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Modify the condition in run-tests.py.

f8ea917

This comment has been minimized.

Sign in to view

dongjoon-hyun reviewed May 23, 2021

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

dongjoon-hyun reviewed May 23, 2021

View reviewed changes

This comment has been minimized.

Sign in to view

HyukjinKwon reviewed May 23, 2021

View reviewed changes

dev/run-tests.py Outdated Show resolved Hide resolved

sarutak added 2 commits May 24, 2021 11:29

Merge branch 'master' of https://github.com/apache/spark into docker-…

74107fd

…integration-test-ga

Separate docker-integration-tests job.

92da5c0

sarutak mentioned this pull request May 24, 2021

[SPARK-35501][SQL][TESTS] Add a feature for removing pulled container image for docker integration tests #32652

Closed

sarutak added 3 commits May 26, 2021 18:56

Merge branch 'master' of https://github.com/apache/spark into docker-…

b10f731

…integration-test-ga.tmp2

Reorder docker-integration-tests job.

32521b6

Introduce ENABLE_DOCKER_INTEGRATION_TESTS.

620af5b

This comment has been minimized.

Sign in to view

HyukjinKwon reviewed May 27, 2021

View reviewed changes

dev/sparktestsupport/modules.py Outdated Show resolved Hide resolved

HyukjinKwon approved these changes May 27, 2021

View reviewed changes

Leave dependencies empty.

19a4a93

HyukjinKwon reviewed May 27, 2021

View reviewed changes

dev/sparktestsupport/modules.py Outdated Show resolved Hide resolved

Change to run only on GA.

0cb6867

HyukjinKwon reviewed May 27, 2021

View reviewed changes

dev/run-tests.py Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

HyukjinKwon reviewed May 27, 2021

View reviewed changes

...r-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerIntegrationFunSuite.scala Outdated Show resolved Hide resolved

sarutak added 2 commits May 27, 2021 21:51

Fix modules.py.

10669ee

super.test(testName)(testBody) -> super.test(testName, testTags: _*)(…

692d95d

…testBody)

This comment has been minimized.

Sign in to view

HyukjinKwon closed this in 0a74ad6 May 27, 2021

HyukjinKwon reviewed May 28, 2021

View reviewed changes

sarutak mentioned this pull request May 28, 2021

[SPARK-35483][INFRA] Add docker-integration-tests to run-tests.py and GA #32691

Closed

sarutak deleted the docker-integration-test-ga branch June 4, 2021 20:40

sarutak restored the docker-integration-test-ga branch June 4, 2021 20:40

This was referenced Jun 25, 2021

[SPARK-34320][SQL][FOLLOWUP] Modify V2JDBCTest to follow the change of the error message. #33074

Closed

[SPARK-34302][FOLLOWUP][SQL][TESTS] Update jdbc.v2.*IntegrationSuite #33124

Closed

[SPARK-35483][INFRA] Add docker-integration-tests to run-tests.py and GA. #32631

[SPARK-35483][INFRA] Add docker-integration-tests to run-tests.py and GA. #32631

Uh oh!

Conversation

sarutak commented May 22, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak commented May 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented May 22, 2021

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 23, 2021

Uh oh!

This comment has been minimized.

Uh oh!

sarutak commented May 23, 2021

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

SparkQA commented May 27, 2021

Uh oh!

HyukjinKwon commented May 27, 2021

Uh oh!

dongjoon-hyun commented May 28, 2021

Uh oh!

HyukjinKwon May 28, 2021

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon May 28, 2021

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon May 28, 2021

Choose a reason for hiding this comment

Uh oh!

sarutak May 28, 2021

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sarutak commented May 22, 2021 •

edited

Loading

HyukjinKwon commented May 28, 2021 •

edited

Loading