Skip to content

[SPARKR][BACKPORT-2.1] backporting package and test changes #19165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
felixcheung wants to merge 10 commits into from
Closed

[SPARKR][BACKPORT-2.1] backporting package and test changes #19165

felixcheung wants to merge 10 commits into from

Conversation

@felixcheung
Copy link
Member

@felixcheung felixcheung commented Sep 8, 2017

What changes were proposed in this pull request?

cherrypick or manually porting changes to 2.1

How was this patch tested?

Jenkins

Wayne Zhang and others added 7 commits September 8, 2017 00:23
@felixcheung
cherrypick 2fdaeb5
b3062eb
@felixcheung
cherrypick c24bdaa
5e94838
@felixcheung
manual port of 0b0be47
15c2894
@felixcheung
port subset of 9f4ff95
79c82c7
@HyukjinKwon @felixcheung
[SPARK-21128][R] Remove both "spark-warehouse" and "metastore_db" bef…
a74ba02 
…ore listing files in R tests

## What changes were proposed in this pull request?

This PR proposes to list the files in test _after_ removing both "spark-warehouse" and "metastore_db" so that the next run of R tests pass fine. This is sometimes a bit annoying.

## How was this patch tested?

Manually running multiple times R tests via `./R/run-tests.sh`.

**Before**

Second run:

```
SparkSQL functions: Spark package found in SPARK_HOME: .../spark
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
....................................................................................................1234.......................

Failed -------------------------------------------------------------------------
1. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
length(list1) not equal to length(list2).
1/1 mismatches
[1] 25 - 23 == 2

2. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
10/25 mismatches
x[16]: "metastore_db"
y[16]: "pkg"

x[17]: "pkg"
y[17]: "R"

x[18]: "R"
y[18]: "README.md"

x[19]: "README.md"
y[19]: "run-tests.sh"

x[20]: "run-tests.sh"
y[20]: "SparkR_2.2.0.tar.gz"

x[21]: "metastore_db"
y[21]: "pkg"

x[22]: "pkg"
y[22]: "R"

x[23]: "R"
y[23]: "README.md"

x[24]: "README.md"
y[24]: "run-tests.sh"

x[25]: "run-tests.sh"
y[25]: "SparkR_2.2.0.tar.gz"

3. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
length(list1) not equal to length(list2).
1/1 mismatches
[1] 25 - 23 == 2

4. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
10/25 mismatches
x[16]: "metastore_db"
y[16]: "pkg"

x[17]: "pkg"
y[17]: "R"

x[18]: "R"
y[18]: "README.md"

x[19]: "README.md"
y[19]: "run-tests.sh"

x[20]: "run-tests.sh"
y[20]: "SparkR_2.2.0.tar.gz"

x[21]: "metastore_db"
y[21]: "pkg"

x[22]: "pkg"
y[22]: "R"

x[23]: "R"
y[23]: "README.md"

x[24]: "README.md"
y[24]: "run-tests.sh"

x[25]: "run-tests.sh"
y[25]: "SparkR_2.2.0.tar.gz"

DONE ===========================================================================
```

**After**

Second run:

```
SparkSQL functions: Spark package found in SPARK_HOME: .../spark
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................................................
...............................................................................................................................
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes apache#18335 from HyukjinKwon/SPARK-21128.
@felixcheung
[SPARK-21805][SPARKR] Disable R vignettes code on Windows
129a979 
## What changes were proposed in this pull request?

Code in vignettes requires winutils on windows to run, when publishing to CRAN or building from source, winutils might not be available, so it's better to disable code run (so resulting vigenttes will not have output from code, but text is still there and code is still there)

fix * checking re-building of vignette outputs ... WARNING
and
> %LOCALAPPDATA% not found. Please define the environment variable or restart and enter an installation path in localDir.

## How was this patch tested?

jenkins, appveyor, r-hub

before: https://artifacts.r-hub.io/SparkR_2.2.0.tar.gz-49cecef3bb09db1db130db31604e0293/SparkR.Rcheck/00check.log
after: https://artifacts.r-hub.io/SparkR_2.2.0.tar.gz-86a066c7576f46794930ad114e5cff7c/SparkR.Rcheck/00check.log

Author: Felix Cheung <felixcheung_m@hotmail.com>

Closes apache#19016 from felixcheung/rvigwind.
@felixcheung
not cran
b53f812
@SparkQA
Copy link

SparkQA commented Sep 8, 2017

Test build #81550 has finished for PR 19165 at commit b53f812.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung
Copy link
Member Author

felixcheung commented Sep 8, 2017

jenkins, retest this please

@SparkQA
Copy link

SparkQA commented Sep 8, 2017

Test build #81565 has finished for PR 19165 at commit b53f812.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung
[TEST][SPARKR][CORE] Fix broken SparkSubmitSuite
bb6f326 
## What changes were proposed in this pull request?

Fix test file path. This is broken in apache#18264 and undetected since R-only changes don't build core and subsequent post-commit with the change built fine (again because it wasn't building core)

actually appveyor builds everything but it's not running scala suites ...

## How was this patch tested?

jenkins
srowen gatorsmile

Author: Felix Cheung <felixcheung_m@hotmail.com>

Closes apache#18283 from felixcheung/rsubmitsuite.

(cherry picked from commit 278ba7a)
Signed-off-by: Felix Cheung <felixcheung@apache.org>
@SparkQA
Copy link

SparkQA commented Sep 9, 2017

Test build #81572 has finished for PR 19165 at commit bb6f326.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 10, 2017

Test build #81595 has finished for PR 19165 at commit 7ded0ae.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 10, 2017

Test build #81600 has finished for PR 19165 at commit 9d53891.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung
Copy link
Member Author

felixcheung commented Sep 10, 2017

merged to 2.1

asfgit pushed a commit that referenced this pull request Sep 10, 2017
@felixcheung
[SPARKR][BACKPORT-2.1] backporting package and test changes
ae4e8ae 
## What changes were proposed in this pull request?

cherrypick or manually porting changes to 2.1

## How was this patch tested?

Jenkins

Author: Felix Cheung <felixcheung_m@hotmail.com>
Author: hyukjinkwon <gurwls223@gmail.com>
Author: Wayne Zhang <actuaryzhang@uber.com>

Closes #19165 from felixcheung/rbackportpkg21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@felixcheung @SparkQA @HyukjinKwon