Downgrade R in Docker image #685
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rshkv
commented
Jun 2, 2020
| @@ -111,7 +111,7 @@ WORKDIR $CIRCLE_HOME | |||
| # Install miniconda, we are using it to test conda support and a bunch of tests expect CONDA_BIN to be set | |||
| ENV CONDA_ROOT=$CIRCLE_HOME/miniconda | |||
| ENV CONDA_BIN=$CIRCLE_HOME/miniconda/bin/conda | |||
| ENV MINICONDA2_VERSION=4.5.11 | |||
| ENV MINICONDA2_VERSION=4.7.12.1 | |||
There was a problem hiding this comment.
This is to get a later Python 2 version. There is a bad interaction between libffi and older Python versions (conda/conda#9884).
rshkv
commented
Jun 2, 2020
| @@ -15,27 +15,29 @@ | |||
| # limitations under the License. | |||
| # | |||
|
|
|||
| FROM palantirtechnologies/circle-spark-base | |||
| FROM palantirtechnologies/circle-spark-base:0.2.3 | |||
There was a problem hiding this comment.
Otherwise this defaults to latest.
rshkv
commented
Jun 2, 2020
| && cat >>.bashrc <<<'eval "$($HOME/.pyenv/bin/pyenv init -)"' \ | ||
| && cat >>.bashrc <<<'eval "$($HOME/.pyenv/bin/pyenv virtualenv-init -)"' | ||
| RUN curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash | ||
| ENV PATH="$CIRCLE_HOME/.pyenv/shims:$CIRCLE_HOME/.pyenv/bin:$PATH" |
There was a problem hiding this comment.
Had to replace the .bashrc edits so they don't override my path hackery below.
robert3005
approved these changes
Jun 2, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem. In #677 we upgraded our Docker images to Ubuntu 20. As result of that we started picking up a higher R version (3.6), whereas we previously used 3.2 (latest available for Ubuntu 14).
Because 3.5 and 3.6 are major releases with serialization breaks, we started compiling a SparkR that is incompatible with the R versions we have to support internally (as low as 3.3). This caused errors like the one below (workspace serialization break in 3.5):
Or the one below (courtesy of the data serialization break in 3.6):
More info on this RStudio issue and release announcements for 3.5 and 3.6.
Change. This change ensures that we compile SparkR using R 3.4. We have to support 3.3 internally, but 3.4 had no breaks. We also have to support 3.5, and I assume 3.4 is good because 3.2 was - I don't believe we internally read 3.2 workspaces with 3.6 and vice versa.
I switched to installing R via Conda instead of Apt. CRAN has channels hosting older R versions, but those channels don't host R's dependencies.