Add Scala as `--build-arg` #1757

bjornjorgensen · 2022-07-26T18:10:32Z

Describe your changes

add Scala version choice

Issue ticket if applicable

[BUG] pyspark-notebook no longer builds for Spark 3.1.3, 3.1.2, 2.4.8 - Fix: #1756

Checklist (especially for first-time contributors)

I have performed a self-review of my code
If it is a core feature, I have added thorough tests
I will try not to use force-push to make the review process easier for reviewers
I have updated the documentation for significant changes

mathbunnyru · 2022-07-26T18:13:26Z

Converted this to a draft - it won't affect you, but no one will merge this accidentally :)

bjornjorgensen · 2022-07-26T18:15:18Z

@mathbunnyru Thank you :)
[WIP] Work in progress...

I have not tested this one, but I hope we can have a choose for scala version.

pyspark-notebook/Dockerfile

bjornjorgensen · 2022-07-26T20:44:16Z

Wooo.. have a runner ;)

docker build --pull --rm -f "Dockerfile" -t dockerstacks:latest .
Sending build context to Docker daemon  5.632kB
Step 1/24 : ARG OWNER=jupyter
Step 2/24 : ARG BASE_CONTAINER=$OWNER/scipy-notebook
Step 3/24 : FROM $BASE_CONTAINER

(...)

Step 15/24 : RUN if [ -z "$scala_version" ] ; then     wget -q "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" &&     echo "${spark_checksum} *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" | sha512sum -c - &&     tar xzf "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" ;  else     wget -q "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" &&     echo "${spark_checksum} *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" | sha512sum -c - &&     tar xzf "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ;  fi
 ---> Running in acf46391dc2f
spark-3.3.0-bin-hadoop3.tgz: OK










docker build --pull --rm -f "Dockerfile" -t dockerstacks:latest . --build-arg scala_version="2.13"
Sending build context to Docker daemon  5.632kB
Step 1/24 : ARG OWNER=jupyter
Step 2/24 : ARG BASE_CONTAINER=$OWNER/scipy-notebook
Step 3/24 : FROM $BASE_CONTAINER

(..)

Step 15/24 : RUN if [ -z "$scala_version" ] ; then     wget -q "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" &&     echo "${spark_checksum} *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" | sha512sum -c - &&     tar xzf "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" ;  else     wget -q "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" &&     echo "${spark_checksum} *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" | sha512sum -c - &&     tar xzf "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ;  fi
 ---> Running in b1b26b44bebd
spark-3.3.0-bin-hadoop3-scala2.13.tgz: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match

mathbunnyru

I made some suggestions.

Also, please remove the docs telling old spark versions won't work properly (because they should now).

pyspark-notebook/Dockerfile

bjornjorgensen · 2022-07-27T11:41:28Z

Tested with and they are OK.
docker build --rm --force-rm -t jupyter/pyspark-notebook:spark-3.3.0 . --build-arg spark_version=3.3.0 --build-arg hadoop_version=3 --build-arg spark_checksum=4c09dac70e22bf1d5b7b2cabc1dd92aba13237f52a5b682c67982266fc7a0f5e0f964edff9bc76adbd8cb444eb1a00fdc59516147f99e4e2ce068420ff4881f0 --build-arg openjdk_version=17 --build-arg scala_version="2.13"

docker build --rm --force-rm -t pyspark-notebook:spark-3.3.0-def . --build-arg spark_version=3.3.0 --build-arg hadoop_version=3 --build-arg spark_checksum=1e8234d0c1d2ab4462d6b0dfe5b54f2851dcd883378e0ed756140e10adfb5be4123961b521140f580e364c239872ea5a9f813a20b73c69cb6d4e95da2575c29c --build-arg openjdk_version=17

pyspark-notebook/Dockerfile

bjornjorgensen · 2022-07-27T12:04:30Z

Testing with

RUN if [ -z "${scala_version}" ] ; then \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && ;\
    #echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
    #tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
    #rm "spark.tgz" ;\
  else \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" && ;\
    #echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
    #tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
    #rm "spark.tgz" ;\
  fi \
    echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
    tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
    rm "spark.tgz"

Gives this error

Step 15/24 : RUN if [ -z "${scala_version}" ] ; then     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && ;  else     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" && ;  fi     echo "${spark_checksum} *spark.tgz" | sha512sum -c - &&     tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark.tgz"
 ---> Running in 64a679d8a650
/bin/bash: -c: line 0: syntax error near unexpected token `;'
/bin/bash: -c: line 0: `if [ -z "${scala_version}" ] ; then     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && ;  else     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" && ;  fi     echo "${spark_checksum} *spark.tgz" | sha512sum -c - &&     tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark.tgz"'

Bidek56 · 2022-07-27T12:35:19Z

You can make it conditional.

RUN  [ -z "${scala_version}" ] && \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"
RUN  [ -n "${scala_version}" ] && \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz"

echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
rm "spark.tgz"

bjornjorgensen · 2022-07-27T12:37:37Z

@Bidek56 Yes, but that is 2 RUN and @mathbunnyru and I don`t will have that.

Bidek56 · 2022-07-27T12:41:20Z

How about this with a single RUN?

RUN  [ -n "${scala_version}" ] && \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" || \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"

echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
rm "spark.tgz"

bjornjorgensen · 2022-07-27T12:46:15Z

Sending build context to Docker daemon 10.24kB
Error response from daemon: dockerfile parse error line 40: unknown instruction: ECHO

bjornjorgensen · 2022-07-27T12:52:31Z

@Bidek56 Try with

RUN  [ -n "${scala_version}" ] && \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" || \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" \

    echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
    tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
    rm "spark.tgz"

But then I get

Step 15/24 : RUN  [ -n "${scala_version}" ] &&     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ||     wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"     echo "${spark_checksum} *spark.tgz" | sha512sum -c - &&     tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark.tgz"
 ---> Running in 18ccdf491990
sha512sum: 'standard input': no properly formatted SHA512 checksum lines found

Bidek56 · 2022-07-27T12:58:43Z

I think it needs to be:

RUN ( [ -n "${scala_version}" ] && \ 
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" || \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" ) && \
    echo "${spark_checksum} *spark.tgz" | sha512sum -c - &&     tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner &&     rm "spark.tgz"

but I am still testing it.

mathbunnyru · 2022-07-27T14:34:27Z

@bjornjorgensen this example should help you to reduce code duplication:

FROM ubuntu

ARG my_var

RUN if [ "${my_var}" = "" ]; then \
        echo "My var was not provided"; \
        touch another_command.txt; \
    else \
        echo "My var was provided and it's value is ${my_var}"; \
        touch another_command.txt; \
    fi && \
    echo "This is common part" && \
    touch doing_common_stuff.txt

Bidek56 · 2022-07-27T14:54:11Z

docker build does not seem to like this command:

RUN if [ "${scala_version}" = "" ] ; then \
        wget --quiet -O "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"; \
    else \
        wget --quiet -O "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ; \
    fi

I get:

executor failed running [/bin/sh -c if [ "${scala_version}" = "" ] ; then wget --quiet -O "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"; else wget --quiet -O "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ; fi]: exit code: 5

bjornjorgensen · 2022-07-27T15:00:58Z

This seams to work..

RUN if [ -z "${scala_version}" ] ; then \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" ;\
  else \
    wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ;\
  fi && \
  echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
  tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
  rm "spark.tgz"

and this one..

RUN if [ -z "${scala_version}" ] ; then \
    ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" ${SPARK_HOME} ;\
  else \
    ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}" ${SPARK_HOME} ;\
  fi && \
  # Add a link in the before_notebook hook in order to source automatically PYTHONPATH && \
  mkdir -p /usr/local/bin/before-notebook.d && \
  ln -s "${SPARK_HOME}/sbin/spark-config.sh" /usr/local/bin/before-notebook.d/spark-config.sh

Bidek56 · 2022-07-27T15:06:34Z

It's failing for me locally but if it works in GA then great!

executor failed running [/bin/sh -c if [ -z "${scala_version}" ] ; then wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" ; else wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" ; fi && echo "${spark_checksum} *spark.tgz" | sha512sum -c - && tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && rm "spark.tgz"]: exit code: 5

bjornjorgensen · 2022-07-27T16:06:14Z

@Bidek56 exit code: 5 is problems with network.

This works for me. I test this on Manjaro with docker version 20.10.17

docker build --rm --force-rm     -t pyspark-notebook:spark-3.3.0 .     --build-arg spark_version=3.3.0     --build-arg hadoop_version=3     --build-arg spark_checksum=4c09dac70e22bf1d5b7b2cabc1dd92aba13237f52a5b682c67982266fc7a0f5e0f964edff9bc76adbd8cb444eb1a00fdc59516147f99e4e2ce068420ff4881f0     --build-arg openjdk_version=17 --build-arg scala_version="2.13"


docker build --rm --force-rm     -t pyspark-notebook:spark-3.3.0-def .     --build-arg spark_version=3.3.0     --build-arg hadoop_version=3     --build-arg spark_checksum=1e8234d0c1d2ab4462d6b0dfe5b54f2851dcd883378e0ed756140e10adfb5be4123961b521140f580e364c239872ea5a9f813a20b73c69cb6d4e95da2575c29c     --build-arg openjdk_version=17

mathbunnyru · 2022-07-27T18:50:54Z

Squash merged this to main.

add scala version choise

ec641d2

mathbunnyru marked this pull request as draft July 26, 2022 18:12

add ; \ fi

3895f32

Bidek56 reviewed Jul 26, 2022

View reviewed changes

pyspark-notebook/Dockerfile Outdated Show resolved Hide resolved

Bidek56 reviewed Jul 26, 2022

View reviewed changes

pyspark-notebook/Dockerfile Outdated Show resolved Hide resolved

bjornjorgensen added 2 commits July 26, 2022 21:33

change checksum and removed default scala version

41ffbe1

remove RUN

84c2938

mathbunnyru requested changes Jul 26, 2022

View reviewed changes

mathbunnyru reviewed Jul 26, 2022

View reviewed changes

pyspark-notebook/Dockerfile Show resolved Hide resolved

bjornjorgensen and others added 7 commits July 26, 2022 23:12

add { } and remove old code

e6b1f36

remove 3 duplicated lines.

c47cbbe

Add the commint as a comment

e5cadea

Add back #Fix

297880d

Rename downloadeds as spark.tgz

c0fe553

Fix doc

e8c0e4c

Update specifics.md

12179bb

mathbunnyru requested changes Jul 27, 2022

View reviewed changes

pyspark-notebook/Dockerfile Outdated Show resolved Hide resolved

bjornjorgensen added 4 commits July 27, 2022 11:22

New fix

4f0c8e5

Fix wget

34232de

Remove make link to spark

abbe09c

Set full path to /usr/local/spark

c7c2c47

mathbunnyru requested changes Jul 27, 2022

View reviewed changes

pyspark-notebook/Dockerfile Outdated Show resolved Hide resolved

pyspark-notebook/Dockerfile Outdated Show resolved Hide resolved

Change /usr/local/spark to ${SPARK_HOME}

290f2d7

bjornjorgensen changed the title ~~[WIP] add Scala as choice~~ Add Scala as choice Jul 27, 2022

bjornjorgensen changed the title ~~Add Scala as choice~~ Add Scala as --build-arg Jul 27, 2022

bjornjorgensen marked this pull request as ready for review July 27, 2022 12:58

mathbunnyru mentioned this pull request Jul 27, 2022

[BUG] - pyspark-notebook no longer builds for Spark 3.1.3, 3.1.2, 2.4.8 #1756

Closed

fix RUN with if

9ed24d9

mathbunnyru added 3 commits July 27, 2022 20:10

Remove empty lines

f2e07f5

Update Dockerfile

0fab467

Update Dockerfile

d279363

mathbunnyru approved these changes Jul 27, 2022

View reviewed changes

Update Dockerfile

b3c8071

mathbunnyru merged commit 1533087 into jupyter:main Jul 27, 2022

ionicsolutions mentioned this pull request Jul 28, 2022

Return to latest docker-stacks images for notebook/examples builds after upstream fix anovos/anovos#248

Merged

bjornjorgensen mentioned this pull request Dec 2, 2022

[ENH] - Release pyspark-notebook with spark 3.2.3 containing fix for CVE-2022-42889 #1838

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Scala as `--build-arg` #1757

Add Scala as `--build-arg` #1757

bjornjorgensen commented Jul 26, 2022 •

edited

Loading

mathbunnyru commented Jul 26, 2022

bjornjorgensen commented Jul 26, 2022

bjornjorgensen commented Jul 26, 2022 •

edited

Loading

mathbunnyru left a comment

bjornjorgensen commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

mathbunnyru commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022 •

edited

Loading

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

mathbunnyru commented Jul 27, 2022

Add Scala as --build-arg #1757

Add Scala as --build-arg #1757

Conversation

bjornjorgensen commented Jul 26, 2022 • edited Loading

Describe your changes

Issue ticket if applicable

Checklist (especially for first-time contributors)

mathbunnyru commented Jul 26, 2022

bjornjorgensen commented Jul 26, 2022

bjornjorgensen commented Jul 26, 2022 • edited Loading

mathbunnyru left a comment

Choose a reason for hiding this comment

bjornjorgensen commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

mathbunnyru commented Jul 27, 2022

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022 • edited Loading

Bidek56 commented Jul 27, 2022

bjornjorgensen commented Jul 27, 2022

mathbunnyru commented Jul 27, 2022

Add Scala as `--build-arg` #1757

Add Scala as `--build-arg` #1757

bjornjorgensen commented Jul 26, 2022 •

edited

Loading

bjornjorgensen commented Jul 26, 2022 •

edited

Loading

bjornjorgensen commented Jul 27, 2022 •

edited

Loading