Installation bug (?) in Colab

[jqbond]

First, I want to thank you for developing this. I use it for teaching as I don't have the ability to add a local Jupyter/Julia installation to the various classroom computers. I used it extensively about a year ago by installing Julia 1.7.2, IIRC.

I am teaching this course again, and I went to run the Julia installation, and it is throwing an error that I can't quite wrap my head around. I am guessing it is on the Colab side. Any guidance on a resolution would be appreciated!!

Installing Julia 1.8.2 on the current Colab Runtime...
2023-01-20 15:45:13 URL:https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz [135859273/135859273] -> "/tmp/julia.tar.gz" [1]
Installing Julia package IJulia...
/bin/bash: line 24:   229 Aborted                 (core dumped) julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
[<ipython-input-1-9b4dbd3cbe07>](https://localhost:8080/#) in <module>
----> 1 get_ipython().run_cell_magic('shell', '', 'set -e\n\n#---------------------------------------------------#\nJULIA_VERSION="1.8.2" # any version ≥ 0.7.0\nJULIA_PACKAGES="IJulia BenchmarkTools"\nJULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions\nJULIA_NUM_THREADS=2\n#---------------------------------------------------#\n\nif [ -z `which julia` ]; then\n  # Install Julia\n  JULIA_VER=`cut -d \'.\' -f -2 <<< "$JULIA_VERSION"`\n  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."\n  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"\n  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"\n  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"\n  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1\n  rm /tmp/julia.tar.gz\n\n  # Install Packages\n  nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0\n  if [ $GPU -eq 1 ]; then\n    JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"\n  fi\n  for PKG in `echo $JULIA_PACKAGES`; do\n    echo "Installing Julia package $PKG..."\n    julia -e \'using Pkg; pkg"add \'$PKG\'; precompile;"\' &> /dev/null\n  done\n\n  # Install kernel and rename it to "julia"\n  echo "Installing IJulia kernel..."\n  julia -e \'using IJulia; IJulia.installkernel("julia", env=Dict(\n      "JULIA_NUM_THREADS"=>"\'"$JULIA_NUM_THREADS"\'"))\'...

2 frames
[/usr/local/lib/python3.8/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in check_returncode(self)
    132   def check_returncode(self):
    133     if self.returncode:
--> 134       raise subprocess.CalledProcessError(
    135           returncode=self.returncode, cmd=self.args, output=self.output)
    136 

CalledProcessError: Command 'set -e

#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#

if [ -z `which julia` ]; then
  # Install Julia
  JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
  rm /tmp/julia.tar.gz

  # Install Packages
  nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
  if [ $GPU -eq 1 ]; then
    JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
  fi
  for PKG in `echo $JULIA_PACKAGES`; do
    echo "Installing Julia package $PKG..."
    julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
  done

  # Install kernel and rename it to "julia"
  echo "Installing IJulia kernel..."
  julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
      "JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
  KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
  KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
  mv -f $KERNEL_NAME "$KERNEL_DIR"/julia  

  echo ''
  echo "Successfully installed `julia -v`!"
  echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
  echo "jump to the 'Checking the Installation' section."
fi
' returned non-zero exit status 134.

[MilesCranmer]

@ageron I am seeing this too, across all Julia versions

[ageron]

Thanks for your feedback, I'll look into this issue right now.

[ageron]

Yikes, it's indeed completely broken. It looks like it's not due to the installation code, but rather some change in Colab's runtime environment which prevents Julia from running. See this discussion. It's not a simple fix on my end, unfortunately. I'll keep looking, but it may require some patch from Google, and since they don't officially support Julia, they might have very little bandwidth for this. I hope it's not the end of Julia on Colab, it's really convenient.

[jqbond]

@ageron that was my guess when I first encountered the error. I ended up borrowing some of the Jupyter Docker Stacks and then customizing a Dockerfile with Julia packages. I'm running it now in Codespaces on GitHub as my cloud-based Julia-in-a-Notebook solution. Let me know if that is of interest, and I can link you to the repository.

[ageron]

That sounds great @jqbond , yes please link to the repo!

[MilesCranmer]

Hey @ageron @jqbond there are some helpful solutions being posted here: googlecolab/colabtools#3385. It sounds like they are trying to patch this in the next runtime update which is good.

[ageron]

It's great that the Colab team has acknowledged this bug and will try to fix it. I've tried to run "Use fallback runtime version" from the Command palette, as they suggested, but unfortunately I got a message saying that the fallback runtime is unavailable.

However, the second solution they suggested worked: I just replaced every julia command in the first cell of the notebooks with LD_PRELOAD= julia (note the space before julia), and then things ran normally.

I'll commit this change and revert it once they've fixed Colab.

[jqbond]

@ageron Thanks for the solution -- I'll make a note of it. I also created a bare bones repository that should run Julia/Jupyter in a Codespace. The default configuration should open up a VS Code environment, and you can run Jupyter tools from the terminal, e.g., jupyter lab

Link below.

https://github.com/jqbond/Julia_Codespace

**Edit: I've been running with an 8-Core virtual machine, which you can change when you configure the codespace. Can't guarantee adequate memory with 2 or 4 core, but you can try them out.

[MilesCranmer]

Hey @ageron @jqbond everything is fixed with the current runtime now – no need for the fallback runtime or LD_PRELOAD=. See googlecolab/colabtools#3385.

[ageron]

That was much faster than I expected, cool! I removed LD_PRELOAD= , everything seems to work fine now. Thanks for your feedback everyone. 👍

[jddorellanao]

Has this error shown up again? Because I am getting this output:
CalledProcessError: Command 'set -e ' returned non-zero exit status 8.

Yesterday everything was fine.

[MilesCranmer]

I just looked and the error is this:

tar: This does not look like a tar archive

so not a colab-specific error? I think the default URLs need to be updated. Perhaps juliaup could be a good choice, since they have a non-interactive installation mode now.

[jddorellanao]

My code cell is this

%%shell
set -e

#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#

if [ -z `which julia` ]; then
  # Install Julia
  JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
  rm /tmp/julia.tar.gz

  # Install Packages
  nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
  if [ $GPU -eq 1 ]; then
    JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
  fi
  for PKG in `echo $JULIA_PACKAGES`; do
    echo "Installing Julia package $PKG..."
    julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
  done

  # Install kernel and rename it to "julia"
  echo "Installing IJulia kernel..."
  julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
      "JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
  KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
  KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
  mv -f $KERNEL_NAME "$KERNEL_DIR"/julia  

  echo ''
  echo "Successfully installed `julia -v`!"
  echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
  echo "jump to the 'Checking the Installation' section."
fi

and the output is this:

Installing Julia 1.8.2 on the current Colab Runtime...
https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz:
2023-04-08 23:13:56 ERROR 403: Forbidden.
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
[<ipython-input-6-5e5c48dc9892>](https://localhost:8080/#) in <cell line: 1>()
----> 1 get_ipython().run_cell_magic('shell', '', 'set -e\n\n#---------------------------------------------------#\nJULIA_VERSION="1.8.2" # any version ≥ 0.7.0\nJULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"\nJULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions\nJULIA_NUM_THREADS=2\n#---------------------------------------------------#\n\nif [ -z `which julia` ]; then\n  # Install Julia\n  JULIA_VER=`cut -d \'.\' -f -2 <<< "$JULIA_VERSION"`\n  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."\n  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"\n  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"\n  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"\n  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1\n  rm /tmp/julia.tar.gz\n\n  # Install Packages\n  nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0\n  if [ $GPU -eq 1 ]; then\n    JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"\n  fi\n  for PKG in `echo $JULIA_PACKAGES`; do\n    echo "Installing Julia package $PKG..."\n    julia -e \'using Pkg; pkg"add \'$PKG\'; precompile;"\' &> /dev/null\n  done\n\n  # Install kernel and rename it to "julia"\n  echo "Installing IJulia kernel..."\n  julia -e \'using IJulia; IJulia.installkernel("julia", env=Dict(\n      ...

3 frames
[/usr/local/lib/python3.9/dist-packages/google/colab/_shell.py](https://localhost:8080/#) in run_cell_magic(self, magic_name, line, cell)
    331     if line and not cell:
    332       cell = ' '
--> 333     return super().run_cell_magic(magic_name, line, cell)
    334 
    335 

[/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py](https://localhost:8080/#) in run_cell_magic(self, magic_name, line, cell)
   2471             with self.builtin_trap:
   2472                 args = (magic_arg_s, cell)
-> 2473                 result = fn(*args, **kwargs)
   2474             return result
   2475 

[/usr/local/lib/python3.9/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in _shell_cell_magic(args, cmd)
    110   result = _run_command(cmd, clear_streamed_output=False)
    111   if not parsed_args.ignore_errors:
--> 112     result.check_returncode()
    113   return result
    114 

[/usr/local/lib/python3.9/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in check_returncode(self)
    135   def check_returncode(self):
    136     if self.returncode:
--> 137       raise subprocess.CalledProcessError(
    138           returncode=self.returncode, cmd=self.args, output=self.output
    139       )

CalledProcessError: Command 'set -e

#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#

if [ -z `which julia` ]; then
  # Install Julia
  JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
  rm /tmp/julia.tar.gz

  # Install Packages
  nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
  if [ $GPU -eq 1 ]; then
    JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
  fi
  for PKG in `echo $JULIA_PACKAGES`; do
    echo "Installing Julia package $PKG..."
    julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
  done

  # Install kernel and rename it to "julia"
  echo "Installing IJulia kernel..."
  julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
      "JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
  KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
  KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
  mv -f $KERNEL_NAME "$KERNEL_DIR"/julia  

  echo ''
  echo "Successfully installed `julia -v`!"
  echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
  echo "jump to the 'Checking the Installation' section."
fi
' returned non-zero exit status 8.

[MilesCranmer]

Yeah, the URL doesn't exist anymore: go to https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz (i.e., it's not a colab issue). Maybe raise a new issue instead?

The default URL needs to be updated. Or just changed to juliaup instead.

[jddorellanao]

Ok, now, how do I install Julia with juliaup?

[MilesCranmer]

https://github.com/JuliaLang/juliaup