Allow Usage of Pre-Existing Docker Images Instead of Image Building

[salvis2]

Use-case here is that for the IceSat-2 Hackweek, we are using hubploy to deploy the Hub. We are managing the image building separately and would like to just point to it with the JupyterHub Helm chart value jupyterhub.singleuser.image.tag. However, hubploy will overwrite this value with the image that is defined in hubploy.yaml and built later.

Our workaround at the moment is to base the image we build around the already built image, but that is messy and it would be much cleaner if we could just point to the previously built one.

[yuvipanda]

If you don't specify the images section at all in hubploy.yaml, hubploy shouldn't do any image tag munging. If it does, we should fix that. Can you try that?

[salvis2]

From some quick testing with a commented out image part for hubploy.yaml, hubploy deploy ... was fine and hubploy build ... errored out, which I think makes sense. I had some issues getting the continuous prepuller in JupyterHub to play nicely with this. It kept trying to find an image for a tag that wasn't represented in ECR. I did have some tags in there though, so maybe that was the problem.

[yuvipanda]

@salvis2 oh, what error did the continuous pre-puller report? Did the hook pre-puller also report these? And was continuous pre-puller trying to pull in the 'right' image?

[salvis2]

I will need to spin up a new test hub and fiddle with it there. I don't recall everything, but the prepuller was doing ImagePullBackOff or something to that effect. The image link should have been fine, it was just the tag that was messed up.

[salvis2]

Actually just testing on our hackweek cluster, since it would be nice to not have to duplicate the Docker image.

I've commented out the entire image: block in hubploy.yaml. I've deleted all images in the repo that hubploy would use if it was pushing images (I have left the repo still there, just empty). I have set prePuller.continuous.enabled: false.

Running

hubploy deploy hackweek-hub hub staging

Yielded the error

...
  File ".../hubploy/config.py", line 194, in get_config
    for image in images:
UnboundLocalError: local variable 'images' references before assignment

This line is the same in your hubploy, just checked.

If I uncomment the images: block, comment out image_name: ..., then run the deploy command, I get

...
  File ".../hubploy/config.py", line 192, in get_config
    images = images_config['images']
KeyError: 'images'

Looks like config.py expects 1+ images to be listed.

Leaving images: as it was to begin with, our other idea was to just delete the /image folder.

git rm -r /deployments/hackweek-hub/image/

The deploy command succeeds. However,

helm get values hackweek-hub-staging -n hackweek-hub-staging

still shows

...
  singleuser:
    image:
      name: ############.dkr.ecr.us-west-2.amazonaws.com/hackweek-hub-user-image
      tag: 98402004d

Trying to login and launch gives me the error ImagePullBackOff because there are no images in the listed ECR repo.

If I specify the image I want within singleuser.profileList, everything is fine. I can launch my JupyterLab instance. However, we would like to have the image listed under singleuser.image so that users don't have to click a button to launch their JupyterLab instance when they don't have any choices to make there. Not a terrible workaround to specify it in profileList, though.

Now I re-enable the prepuller, deploy, and the continuous-image-puller pods are stuck trying to pull the same image as above, so they sit with STATUS Init;ImagePullBackOff. Don't see the hook pre-puller (unsure what that is though). Do I need to purge the Hub of old images it used to run? Or is it just trying to pull everything in the profileList and singleuser.image?

I guess the conclusions are that currently,

1. You must specify images: or no hubploy commands work and
2. singleuser.image is forced to be set, even if you aren't using it, and you can't overwrite it.
3. The prepuller is trying to get the image listed in singleuser.image and failling, but that might just be because there used to be an image in that ECR repo that I was using. A fresh install / purge of that information might help.
4. I don't think deleting the deployments/hackweek-hub/image folder did anything, but it will obviously give errors if you attempt to run hubploy build ....