Automatically build and push a docker image #696

jhkennedy · 2021-11-15T06:45:53Z

This PR:

adds an GitHub Actions workflow for docker which:
- Builds a docker image for any PR to main
- Builds a docker image for any push to main and then pushes the images to the GitHub Container Registry
  - the image will be tagged with the short commit SHA
  - the image will be tagged with latest
- Builds a docker image for any pushed v* tag (e.g., v1.3.1) and then pushes the images to the GitHub Container Registry
  - the image will be tagged with the short commit SHA
  - the image will be tagged with the version number (e.g., 1.3.1 for a v1.3.1 tag)
updates the Dockerfile (again)
- moved to the top level (repository root)
- copies in the current repository instead of cloning or installing via git+https
- includes jupyterlab and ipympl in the environment by defafult
updates the environment/requirements files to use the conda-forge version of pyaps3 and pysolid instead of cloning or installing via git+https

Reminders

Fix regularly updated docker image #123
Pass Codacy code review (green)
Pass Circle CI test (green)
Make sure that your code follows our style. Use the other functions/files as a basis.
If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
If adding new functionality, add a detailed description to the documentation and/or an example.

jhkennedy

@yunjunz @forrestfwilliams I've added some review comments with more details about some of the choices I've made here.

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Right now, I'm tagging any push to main latest so if you docker pull ghcr.io/insarlab/mintpy:latest you'll get whatever is on main.

However, latest in the docker world is typically used as "the image I want users to pull" which is generally "the last stable release", so I could also see instead tagging the last release latest and any merge to main as test (or develop or similar)

Dockerfile

docs/requirements.txt

.github/workflows/docker-build.yml

jhkennedy · 2021-11-15T08:29:02Z

You can get a container build off this branch like:

docker pull jhkennedy/mintpy:test

You can test the docker container by running:

docker run --rm -it jhkennedy/mintpy:test python tools/MintPy/tests/test_smallbaselineApp.py

Or drop into the container to run the tests:

docker run -it jhkennedy/mintpy:test
cd tools/MintPy/tests
python test_smallbaselineApp.py

You can also run the jupyterlab instance by running:

docker run --rm -p 8888:8888 -it jhkennedy/mintpy:test jupyter lab --no-browser  --port=8888 --ip=0.0.0.0 --NotebookApp.token=mintpy

(Might be worth making a jupyter launch script...)

yunjunz · 2021-11-16T08:04:54Z

Thank you for the so long desired PR @jhkennedy!!!

I feel that we are better off if splitting the "requirements + circle CI" related changes to another PR. For each of them, I would like to update the related documentation and links as well during the PR review to make sure that 1) code and doc are consistent and 2) we are all on the same page for the related instruction. What do you say?

I have updated pyaps3 in the following aspects, hoping to help this PR:

moved the repo to the insarlab org (https://github.com/insarlab/pyaps)
re-structurer the code following the standard practice (together with necessary changes in the code)
made a new release of version 0.3.0 and upload it to PyPI (https://pypi.org/project/pyaps3/)

We should update the version of pyaps3 on the conda-forge channel accordingly, maybe you could do it?

I will take a closer look at the docker file and workflow related changes later this week.

yunjunz · 2021-11-16T08:15:33Z

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Having latest to the last stable release and develop to main sounds accurate and clear to me.

jhkennedy · 2021-11-16T18:43:24Z

I feel that we are better off if splitting the "requirements + circle CI" related changes to another PR.

@yunjunz Agreed! I'll split those out into a different PR. Changes kinda cascaded on me.

For each of them, I would like to update the related documentation and links as well during the PR review to make sure that 1) code and doc are consistent and 2) we are all on the same page for the related instruction. What do you say?

Yes! I wanted to make sure the tagging pattern matched what you and @forrestfwilliams thought before investing in the docs.

I have updated pyaps3 in the following aspects, hoping to help this PR:

moved the repo to the insarlab org (https://github.com/insarlab/pyaps)

re-structurer the code following the standard practice (together with necessary changes in the code)

made a new release of version 0.3.0 and upload it to PyPI (https://pypi.org/project/pyaps3/)

We should update the version of pyaps3 on the conda-forge channel accordingly, maybe you could do it?

That sounds great! I'll work on the conda-forge PR first, then split out the requirements into a seperate PR that can go before this one, and then we can all circle back to this one, giving @forrestfwilliams a little time to look it over.

jhkennedy · 2021-11-16T18:44:08Z

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Having latest to the last stable release and develop to main sounds accurate and clear to me.

👍, @forrestfwilliams does that match your feeling as well?

forrestfwilliams · 2021-11-16T19:57:41Z

Hi @jhkennedy, yes I think it makes sense to split this into two separate PRs as @yunjunz suggested. As for the image tags, I like the idea of having the latest tag correspond to the last stable release and the develop tag correspond to main. I think people not involved in this discussion will find this to be more intuitive.

jhkennedy · 2021-11-19T01:02:19Z

Okay, conda-forge/pyaps3-feedstock#2 is open, so the PyAPS v0.3.0 should be available soon.

jhkennedy · 2022-02-17T20:51:36Z

@yunjunz I'll have time to pick this up again tomorrow -- I've been slammed the last couple months

Dockerfile

yunjunz · 2022-02-17T21:04:30Z

Sounds great @jhkennedy, please take your time.

I have been using pip-install for the development version of mintpy as you suggested, instead of path setup, and it feels great! With a few minor fixes that will be included in the next release, I believe the conda-installed version of mintpy will be functioning in its full capacity.

yunjunz · 2022-05-19T04:08:34Z

I rebase this PR against the latest insarlab/main branch. I tried the jhkennedy/mintpy:test which has version 1.3.1, and it works well. It would be great if we could finish this PR @jhkennedy @forrestfwilliams.

Regarding the documentation, I could think of the following:

add a link to the mintpy docker image registrary?
in docs/installation.md: how to pull from the latest or stable version; explain the two tags; and how to update them
since we are updating the docker image for every commit, there is no need to write a note on how to build the image from scratch, right?
the various styles of running the docker image, as in the current docs/docker.md

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

jhkennedy · 2022-05-19T17:00:18Z

Thanks @yunjunz I think I'll be able to work on this again next week some -- I'll work on addressing 1-4.

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

There's a lot of options -- easiest is create a new conda environment for isce2 when you jump into the container. It's also possible to build a docker image that combines an ISCE2 image and this image though I'd have to look at how the ISCE2 one is laid out. We could also build an ISCE2 env into this container...

yunjunz · 2022-05-19T20:57:23Z

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

There's a lot of options -- easiest is create a new conda environment for isce2 when you jump into the container. It's also possible to build a docker image that combines an ISCE2 image and this image though I'd have to look at how the ISCE2 one is laid out. We could also build an ISCE2 env into this container...

I like the idea of adding isce2 as another dependency to this container, so that the users could import isce as a library and run mintpy on the output of isce2 stack processors. It's simple on our side (don't have to deal with the isce2 setup for topsStack/stripmapStack). Although one won't be able to run isce2 data processing here, he/she could do that in another dedicated isce2 container.

jhkennedy · 2022-06-20T22:16:26Z

@yunjunz finally getting back to this -- I just forced push rebasing onto insarlab/MintPy@main so I can see where we're at and will work on the outstanding stuff now

jhkennedy · 2022-06-21T00:36:14Z

@yunjunz and @forrestfwilliams alright, I updated the documentation now as well, so I think this is ready for another review.

I did push a couple container images to insarlab/mintpy in the GitHub Container Registry, so you can:

Pull a container image built with v1.3.3 like: docker pull ghcr.io/insarlab/mintpy:v1.3.3 or
docker pull ghcr.io/insarlab/mintpy:latest
Pull a container image built from this branch like: docker pull ghcr.io/insarlab/mintpy:v1.3.3-38-g6690f9a1

Note: By default the MintPy container images repository is set to private, so any MintPy maintainer can pull these as long as they log into ghcr.io with docker, but general users won't be able to until the repository is made public.

@yunjunz if you want to make the container repository public, you (or another owner) will need to go to:
https://github.com/orgs/insarlab/packages/container/mintpy/settings
and scroll to the "Danger Zone" at the bottom and change the visibility to public:

You may also notice the container "packages" show up on the side of the GitHub repository now (if you have access to them):

Dockerfile

jhkennedy · 2022-06-21T16:21:06Z

@yunjunz @forrestfwilliams I pushed some changes to make the Jupyter integration a little easier, and updates the docs on how to start the Jupyter server. As such, I pushed an updated image based on this PR:

docker pull ghcr.io/insarlab/mintpy:v1.3.3-39-g5f718b2b

I'll rebuild and push the v1.3.3/latest images again once this PR is merged so they match all the documentation.

Also, Codacy is complaining about two lines in the scripts/jupyter_notebook_config.py file, but those are written and distributed by the Jupyter Team so I think they can safely be ignored.

yunjunz

Thank you @jhkennedy for finishing this exciting PR!!! Both the implementation and documentations look great. I have changed the container image (package) visibility to public, following your suggestion.

Testings, including tools/MintPy/tests/test_smallbaselineApp.py, on my laptop work very well, except for one issue: when I started an interactive shell session as below, view.py seems to work well (normal print-out messages), except that the plotting window is not popping up. Any idea?

docker run -it -v ~/data:/home/mambauser/data --name mintpy ghcr.io/insarlab/mintpy:latest
cd data/test/FernandinaSenDT128/mintpy
view.py velocity.h5

FYI, view.py velocity.h5 --nodisplay also works (successfully plotted and saved the figure into a file). So this seems to be an interactive plotting window issue.

jhkennedy · 2022-06-23T18:29:16Z

except for one issue: when I started an interactive shell session as below, view.py seems to work well (normal print-out messages), except that the plotting window is not popping up.

@yunjunz I'm not sure -- we're using a pretty minimal base image so it might be a missing x11 or the like; I'll see if I can get it going

jhkennedy · 2022-06-28T17:25:15Z

@yunjunz the issue does come down to X.Org windows system issue, that is a bit annoying to work around and can induce some security risks. Further reading:

There's a couple tools that solve a lot of the issues that are more-or-less straightforward to use that set up X.Org windows system between the host and the container:

x11docker on Linux
x11docker plus XQuartz on Mac

But since we include Jupyter, that's the easiest/best path to plotting figures in the container.

yunjunz

Thank you very much for the useful information regarding GUI @jhkennedy. I agree that Jupyter is a good and easy path for us right now. More work on the GUI in a Docker can be done laterward in another PR, I have created an issue for it (#801).

As for this PR, it looks all great to me. Cheers!

yunjunz · 2022-06-28T18:56:47Z

The Github workflow failed with the following error, which seems to be related to some permission issue. @jhkennedy I have added you as a maintainer role. Any idea how we could resolve this?

https://github.com/insarlab/MintPy/runs/7099087138?check_suite_focus=true#step:6:1603

#12 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

jhkennedy · 2022-06-28T19:42:13Z

Hmm, can you confirm that the insarlab/MintPy actions setting -> workflow settings:
https://github.com/insarlab/MintPy/settings/actions

Looks like this:

I don't think I can look at the org/repo setting without admin or owner level permissions.

jhkennedy · 2022-06-28T19:46:49Z

@yunjunz I think I found the setting needed:
https://github.com/orgs/insarlab/packages/container/mintpy/settings

Re-running the action now to check

jhkennedy · 2022-06-28T19:58:05Z

@yunjunz that did it!
https://github.com/insarlab/MintPy/runs/7100056427?check_suite_focus=true

yunjunz · 2022-06-28T20:25:14Z

Great!

jhkennedy requested review from yunjunz and forrestfwilliams November 15, 2021 06:45

jhkennedy commented Nov 15, 2021

View reviewed changes

Dockerfile Show resolved Hide resolved

Dockerfile Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

docs/requirements.txt Outdated Show resolved Hide resolved

jhkennedy marked this pull request as ready for review November 15, 2021 07:34

jhkennedy commented Nov 15, 2021

View reviewed changes

.github/workflows/docker-build.yml Show resolved Hide resolved

yunjunz reviewed Feb 17, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

yunjunz force-pushed the auto-docker branch from 1d100ae to f13861c Compare May 19, 2022 02:35

jhkennedy force-pushed the auto-docker branch from f2ab27e to 681daec Compare June 20, 2022 22:14

jhkennedy and others added 9 commits June 20, 2022 14:14

Move Dockerfile to top-level

0a8d319

Update dockerfile to build from local repo and include jupyter

a34ecf5

Add dockerize workflow

0e6e8d3

Simplify docker tags

85b48d4

update container description

e6c274f

dockerfile: update requirements.txt path

e95dabe

docker: add gdal>=3

0bbecf5

Pin to pyaps >=0.3

fe71d9b

Add isce2 to docker environment

c1f07f8

jhkennedy force-pushed the auto-docker branch from 681daec to c1f07f8 Compare June 20, 2022 22:15

jhkennedy added 5 commits June 20, 2022 14:22

update docker tagging strategy

5de0802

Update base image

02e2753

Update dockerfile to micromamba user name change

000557e

Update documentation

0254d4a

fix typo

87ecfa3

jhkennedy commented Jun 21, 2022

View reviewed changes

Dockerfile Show resolved Hide resolved

Provide easer access to Jupyter

5f718b2

yunjunz added 2 commits June 21, 2022 22:14

group notes and typo fix

23b164c

Update docker-build.yml

e5b9205

yunjunz reviewed Jun 22, 2022

View reviewed changes

yunjunz mentioned this pull request Jun 28, 2022

matplotlib GUI in a Docker container #801

Open

yunjunz approved these changes Jun 28, 2022

View reviewed changes

yunjunz merged commit 719c2b4 into insarlab:main Jun 28, 2022

jhkennedy deleted the auto-docker branch June 28, 2022 20:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically build and push a docker image #696

Automatically build and push a docker image #696

jhkennedy commented Nov 15, 2021 •

edited

Loading

jhkennedy left a comment

jhkennedy commented Nov 15, 2021 •

edited

Loading

yunjunz commented Nov 16, 2021 •

edited

Loading

yunjunz commented Nov 16, 2021

jhkennedy commented Nov 16, 2021

jhkennedy commented Nov 16, 2021

forrestfwilliams commented Nov 16, 2021

jhkennedy commented Nov 19, 2021

jhkennedy commented Feb 17, 2022

yunjunz commented Feb 17, 2022

yunjunz commented May 19, 2022 •

edited by jhkennedy

Loading

jhkennedy commented May 19, 2022

yunjunz commented May 19, 2022

jhkennedy commented Jun 20, 2022

jhkennedy commented Jun 21, 2022

jhkennedy commented Jun 21, 2022

yunjunz left a comment •

edited

Loading

jhkennedy commented Jun 23, 2022

jhkennedy commented Jun 28, 2022

yunjunz left a comment

yunjunz commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

yunjunz commented Jun 28, 2022

Automatically build and push a docker image #696

Automatically build and push a docker image #696

Conversation

jhkennedy commented Nov 15, 2021 • edited Loading

jhkennedy left a comment

Choose a reason for hiding this comment

jhkennedy commented Nov 15, 2021 • edited Loading

yunjunz commented Nov 16, 2021 • edited Loading

yunjunz commented Nov 16, 2021

jhkennedy commented Nov 16, 2021

jhkennedy commented Nov 16, 2021

forrestfwilliams commented Nov 16, 2021

jhkennedy commented Nov 19, 2021

jhkennedy commented Feb 17, 2022

yunjunz commented Feb 17, 2022

yunjunz commented May 19, 2022 • edited by jhkennedy Loading

jhkennedy commented May 19, 2022

yunjunz commented May 19, 2022

jhkennedy commented Jun 20, 2022

jhkennedy commented Jun 21, 2022

jhkennedy commented Jun 21, 2022

yunjunz left a comment • edited Loading

Choose a reason for hiding this comment

jhkennedy commented Jun 23, 2022

jhkennedy commented Jun 28, 2022

yunjunz left a comment

Choose a reason for hiding this comment

yunjunz commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

jhkennedy commented Jun 28, 2022

yunjunz commented Jun 28, 2022

jhkennedy commented Nov 15, 2021 •

edited

Loading

jhkennedy commented Nov 15, 2021 •

edited

Loading

yunjunz commented Nov 16, 2021 •

edited

Loading

yunjunz commented May 19, 2022 •

edited by jhkennedy

Loading

yunjunz left a comment •

edited

Loading