DEV: Add gitpod files #48107

noatamir · 2022-08-16T10:17:00Z

I split the Docker file into 2 Dockers
The base Docker includes the pandas environment, and has a new way to run Mamba in Docker, and builds on my machine (M1 mac).
The gitpod.docker copies the base docker, which will speed things up for any Gitpod user, and includes a shallow clone of the repository (more speed).
The workspace file defines some nice VS Code features, and the caches for C, C++, and the creation of .profile and .bashrc. Some things in there are finicky to order of operations (also in relation to the GitHub action which is coming in the next PR), so I recommend avoiding rearranging that file 😉
I added the Gitpod files in a new library, currently under root because there isn't a tools library. Maybe in the future, we can discuss renaming scripts to tools to avoid adding another folder? I don't have a strong opinion on this, I just felt awkward about adding a folder there.

Sidenote: I'm going on a few days vacation, and will be back to test the actions next week so I made this a draft PR for now. If someone with docker/actions experience wants to leave a quick review—awesome. When I'm back, I will be testing it with my private account until I can access a pandas docker hub account.

CC @jorisvandenbossche @mroeschke

WillAyd · 2022-08-16T16:23:04Z

gitpod/Dockerfile

+
+# -----------------------------------------------------------------------------
+# ---- Installing mamba  ----
+RUN wget -q -O mambaforge3.sh \


Do we even need to use conda / mamba within Docker? Wondering if it’s worth having an extra layer of virtualization versus simplification not going through all these steps

(Maybe related), if we are not expecting users to be able to alter the dev environment when using gitpod, maybe mamba/conda can be removed entirely? https://pythonspeed.com/articles/conda-docker-image-size/

If we want to keep using conda / mamba, another option could also be to use micromamba, which avoids having an additional base environment (the second point mentioned in the linked blog post for why the resulting docker images become big)

And we should probably also clean up the cached downloaded packages with conda.

I was thinking we could simply pip install from requirements.txt

FWIW I put some comparisons on the mamba within Docker versus without in #49981

FWIW I put some comparisons on the mamba within Docker versus without in #49981

Thanks, that's useful. I will comment about the usage of mamba there further.

WillAyd · 2022-08-16T16:40:20Z

.gitpod.yml

+
+# --------------------------------------------------------
+# some useful extensions to have
+vscode:


Is this common to add for gitpod? I don’t think all of our developers use VSCode so am somewhat hesitant to have a say on what “good” extensions are

I don’t think all of our developers use VSCode

We might rather want to look at contributors that do use VSCode, and see which extensions those typically use, to ensure we can give a good base experience here (if you don't use VSCode, you also won't have expectations about which extension should be enabled, so as long we provide a useful set, those users will be happy I think)

In any case I suppose we want to enable some extensions. For example at least the python extension? Further of course it's a bit the question how far we want to go (eg restructuredtext highlighting can be useful since our docs use that a lot, maybe cython highlighting could be added as well)

these seem useful, but I'd remove autodocstring as docstrings in pandas are often created via decorators

mroeschke · 2022-08-16T17:55:53Z

gitpod/Dockerfile

+RUN conda activate ${CONDA_ENV} && \
+    mamba install ccache -y && \
+    # needed for docs rendering later on
+    python -m pip install --no-cache-dir sphinx-autobuild && \


I wouldn't be opposed to adding this to environment.yml

jorisvandenbossche · 2022-08-16T19:30:10Z

gitpod/workspace_config

+git config --global alias.co checkout
+git config --global alias.ci commit
+git config --global alias.st status
+git config --global alias.br branch


Should we add those aliases? I can imagine that those will typically different for contributors (eg I use git s instead of git st for status ;))

I agree that this could be quite user-specific, so at first, I thought we sould remove them.
But on second thought, I can see how for very new users, it can be useful to put something in for sprints. They can be quickly introduced to using aliases without learning how to set them, and more experienced users can over-write them easily by setting their own aliases, no?!
WDYT, keep or remove?

github-actions · 2022-10-15T00:08:54Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

noatamir · 2022-11-09T22:43:22Z

I updated the docker files after local testing. The resulting images are on:
https://hub.docker.com/r/pythonpandas/pandas-dev
https://hub.docker.com/r/pythonpandas/pandas-gitpod

I tried to optimize using micromamba for a while, but couldn't make it work yet. For now, I prefer to have a working gitpod for the upcoming sprints. I am happy to give this another try at a later point in time.

MarcoGorelli

~~Thanks for working on this, curious to see how people will find using it~~

MarcoGorelli

Thanks for working on this, curious to see how people find using it

MarcoGorelli · 2022-11-10T10:43:49Z

.gitpod.yml

+
+# --------------------------------------------------------
+# some useful extensions to have
+vscode:


these seem useful, but I'd remove autodocstring as docstrings in pandas are often created via decorators

jorisvandenbossche · 2022-11-11T12:47:30Z

.gitpod.yml

+
+# assuming we use dockerhub: name of the docker user, docker image, tag, e.g. https://hub.docker.com/r/pandas/pandas-gitpod/tags
+image:
+  file: pandas-dev/pandas-gitpod:latest


Should this be updated?

No. Once we have the automation, the latest tag will always have the latest image, and otherwise we will also have an archive of previous images with a date tag. See https://hub.docker.com/r/numpy/numpy-gitpod/tags. For now, I have been manually over-riding the latest tag with pushes from my local machine.

Sorry, I wasn't very specific. I meant the "pandas-dev" org name (instead of "pythonpandas"), assuming this is the dockerhub org/image:version naming?

jorisvandenbossche · 2022-11-11T12:49:08Z

gitpod/Dockerfile

+# are visible in the host and container.
+# The docker image is retrieved from the pandas dockerhub repository
+#
+# docker run --rm -it -v $(pwd):/home/pandas pandas/pandas-dev:<image-tag>


jorisvandenbossche · 2022-11-11T12:51:59Z

gitpod/gitpod.Dockerfile

+FROM gitpod/workspace-base:latest as clone
+
+# the clone should be deep enough for versioneer to work
+RUN git clone https://github.com/pandas-dev/pandas --depth 12 /tmp/pandas


Is depth of 12 then deep enough? Or will this still automatically fetch tags? (otherwise you can also explicitly fetch tags in a separate command)

I thought it should be when we last chatted about this?!

jorisvandenbossche · 2022-11-11T12:54:42Z

gitpod/settings.json

+    "restructuredtext.linter.disabledLinters": ["doc8","rst-lint", "rstcheck"],
+    "python.defaultInterpreterPath": "/home/gitpod/mambaforge3/envs/pandas-dev/bin/python",
+    "esbonio.sphinx.buildDir": "${workspaceRoot}/doc/build/html",
+    "esbonio.sphinx.confDir": ""


esbonio is not in the list of extensions in gitpod.yml, would that be needed? (or those settings removed) Or is this a default extension?

.gitpod.yml

jorisvandenbossche · 2022-12-01T14:08:43Z

gitpod/Dockerfile

+RUN mamba env create -f /tmp/environment.yml
+# ---- Create conda environment ----
+RUN conda activate $CONDA_ENV && \


Suggested change

RUN mamba env create -f /tmp/environment.yml

# ---- Create conda environment ----

RUN conda activate $CONDA_ENV && \

# ---- Create conda environment ----

RUN mamba env create -f /tmp/environment.yml && \

conda activate $CONDA_ENV && \

Ensuring this is done in a single step makes this more efficient because then the conda clean also happens directly in the same layer

.gitpod.yml

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

jorisvandenbossche · 2022-12-01T22:22:09Z

I am planning to merge this, so that people can test it in practice and get some experience with it.

We can consolidate/improve the docker files in subsequent PRs (eg #48108)

jorisvandenbossche

Let's give this a try

WillAyd reviewed Aug 16, 2022

View reviewed changes

mroeschke reviewed Aug 16, 2022

View reviewed changes

mroeschke added the Build Library building on various platforms label Aug 16, 2022

jorisvandenbossche reviewed Aug 16, 2022

View reviewed changes

github-actions bot added the Stale label Oct 15, 2022

noatamir added 2 commits November 9, 2022 23:27

add gitpod files

ca20736

updated docker files after local testing

7fe67e6

noatamir force-pushed the gitpod branch from 5c40b4d to 7fe67e6 Compare November 9, 2022 22:34

clean up

1c8929f

noatamir marked this pull request as ready for review November 9, 2022 22:49

noatamir added 4 commits November 10, 2022 09:10

clean up

4b70c74

clean up

e527210

updated mamba version

1795077

clean up

4fd5cdd

MarcoGorelli reviewed Nov 10, 2022

View reviewed changes

removed autodocstring

bdbc11a

jorisvandenbossche reviewed Nov 11, 2022

View reviewed changes

noatamir mentioned this pull request Nov 11, 2022

DOC: Gitpod documentation #49642

Merged

noatamir commented Nov 16, 2022

View reviewed changes

.gitpod.yml Outdated Show resolved Hide resolved

noatamir added 2 commits November 16, 2022 15:41

Update .gitpod.yml

82e771e

Merge remote-tracking branch 'upstream/main' into gitpod

540b144

jorisvandenbossche reviewed Dec 1, 2022

View reviewed changes

.gitpod.yml Outdated Show resolved Hide resolved

noatamir and others added 3 commits December 1, 2022 15:24

Update .gitpod.yml

f3beca0

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Merge branch 'pandas-dev:main' into gitpod

424b199

remove esbonio

38e096e

jorisvandenbossche approved these changes Dec 2, 2022

View reviewed changes

jorisvandenbossche merged commit 39e0964 into pandas-dev:main Dec 2, 2022

jorisvandenbossche mentioned this pull request Dec 2, 2022

DEV: update gitpod docker #50007

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEV: Add gitpod files #48107

DEV: Add gitpod files #48107

noatamir commented Aug 16, 2022

WillAyd Aug 16, 2022

mroeschke Aug 16, 2022

jorisvandenbossche Aug 16, 2022 •

edited

Loading

WillAyd Aug 16, 2022

WillAyd Dec 1, 2022

jorisvandenbossche Dec 1, 2022

WillAyd Aug 16, 2022

jorisvandenbossche Aug 16, 2022

MarcoGorelli Nov 10, 2022

mroeschke Aug 16, 2022

jorisvandenbossche Aug 16, 2022

noatamir Sep 14, 2022

github-actions bot commented Oct 15, 2022

noatamir commented Nov 9, 2022

MarcoGorelli left a comment •

edited

Loading

MarcoGorelli left a comment

MarcoGorelli Nov 10, 2022

jorisvandenbossche Nov 11, 2022

noatamir Nov 11, 2022

jorisvandenbossche Nov 11, 2022

jorisvandenbossche Nov 11, 2022

jorisvandenbossche Nov 11, 2022

noatamir Nov 11, 2022

jorisvandenbossche Nov 11, 2022

jorisvandenbossche Dec 1, 2022

jorisvandenbossche commented Dec 1, 2022 •

edited

Loading

jorisvandenbossche left a comment

DEV: Add gitpod files #48107

DEV: Add gitpod files #48107

Conversation

noatamir commented Aug 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 15, 2022

noatamir commented Nov 9, 2022

MarcoGorelli left a comment • edited Loading

Choose a reason for hiding this comment

MarcoGorelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Dec 1, 2022 • edited Loading

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche Aug 16, 2022 •

edited

Loading

MarcoGorelli left a comment •

edited

Loading

jorisvandenbossche commented Dec 1, 2022 •

edited

Loading