Skip to content

Commit

Permalink
chore: Add a early dev notice (#253)
Browse files Browse the repository at this point in the history 
Signed-off-by: Ce Gao <cegao@tensorchord.ai>
  • Loading branch information
@gaocegege
gaocegege committed Jun 9, 2022
1 parent 08a75eb commit b64dbf8
Showing 1 changed file with 90 additions and 63 deletions.
153 changes: 90 additions & 63 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,21 @@
<a href="https://trackgit.com"><img src="https://us-central1-trackgit-analytics.cloudfunctions.net/token/ping/l3ldvdaswvnjpty9u7l3" alt="trackgit-views" /></a>
</p>

> **⚠️ envd is still under heavy development, and subject to change. it is not feature-complete or production-ready. Please contact us in [discord](https://discord.gg/KqswhpVgdU) if there is any problem.**
envd is a development environment management tool for data scientists.

🐍 **No docker, only python** - Write python code to build the development environment, we help you take care of Docker.
🐍 **No docker, only python** - Write python code, we help you take care of Docker, CUDA, SSH keys and so on.

🖨️ **Built-in jupyter/vscode** - Jupyter and VSCode remote extension are the first-class support.
🖨️ **Built-in jupyter/vscode** - Jupyter and VSCode remote extension are first-class support.

⏱️ **Save time** - Better cache management to save your time, keep the focus on the model, instead of dependencies
⏱️ **Save time** - Better cache management to save your time, keep the focus on the model, instead of dependencies.

☁️ **Local & cloud** - Run the environment locally or in the cloud, without any code change
☁️ **Local & cloud** - Run the environment locally or in the cloud, without any code change.

🐳 **Container native** - Leverage container technologies but no need to learn how to use them, we optimize it for you
🐳 **Container native** - Leverage container technologies but no need to learn how to use them, we optimize it for you.

🤟 **Infrastructure as code** - Describe your project in a declarative way, 100% reproducible
🤟 **Infrastructure as code** - Describe your project in a declarative way, 100% reproducible.

## Why use envd?

Expand All @@ -35,85 +37,114 @@ envd provides another way to solve the problem. As the infra guys, we accept the

envd provides build language similar to Python and has first-class support for jupyter, vscode, and python dependencies in container technologies.

## How does envd work?
## Documentation

See [envd documentation](https://envd.tensorchord.ai/docs/intro).

## Getting Started

Get started by **creating a new envd environment**.

### What you'll need

## Install
- Docker (20.10.0 or above)

### From binary
### Install envd

You can download the binary from the [latest release page](https://github.com/tensorchord/envd/releases/latest).
You can download the binary from the [latest release page](https://github.com/tensorchord/envd/releases/latest), and add it in `$PATH`.

After the download, please run `envd bootstrap` to bootstrap.

### From source code
> You can add `--dockerhub-mirror` or `-m` flag when running `envd boostrap`, to configure the mirror for docker.io registry:
>
>```bash title="Set docker mirror"
>envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn
>```
```bash
git clone https://github.com/tensorchord/envd
go mod tidy
make
./bin/envd --version
```
### Create an envd environment
## Quickstart
Please clone the [`envd-quick-start`](https://github.com/tensorchord/envd-quick-start):
```
git clone https://github.com/tensorchord/envd-quick-start.git
```
Checkout the [examples](./examples/mnist), and configure envd with the manifest `build.envd`:
The build manifest `build.envd` looks like:

```python
```python title=build.envd
def build():
install.vscode_extensions([
"ms-python.python",
])

base(os="ubuntu20.04", language="python3")
install.python_packages(name = [
"tensorflow",
"numpy",
])
install.cuda(version="11.6", cudnn="8")
shell("zsh")
config.jupyter(password="", port=8888)
```

Then you can run `envd up` to create the development environment.

<a href="https://asciinema.org/a/498012" target="_blank"><img src="https://asciinema.org/a/498012.svg" /></a>
Then please run the command below to setup a new environment:

TODO: illustrate that the cache will be persistent.
```
cd envd-quick-start && envd up
```

```
$ envd up
[+] ⌚ parse build.envd and download/cache dependencies 0.0s ✅ (finished)
=> 💽 (cached) download oh-my-zsh 0.0s
=> 💽 (cached) download ms-python.python 0.0s
[+] 🐋 build envd environment 7.7s (24/25)
=> 💽 (cached) (built-in packages) apt-get install curl openssh-client g 0.0s
=> 💽 (cached) create user group envd 0.0s
=> 💽 (cached) create user envd 0.0s
=> 💽 (cached) add user envd to sudoers 0.0s
=> 💽 (cached) (user-defined packages) apt-get install screenfetch 0.0s
=> 💽 (cached) install system packages 0.0s
=> 💽 (cached) pip install jupyter 0.0s
=> 💽 (cached) install PyPI packages 0.0s
=> 💽 (cached) install envd-ssh 0.0s
=> 💽 (cached) install vscode plugin ms-python.python 0.0s
=> 💽 (cached) copy /oh-my-zsh /home/envd/.oh-my-zsh 0.0s
=> 💽 (cached) mkfile /home/envd/install.sh 0.0s
=> 💽 (cached) install oh-my-zsh 0.0s
...
# You are in the docker container for dev
(envd 🐳) ➜ mnist-dev git:(master) python3 ./main.py
$ cd envd-quick-start && envd up
[+] ⌚ parse build.envd and download/cache dependencies 2.8s ✅ (finished)
=> download oh-my-zsh 2.8s
[+] 🐋 build envd environment 18.3s (25/25) ✅ (finished)
=> create apt source dir 0.0s
=> local://cache-dir 0.1s
=> => transferring cache-dir: 5.12MB 0.1s
...
=> pip install numpy 13.0s
=> copy /oh-my-zsh /home/envd/.oh-my-zsh 0.1s
=> mkfile /home/envd/install.sh 0.0s
=> install oh-my-zsh 0.1s
=> mkfile /home/envd/.zshrc 0.0s
=> install shell 0.0s
=> install PyPI packages 0.0s
=> merging all components into one 0.3s
=> => merging 0.3s
=> mkfile /home/envd/.gitconfig 0.0s
=> exporting to oci image format 2.4s
=> => exporting layers 2.0s
=> => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f 0.0s
=> => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717 0.0s
=> => sending tarball 0.4s
(envd) ➜ demo git:(master) ✗ # You are in the container-based environment!
```

Jupyter notebook service and sshd server are running inside the container. You can use jupyter or vscode remote-ssh extension to develop AI/ML models.
### Play with the environment

You can run `ssh envd-quick-start.envd` to reconnect if you exit from the environment. Or you can execute `git` or `python` commands inside.

```bash
$ python demo.py
[2 3 4]
$ git fetch
$
```
$ envd get envs
NAME JUPYTER SSH TARGET CONTEXT IMAGE GPU CUDA CUDNN STATUS CONTAINER ID
mnist http://localhost:9999 mnist.envd /mnist mnist:dev true 11.6 8 Up 23 hours 74a9f1007004
$ envd get images
NAME CONTEXT GPU CUDA CUDNN IMAGE ID CREATED SIZE
mnist:dev /mnist true 11.6 8 034ae55c5f4f 23 hours ago 7.28GB

### Setup jupyter notebook

Please edit the `build.envd` to enable jupyter notebook:

```python title=build.envd
def build():
base(os="ubuntu20.04", language="python3")
install.python_packages(name = [
"numpy",
])
shell("zsh")
config.jupyter(password="", port=8888)
```

You can get the endpoint of jupyter notebook via `envd get envs`.

```bash
$ envd up --detach
$ envd get env
NAME JUPYTER SSH TARGET CONTEXT IMAGE GPU CUDA CUDNN STATUS CONTAINER ID
envd-quick-start http://localhost:8888 envd-quick-start.envd /home/gaocegege/code/envd-quick-start envd-quick-start:dev false <none> <none> Up 54 seconds bd3f6a729e94
```

## Features
Expand Down Expand Up @@ -155,10 +186,6 @@ install.vscode_extensions([
])
```

## Join Us

envd is backed by [TensorChord](https://github.com/tensorchord) and licensed under Apache-2.0. We are actively hiring engineers to build developer tools for machine learning practitioners in open source.

## Contribute

We welcome all kinds of contributions from the open-source community, individuals, and partners.
Expand Down

0 comments on commit b64dbf8

Please sign in to comment.