feat: Add envd-server runtime proposal

[gaocegege]

Ref #179

Proposal preview: https://github.com/gaocegege/envd/blob/proposal/docs/proposals/20220603-kubernetes-vendor.md

[gaocegege]

/cc @Xuanwo @hezhizhen @knight42

[Xuanwo]

Will envd play with charts?

[gaocegege]

Do you mean helm charts?

[Xuanwo]

Yes

[gaocegege]

Some users expect this, but I am not sure how to integrate it with charts.

[VoVAllen]

Overall looks good to me. One thing I'm concerning is that whether syncing is needed at MVP. Code should be on a PV-like thing I think and usually people works with git. Might not need syncing when working on cluster.

The problem of syncing is about the dataset related stuffs. User may need fine-grained control of what to sync, which increases the complexity

[VoVAllen]

Other options for file syncing:

• reverse sshfs https://github.com/lima-vm/sshocker
• ksync https://github.com/ksync/ksync

[knight42]

One thing I'm concerning is that whether syncing is needed at MVP. Code should be on a PV-like thing I think and usually people works with git

My 2 cents is syncing might be needed ultimately, but might in some different ways. Let's say we are working with git, and push some new commits to a branch, I think it would be better if envd could pull the new commits automatically to simulate the local development experience.

[gaocegege]

My 2 cents is syncing might be needed ultimately, but might in some different ways. Let's say we are working with git, and push some new commits to a branch, I think it would be better if envd could pull the new commits automatically to simulate the local development experience.

Thanks for the advice! Automatic push/pull looks magic to me. And it is complex. If the container is crashed, we also may lost the commits if we do not run push.

[gaocegege]

As discussed with some infra engineers interested in envd, port-forwarding may consume many API server CPUs. And tools like virtual kubelet does not support port forwarding.

[knight42]

port-forwarding may consume many API server CPUs.

Would you mind elaborating? AFAIK if there is no much traffic, port-forwarding should not consume too much cpu resources, as it is simply a SPDY connection under the hood.

virtual kubelet does not support port forwarding

Indeed. But what are we going to do to access the services inside the container without port-forwarding?

[gaocegege]

Thanks for the advice! Automatic push/pull looks magic to me. And it is complex. If the container is crashed, we also may lost the commits if we do not run push.

Or, we can forget the sync things. We request users to develop on the remote container, instead of the host. The build.envd may look like this:

def build():
    base(os="ubuntu20.04", language="python3")
    install.vscode_extensions([
        "ms-python.python",
    ])
    #config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
    install.python_packages([
        "tensorflow",
        "numpy",
    ])
    shell("zsh")
    config.jupyter(password="", port=8888)
+    config.working_dir(local=".", remote="https://github.com/tensorchord/envd.git")

envd mounts the local dir with docker runner and downloads the repo with Kubernetes runner.

[gaocegege]

Would you mind elaborating? AFAIK if there is no much traffic, port-forwarding should not consume too much cpu resources, as it is simply a SPDY connection under the hood.

There should not be huge traffic by design. But algorithm engineers may use it to copy data:

scp <10G-file> container:~

[gaocegege]

But what are we going to do to access the services inside the container without port-forwarding?

They may use service and ingress to achieve this. Thus we may need a mechanism to support customization here.

Maybe just like the design of the device plugin, we provide an interface to communicate between envd and a CLI shim. The shim does the critical logic like port forwarding. The envd just communicate with the shim and show information to users.

Port forwarding can be used in our default shim, while users can write their own shim to customize, e.g. using service and ingress.

[gaocegege]

But what are we going to do to access the services inside the container without port-forwarding?

They may use service and ingress to achieve this. Thus we may need a mechanism to support customization here.

Maybe just like the design of the device plugin, we provide an interface to communicate between envd and a CLI shim. The shim does the critical logic like port forwarding. The envd just communicate with the shim and show information to users.

Port forwarding can be used in our default shim, while users can write their own shim to customize, e.g. using service and ingress.

Device plugin is a not-so-good comparison. Let's say kubectl plugin mechanism. The shim maintained by users can be integrated into the envd.

[knight42]

• config.working_dir(local=".", remote="https://github.com/tensorchord/envd.git")

My concern is that working dir seems to be a command line argument to me, otherwise it might prevent the reuse of build.envd in different working dir, just like we don't specify the build context in Dockerfile. Besides if we need to specify the repo address, should we need to specify the branch as well?

algorithm engineers may use it to copy data:

Got it 👌 If we need to transfer such huge file via port-forwarding without rate limiting, the functionality of apiserver might be affected.

while users can write their own shim to customize, e.g. using service and ingress.

I think it make sense 👍

[gaocegege]

My concern is that working dir seems to be a command line argument to me, otherwise it might prevent the reuse of build.envd in different working dir, just like we don't specify the build context in Dockerfile. Besides if we need to specify the repo address, should we need to specify the branch as well?

Sounds reasonable. It should be a runtime argument instead of build time.

@VoVAllen Do you have opinion on it?

[VoVAllen]

Agree it better to be a runtime option. If using git, user should handle the sync related thing by himself (clone git repo and git pull/push). Therefore config.working_dir might not be needed.

[VoVAllen]

Found one new syncing tool:
https://github.com/mutagen-io/mutagen

[aseaday]

How is it going now?

[gaocegege]

PTAL

[Xuanwo]

Typo cleanup

[gaocegege]

@Xuanwo Thanks for your fix!

[kemingy]

@Xuanwo Thanks for your fix!

Probably you should try Grammarly. There are still some syntax errors.

[kemingy]

Shall we expose the backend implementation to the starlark frontend?
Do we need to rename the current Docker-specific implementations?

[gaocegege]

Not sure about it. I opened an issue to keep track #530

[gaocegege]

@Xuanwo Thanks for your fix!

Probably you should try Grammarly. There are still some syntax errors.

It should be fixed. PTAL.

[gaocegege]

I am merging it since we are already starting the development of envd-server.