Support Draft for packaging

[jlewi]

Draft seems like a really nice tool for packing up code to deploy apps on K8s.

Should we make it easy for people to use Draft as a way of deploying TF training jobs using the CRD to K8s?

What would that take?

• I think we need to define a pack for TensorFlow.

It looks like Draft has a server side component. Do you need to setup INGRESS to use it or can you use it via kubectl proxy?

/cc @wbuchwalter @sozercan

For reference; here's a blog post about running it on GKE.

[squillace]

A draft pack contains a Dockerfile for the code and a Helm chart for deployment; unless there's something in a manifest or Ksonnet that cannot be expressed in a Helm chart -- and I'm not clear what that would be -- a custom draft pack is exactly what you would do. :-)

draft up --pack=<draft pack> would then apply the dockerfile and chart so that the developer doesn't have to. This would provide most anyone the bootstrapping needed to get a KubeFlow app up and running either locally or in any other K8s cluster to which they had access.

[bhack]

See also https://github.com/Azure/draft/issues/482

[bhack]

@jlewi do you have any info if Skaffold is planned to be GKE only or multicloud?

[jlewi]

No specific information; but I'd be very surprised if it was just GKE.

/cc @r2d4

[r2d4]

@bhack theres nothing specific to GKE in skaffold. In fact it works well on local clusters too (kubernetes for docker for desktop and minikube).

Happy to work with anyone here to help integrate skaffold in your workflow. I'm already in the kubeflow slack under mrick@.

[bhack]

@jlewi Do you think that this issue need to be moved at Kubeflow level or is it still specific of the tf-operator?

[jlewi]

@bhack probably makes sense to open a new issue either in kubeflow/kubeflow or maybe kubeflow/examples.

[gaocegege]

Do we want this now?

[bhack]

I like Skaffold but Draft has a more integrated tool for Ide and people like to do language server and debug in containers instead of replicating all the dependencies on the host.

[bhack]

P.s. And also breakpoint for remote debugging on the provisioned Minikube/Cluster pod..

[gaocegege]

It is so cool. While I think we use ksonnet to deploy our components, does it has conflicts with draft?

[bhack]

@gaocegege See https://github.com/Azure/draft/issues/482 and GoogleContainerTools/skaffold#10

[gaocegege]

Thanks for the link 👍

[jlewi]

If a user wanted to use draft or skaffold to create TFJobs or model deployments nothing would prevent them from doing so. My understanding of skaffold is that it's primarily intended for while you are developing. I think that it continually rebuilds the image as your code changes and updates your deployment. That workflow makes a lot of sense if your working on a webserver and want to continually update the server to see the latest changes. It's less clear to me what the update semantics would be for a job. Would an update restart the job and overwrite the output? Would we start a new job on each update and write to a new location?

…

On Wed, May 9, 2018, 5:07 AM Ce Gao ***@***.***> wrote: Thanks for the link 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#136 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAvcAz0ig9fgNHR_Fsk4vgoBSiQ8HIH6ks5twtwEgaJpZM4QaLHJ> .

[bhack]

@jlewi Generally we restart a new job and or we update the deployed job so that the pod restart but with a new location related to not ephemeral resources (mainly the model). So generally the policy is to use a new location trough the configmap or whatever you want to define you parameter. What really change is the policy about deleting the old location about permanent file related to the output especially if you write on a shared resources.

[bhack]

@jlewi Generally I think the dev loop automation behavior is defined by your interpretation of what is the "experimental/dev" phase of an experiment and what is the production phase of an experiment. See kubeflow/kubeflow#264 (comment)

[squillace]

We would generally love to help get a custom draft-pack working with kubeflow. I would think that skaffold would be great for it as well -- the community will naturally choose the tooling it prefers, as it should, but these are two great tools to help people use KubeFlow.

[bhack]

@squillace We have also a more general issue at KubeFlow level at kubeflow/kubeflow#465

[squillace]

The general issue being a skaffold one? Is there an inner issue I can look at here? I'm missing something.

[bhack]

I will re-edit the title so that you can comment the entry point of Draft and we can compare with Skaffold.

[jlewi]

@squillace I think it would be very helpful to have concrete examples illustrating how draft and/or skaffold can be used.

Some areas where Draft might be useful

• Launching tensorboard
• Launching TFJobs
• Deploying models
• Argo workflows

For TFJobs I think we can actually ignore the subtleties of TensorFlow and TFJob and consider the following use cae

• I have a shell script "hello_world.sh"
• I want to build a Docker image with the script
• I want to submit a K8s job with that script

This use case closely models what we would do with TFJob but avoids some of the complexities of using TFJob and TensorFlow.

If we had a concrete example of this (perhaps one already exists) that would help me at least understand how draft could be used to simplify the corresponding workflow using TFJobs as opposed to K8s jobs.

/cc @wbuchwalter

[squillace]

I chatted with @wbuchwalter about this at kubecon, and we shall definitely kick off some examples. In general, draft builds containers and deploys helm charts for iterative development, which largely means you can use draft for almost any thing that is kubernetes native.

I'll hop back into this once I get back home (end of the week).

[inc0]

@squillace hey:) we also chatted about that (I'm this long-haired Intel guy). I've started working towards having available private registry for pushing images, which ofc is problem of on-prem.

[squillace]

cheers! looks like we got all the peeps to make some progress to help people make the best of kubeflow whatever they want to use, which is great.

[inc0]

@jlewi also Draft will solve slightly different use case - image mgmt or, what we discussed, code -> image path.
In short me, @squillace and @wbuchwalter discussed idea where we could use draft to point to local code, draft would then use this to build docker image in cluster (using img or something) and publish this image in registry somewhere.

[bhack]

I really hope also that the two teams could share common components of the ide integration tools to improve dev experience on in pod language server support and remote debugging:

GoogleContainerTools/skaffold#348

[jlewi]

I'm closing this issue as stale.