Use K8s also for local development

[sschuberth]

Maintaining the Docker Compose setup for local development only is cumbersome. We should try to share as much infrastructure setup as possible with the production K8s deployment. As such, probably using any of https://github.com/k3d-io/k3d, https://github.com/kubernetes/minikube or https://github.com/kubernetes-sigs/kind should replace the current Docker Compose setup.

[mnonnenmacher]

Maintaining the Docker Compose setup for local development only is cumbersome.

Why? Docker Compose is a lot simpler than Kubernetes. Just compare docker-compose.yml with the content of scripts/helm, and the latter is not even up-to-date and would have to be extended to work.

We should try to share as much infrastructure setup as possible with the production K8s deployment. As such, probably using any of https://github.com/k3d-io/k3d, https://github.com/kubernetes/minikube or https://github.com/kubernetes-sigs/kind should replace the current Docker Compose setup.

I think we should provide an up-to-date Helm chart to make it easy to deploy the server to any Kubernetes cluster including a local one. However, to me there are several good arguments to keep the Docker Compose file:

• Like I said before, it is a lot simpler than the Kubernetes configuration.
• It reduces the hurdle to test the server for potential users. Many more people have Docker installed than a Kubernetes cluster at hand (even a local one), and just cloning the repository an running docker compose up is easy.
• It consumes less resources which is good when using it for testing during development.
• A lot of development tasks do not require the added complexity of Kubernetes and can be tested just fine in the Docker Compose environment.

[sschuberth]

Why?

Well, it's maintaining two ways instead of one to run / deploy ORT server. Ideally, there should not be much difference between deploying locally / for development and deploying for production, besides the "target" and the respective infrastructure configurations. But both could be done with basically the same tooling.

Docker Compose is a lot simpler than Kubernetes.

Sure. That's exactly the topic that the above-mentioned "Mini-K8s-Distributions" try to address.

Just compare docker-compose.yml with the content of scripts/helm

Still, it would be nice to only have to maintain either, not both.

the latter is not even up-to-date and would have to be extended to work.

True, but I thought that was planned for quite some time anyway.

I think we should provide an up-to-date Helm chart to make it easy to deploy the server to any Kubernetes cluster including a local one.

Exactly. Maybe I'm missing something due to my limited K8s knowledge, but again I though that's exactly the purpose of the mentioned tools.

there are several good arguments to keep the Docker Compose file:
[...]
Like I said before, it is a lot simpler than the Kubernetes configuration.

Maybe tutorials like this and tools like this help to convince you.

Many more people have Docker installed than a Kubernetes cluster at hand

Note that e.g. k3d actually runs k3s in Docker.

It consumes less resources which is good when using it for testing during development.

You might want to have a look at this table, even if there is not direct comparison to Docker Compose.

[sschuberth]

Also see What is k3d Used for?:

The main use case for k3d is local development on Kubernetes with little hassle and resource usage. The intention behind the initial development of k3d was to provide developers with an easy tool that allowed them to run a lightweight Kubernetes cluster on their development machine, giving them fast iteration times in a production-like environment (as opposed to running docker-compose locally vs. Kubernetes in production).

[mnonnenmacher]

Well, it's maintaining two ways instead of one to run / deploy ORT server.

Yes, but I don't see the the Compose file causes much maintenance effort.

Docker Compose is a lot simpler than Kubernetes.

Sure. That's exactly the topic that the above-mentioned "Mini-K8s-Distributions" try to address.

They make setting up Kubernetes easier, but not making deployments. To me deploying a Helm chart is more complex than running docker compose up.

the latter is not even up-to-date and would have to be extended to work.

True, but I thought that was planned for quite some time anyway.

Yes, but making a generic Helm chart is a lot more difficult than making one for a specific environment because it needs to offer many configuration options to be usable. For example, for local development it would have to deploy a message service like RabbitMQ, in production likely some managed service would be used instead.

Maybe tutorials like this and tools like this help to convince you.

Not really. I'm not against anyone updating the Helm chart and using Kubernetes for local development. But I don't like the idea of making Kubernetes a hard requirement for running the server, mainly for these reasons:

• Like mentioned earlier, I believe that requiring a Kubernetes environment to be able to test the server is bad for adoption.
• I would really like if users that do not need to handle a high load could simply set up the server on a single host as a monolith, avoiding all the hassle that Kubernetes brings.

[sschuberth]

I would really like if users that do not need to handle a high load could simply set up the server on a single host as a monolith, avoiding all the hassle that Kubernetes brings.

Ok, that makes sense.

[sschuberth]

I think we should provide an up-to-date Helm chart to make it easy to deploy the server to any Kubernetes cluster including a local one.

As somewhat related news to that, JKube (which we currently do not use) can now "run tests on your Helm charts".