[WIP] Add Kubernetes provider

[radeksimko]

Resubmitting #3132 from hashicorp branch since there will be a lot more work involved in this and @lwander volunteered to help out. 😉

Feel free to submit PRs against this new branch.

[xied75]

Guys, when are we going to have this into the master?

[lwander]

@xied75, once we have the full implementation finished; you can see partial progress here: #3584. Since the resources are so verbose (about 1KLOC for just the pod description) I ran a proposal by the Hashicorp guys on a way to speed up this implementation, but haven't heard back yet - it might be time for me to speed up implementation again.

[Fodoj]

@radeksimko what are your plans on this PR?

[hingstarne]

+1 Would love to see it implemented

[radeksimko]

Thanks for the interest, the current state (@lwander correct me if I'm wrong) is that we'd love to have it in upstream, but we aren't sure about the approach and weight of all pros/cons of either translating the K8S API fields of all resources completely into the Terraform's HCL or letting the user define everything via YAML/JSON.

Each approach has different caveats which were mostly discussed in #3132

What is your POV on this @Fodoj @hingstarne ?

[lwander]

Yes, that's correct. The pod resource is fully implemented using Terraform's config language, and the PR is huge: #3584. I'd prefer to switch to code generation to generate provider resources to avoid this kind of thing before writing another several thousand lines of code.

[mtougeron]

Hi, wanted to throw in my 2c on this. Having a provider where we have to use yaml/json for spec is better than having nothing at all. Then having another release/update with the full HCL implementation for spec once the code generator part is figured out.

At the moment we are "hacking" things together by using the template resource and local-exec provider for interacting with Kubernetes. Having the basic kubernetes resources would save us a ton of headaches.

Thanks for your great work!
-Mike

[pmoust]

What are the cons in letting users defining K8s stuff in json/yaml @radeksimko from your PoV?

[radeksimko]

@pmoust From user's perspective there may be a few - non-idiomatic Terraform configs is one I can think of. e.g. it would be tricky/impossible to make "subresources" off resources that you'd configure via JSON/YAML.

Imagine there was something worth managing separately (to make use of count and/or interpolation functions) - e.g. kubernetes_pod_container which would be a subset of kubernetes_pod YAML/JSON definition.

From Terraform's developer perspective it is very tricky to get around the way K8S API works where you send an array [a, b, c] to the Create API and then you Get back [a, b, c, d]. This happens for example with pods that get some secret volumes attached automatically, but happens with most other resources I had the chance to play with. The whitelisting/blacklisting is tricky part.

[lwander]

Also - not all Kubernetes resource attributes can be updated, but the Terraform provider cannot be made aware of this unless it parses and reads all the JSON/YAML input.

[errordeveloper]

Would love to try this out and help where I can, but it'd much easier if it was in a separate repo, i.e. a plugin.

[devth]

👍 I'd also like to see K8S ConfigMap support in provider/kubernetes.

[Seraf]

👍, I'd be happy to help on this topic, we will need it soon for our internal usage.

[radeksimko]

Hi folks,
I'd be happy for more people to get involved in this + do some work myself.

The K8S provider is somewhat special in terms of size & complexity of the API hence there are some core-enhancements that most likely need to be done to make creation and ongoing maintenance of such provider less painful:

1. Nestable resources - https://github.com/hashicorp/terraform/issues/2275
2. Have a way to generate (or at least partially generate) resource schemas. I don't think it's a good idea to reimplement the whole K8S API by hand in schema and keep maintaining it by hand. See Add Kubernetes provider #3132 & helper/schema: arbitrary JSON attributes #3026 (comment) to understand why handling raw JSON is probably not the right approach either.
3. Have a way to generate (or at least partially generate) docs - see WIP at https://github.com/jen20/larceny

Suggestions and (WIP) PRs regarding each point are welcomed. 😉

[radeksimko]

As discussed privately with @lwander I'm closing this stale PR for now along with #3584 .

The Kubernetes provider implementation is open for anyone else to revisit, but I'd highly recommend doing so after the mentioned things are at least partially solved.

Implementing even a very basic (and actually useful) K8S provider manually (like I did here) doesn't seem to be a good approach which is a lesson learnt from this closed PR - so in that sense the effort & time here was not wasted.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.