Support running apply and other commands with input from stdin

[mattrobenolt]

I'm working towards building something that leverages topicctl for our kafka cluster in kubernetes, and with this, my plan is to have a simple wrapper around topicctl that listened to a Kubernetes ConfigMap containing cluster configs. On ConfigMap changes, it'd apply to the cluster.

To better support that, it'd be nice to be able to pass everything through stdin. I looked briefly at code thinking it'd be simple, but there are some assumptions made with file system layout (discovering cluster config.yaml) and not handling multiple documents (#26).

I think ideally, it'd be nice to pass one big document through stdin that contained the cluster config as well as the topics through stdin. Then our loop in Kubernetes will be:

while:
  configmap <- configmap.wait_for_change()
  configmap -> subprocess topicctl apply -

Am I just off base on wanting to use topicctl in this manner? It seems like a pretty obvious thing for us to do. We can work around the issues by having another intermediate step where we write all the configs to disk first, then call topicctl apply topics/*yml or whatever, but is an unnecessary step if we can pass through stdin.

Let me know your thoughts, we haven't started building anything for this yet, just brainstorming.

[yolken-segment]

Thanks for the suggestion. This is not unreasonable given that kubectl and other, similar type tools typically allow input from stdin. However, it will require some non-trivial refactoring in the various entrypoints, so I would advise sticking with the "write to disk" intermediate step for now. From past experience, the latter is a fairly common pattern when trying to get tools with non-Kubernetes-optimized interfaces working in Kubernetes.

[mattrobenolt]

Yeah, for sure. And I didn't want to start a large refactor to support this without at least some communication. It's easy enough to write to disk as an intermediate solution.

[yolken-segment]

Cool! #26 is a lot easier to implement and I could see that helping in more situations, so I'll take a crack at that one soon.

[mattrobenolt]

Just to add another bump for support here, another use case for us would be another tool which generated the configs that we could write to stdout and pass in to topicctl.

Something like: sentry config generate-topicctl | topicctl apply -

I think the doors will open up for lots of possibilities if we can do this for us.