Allow JSON output where it makes sense

[radeksimko]

As discussed previously elsewhere, it would be useful to have JSON output coming from certain terraform commands, especially when people use it as an abstraction layer in another project/tool instead of a standalone tool in CLI.

The assumption is we'd have -format=json argument accepted for the commands below.

At some point all the commands could have JSON output, but I do aim to cover at least the basic use-cases first:

apply

• apply implying -input=false & -no-color
• apply - list of errors
• apply - refresh events log (e.g. aws_vpc.main: Refreshing state... (ID: vpc-1a131c7f)) -> I'd begin with (potentially long) string 1:1 buffered output of what we have already
• apply - execution events log (e.g. aws_vpc.main: Creation complete) -> I'd begin with (potentially long) string 1:1 buffered output of what we have already
• apply - sums of modified resources (e.g. {"created": 1, "changed": 0, "removed": 0})
• apply - list of outputs

destroy

• destroy implying -force & -no-color
• destroy - list of errors
• destroy - refresh events log (e.g. aws_vpc.main: Refreshing state... (ID: vpc-1a131c7f)) -> I'd begin with (potentially long) string 1:1 buffered output of what we have already
• destroy - execution events log (e.g. aws_vpc.main: Destruction complete) -> I'd begin with (potentially long) string 1:1 buffered output of what we have already
• destroy - sums of modified resources (e.g. {"created": 0, "changed": 0, "removed": 3})

plan

• plan implying -no-color
• plan - refresh events log (e.g. aws_vpc.main: Refreshing state... (ID: vpc-1a131c7f)) -> I'd begin with (potentially long) string 1:1 buffered output of what we have already
• plan - readable version of terraform plan -out

output

This will likely require #1209 / #2335 / #2390 to make the output actually useful.

• output implies -no-color
• output - list of outputs

version

Just because any 3rd party tool likes to know that and because it's IMO super easy to do this one.

• version returns {"version": "0.6.0-dev", "git_commit": "62baf79+CHANGES"}

The suggested approach for events log will likely increase memory usage, but it's the easiest solution I can think of now. I'd love to see something more sophisticated in the future though - e.g. Terraform exposing sockets API that streams out event by event.

[svanharmelen]

Almost sounds like you're going to use TF as an API 😉

[radeksimko]

@svanharmelen Cool! That's basically what I'm looking for. Would you mind maintaining that as a branch here in upstream instead?

It would be much easier to rebase changes from upstream master + you can get rid of all those import changes and focus on the functionality + make it easier for cherry-picking and eventually easier to create a PR => make it part of Terraform.

[svanharmelen]

@radeksimko sure thing! If @mitchellh and @phinze are fine with that, I would certainly prefer that over having this kind of weird fork...

So @mitchellh and @phinze what are your votes on this suggestion?

[mitchellh]

The easiest way to implement this would be as a new UI hook that can be toggled with the -ui flag. I'll submit a PR now to make this flag exist so you can build on it. That way it requires no core changes to TF other than adding a new object...

[sl1pm4t]

What's the status of the terraform-api project?

[radeksimko]

I was talking to @svanharmelen recently at dotGo and he was mentioning a conversation w/ @mitchellh where he wasn't quite inclined to add "API functionality" into Terraform since it significantly widens the impact/focus of Terraform (correct me if I'm wrong in the interpretation either of you).

I was thinking about this for a while after dotGo and I must say I'm also inclined (I wasn't originally) to not have API included in TF core. It would probably simplify some of the projects I'm building on top of TF, but TBH all I care about is the output and I still believe we can do that without exposing everything over an API and bringing more complexity into Terraform's core.

Hence I'm keeping this issue open, because I believe it would be still useful to have a machine-readable output. Many CLI tools in this area have a machine-readable output (e.g. kubectl, AWS CLI) and most of them don't need an API to accomplish that.

What's the status of the terraform-api project?

@svanharmelen will be best to answer that question. I think it's now maintained in the linked project. We were also discussing how to maintain that project without having to rewrite all the imports which truly makes maintenance bothersome. I think it would make sense to treat https://github.com/xanzy/terraform-api as a separate project that will be importing hashicorp/terraform. I think most of the functionality is exposed via packages and if not, I think this would be a great way of fixing it.

[svanharmelen]

The current status of the terraform-api project is that it is fully functional and up-to-date with Terraform (give or take a few commits). I did however change the protocol from HTTP+JSON to gRPC a little while ago, as we switched our internal microservices to using gRPC.

Next to that I think I have a solution for the current approach which clearly isn't ideal (certainly with regards to the import rewriting stuff). When looking at the amount of code I added to make TF behave like an API, you can see it's actually very little code and mainly contained into the api package.

So I'm currently thinking about having a microservice that supports both an HTTP+JSON and gRPC endpoint that only imports Terraforms' packages. This way there is no need to fork the whole project and rewrite all imports. So that would make it a very lean and mean API wrapper around Terraform IMO.

[radeksimko]

#6299 seems to be addressing a few points from the list (output specifically).

[rjinski]

+1

[paulte]

+1

[luntzel]

+1

[apparentlymart]

The comments on #10665 probably apply to much of what's described here.

[OferE]

+1

[dellelce]

Is still this worked on ?

[apparentlymart]

Hi all! Thanks for the great discussion here.

Over time experience has shown us that this sort of "meta-issue" is not that useful, since it represents work that can potentially span across a long period of time.

Machine-readable output from Terraform is definitely a goal, but we are being cautious about it while the product is still pre-1.0 and so details about it are changing a lot between releases. Supporting JSON output would create compatibility constraints that are likely to hurt our ability to improve the core functionality of Terraform.

So for now I'm going to close this, to make it explicit that machine-readable output is not on the docket at the moment. When things are getting more stable, we'll come back to this by making a set of more-specific issues that each cover particular use-case, but it's too early to do that today.

[andrejs-sisojevs]

We have a case with our own representation of resources, which is the processed in various ways: configures environments in special ways and renders to terraform format. This way we are handling many deployments.
Test-covering this deployment framework would significantly benefit, if we could do offline terraform plan with output in JSON format, an verify it produces target resources/properties.
But currently there is no easy way to do such verification.
I understand the problem, that structure of plan can change significantly with time, but we could support those changes on our side. This format instability is not as bad, as having no JSON at all.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.