Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

command: terraform show -format=json #10665

Closed
wants to merge 3 commits into from
Closed

Conversation

apparentlymart
Copy link
Contributor

@apparentlymart apparentlymart commented Dec 12, 2016

This set of changes adds a -format=json option to the terraform show command to allow users to obtain machine-readable plan and state output.

There are (and have been) several other variants of adding some JSON output support to Terraform:

  • In [WIP] JSON plan format #4610 I tried to make JSON the primary format for plans, for consistency with states. However, this would've likely tempted people to try to hand-edit plan files, which is likely to cause Terraform to misbehave or crash.
  • Initial stab at JSON plan output #3170 proposes to add JSON output to the terraform plan command itself. That would work, but is a little problematic because the plan command produces more output than just the plan, including the log of refreshing any existing resources and warnings about various things, and so it feels better (to me) to leave terraform plan as human-readable and provide a separate command for JSONifying plans.
  • Allow JSON output where it makes sense #2460 is a bit of a "meta-issue" aiming to get JSON support in a bunch of different places. JSON-readable logs of Terraform's ongoing behavior would be interesting, but is a lot of work to bite off.

So this change focuses on one primary use-case: I already made a plan using terraform plan -out=... and now I want to use some separate tool to analyze the plan.

$ terraform plan -out=tfplan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: tfplan

(... etc, etc ...)

$ terraform show -format=json tfplan
{
   ...a big blob of JSON...
}

terraform show is used because it is focused on obtaining information without any other side-effects. This keeps the multi-format complexity from spreading all over the command module and should make a reasonable interface for producing a plan once and then safely rendering it in a number of different formats. (A maximum of two to start, but who knows...)

Since it is the terraform show command that is being extended here, it also supports JSON output of states. This is pretty redundant, since the state is already JSON-readable in exactly the same format in the local state file. It is thus included just for consistency and completeness for now, but could become more useful in future if, for example, we were to implement encryption of the local state file at rest (#9556), or if the local cache were to go away altogether as part of some forthcoming redesign of the remote state management workflow (#1964 or something like it).

This includes some of the work I did for #4610, though it required some tweaking to catch up with changes to Terraform's data structures in the mean time.


A big reason to be cautious about supporting something like this is that it creates a pretty big API surface area to maintain compatibility with. Terraform is still evolving very fast, so it's basically impossible to retain 100% compatibility with any machine-readable format. As a pragmatic compromise this change adds a JSON format for adventurous people to use, but specifically disclaims any defined behavior of it across Terraform versions and declines to document it in any way.

My thought was that for now this would remain a "buyer beware" sort of feature for adventurous early adopters, and then we'll get some experience with what people tend to do with it and then, at some later point, possibly commit to some level of compatibility for this format and produce some documentation on it.

Making the plan, diff and configuration structures JSON-serializable
creates the possibility for machine-readable Terraform output of various
kinds.
There are some steps that get done before writing a state to disk.

Here we factor out those operations so that they can be used when a
state is included as part of another structure that is being serialized,
such as a plan.
As a simple, non-intrusive way to get machine-readable plan output, add
a -format=json argument to the "terraform show" command. This then allows
external tools to interrogate plan files, e.g. to enforce additional
restrictions on what can be changed that Terraform itself is unable to
represent.

For completeness this also supports JSON-formatted state output, but of
course that is rather less interesting because the on-disk state file is
already in this same format.
@mitchellh
Copy link
Contributor

There is probably meaningful work in here, but we're going to be completely revamping terraform show since it is not usable very quickly as state size increases. JSON output may help with that but there are other things I don't like about this:

The JSON format now creates some sort of API promise implicitly (though we definitely explicitly don't) of the structure, but the diff/state structure is going to continue to change heavily in upcoming versions, so anyone relying on this format will have to update their tools for each new version.

This has been a primary de-motivator for me for JSON output for state/diffs and why I push folks to just use UNIX tools with terraform state commands if possible for example. I understand there is a large class of problems that can't be solved that way, though.

And the people super motivated can easily (Go knowledge required, but past that it is easy) use the Terraform Go API (not promised stable but you can vendor) to parse states/plans and do clever things. That is a VERY high touch solution but at least its definitely "buyer beware" as you might say.

So given that... I'm gonna say "no" to this and close it but I don't think your work will be a waste: we probably can and should adopt various parts of your diff in the future and knowing this exists will be useful.

@mitchellh mitchellh closed this Dec 12, 2016
@apparentlymart
Copy link
Contributor Author

Cool... that all makes sense. Thanks for the explanation, @mitchellh. A chunk of the code here was already from another rejected PR so maybe it will get a third chance later. 😀

As an aside: I previously tried to make use of Terraform's "API" from another Go program in the past; I guess it was probably my prototype of wrapping Terraform in a Packer-like UI. Ever since Terraform started using govendor this sort of usage has been generally broken, because any time a vendored library appears in Terraform's public interface (which happens with e.g. HIL's AST and HCL's concepts) you end up with types whose names cannot be written down by any code outside of the Terraform package.

Of course I don't expect anything to be done to fix that, since Terraform is primarily an application rather than a library, but just something to keep in mind when thinking about ways people might want to integrate with Terraform.

@mitchellh mitchellh deleted the f-show-json branch December 12, 2016 22:04
@jalil
Copy link

jalil commented Dec 16, 2016

Hey Michell,
Can you give use the link to the Terraform Go API with some samples usages.

Thank you

@hraban
Copy link

hraban commented Dec 28, 2016

Hi @jalil I think he's talking about this very package we're commenting on right now. See #415 (comment)

@gazoakley
Copy link
Contributor

It's a shame we can't have this built into Terraform - it'd be particularly helpful to include an option to strip away any attributes marked as sensitive in the schema. You could then build tools which take in plans/states for analysis. Maybe the exported JSON could include a schema semver?

@ghost
Copy link

ghost commented Apr 13, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants