terraform show, terraform state show ADDR, and terraform show -json not showing all (or consistent) data

[lexrj]

Terraform Version

Terraform v0.12.1
+ provider.datadog v1.9.0

Terraform Output

No configs except for the standard boilerplate ones.

Same for terraform show and terraform state show

# module.datadog.datadog_timeboard.alexiss-timeboard-6-jun-2019-1341: 
resource "datadog_timeboard" "alexiss-timeboard-6-jun-2019-1341" {
    id    = "1234573"
    title = "Alexis's Timeboard 6 Jun 2019 13:41"

    graph {
        autoscale               = true
        events                  = []
        group                   = []
        include_no_metric_hosts = false
        include_ungrouped_hosts = false
        precision               = "2"
        scope                   = []
        style                   = {}
        title                   = "Avg of system.load.1 over *"
        viz                     = "query_value"
        yaxis                   = {}

        request {
            aggregator    = "avg"
            increase_good = false
            q             = "avg:system.load.1{*}"
            stacked       = false
            style         = {}
        }
    }
}

terraform show -json | jq . (The most complete)

{
  "format_version": "0.1",
  "terraform_version": "0.12.1",
  "values": {
    "root_module": {
      "child_modules": [
        {
          "resources": [
            {
              "address": "datadog_timeboard.alexiss-timeboard-6-jun-2019-1341",
              "mode": "managed",
              "type": "datadog_timeboard",
              "name": "alexiss-timeboard-6-jun-2019-1341",
              "provider_name": "datadog",
              "schema_version": 0,
              "values": {
                "description": "",
                "graph": [
                  {
                    "autoscale": true,
                    "custom_unit": "",
                    "events": [],
                    "group": [],
                    "include_no_metric_hosts": false,
                    "include_ungrouped_hosts": false,
                    "marker": [],
                    "node_type": "",
                    "precision": "2",
                    "request": [
                      {
                        "aggregator": "avg",
                        "change_type": "",
                        "compare_to": "",
                        "conditional_format": [],
                        "extra_col": "",
                        "increase_good": false,
                        "order_by": "",
                        "order_direction": "",
                        "q": "avg:system.load.1{*}",
                        "stacked": false,
                        "style": {},
                        "type": ""
                      }
                    ],
                    "scope": [],
                    "style": {},
                    "text_align": "",
                    "title": "Avg of system.load.1 over *",
                    "viz": "query_value",
                    "yaxis": {}
                  }
                ],
                "id": "1234573",
                "read_only": null,
                "template_variable": [],
                "title": "Alexis's Timeboard 6 Jun 2019 13:41"
              }
            }
          ],
          "address": "module.datadog"
        }
      ]
    }
  }
}

For the truncation issue, here is the first missing embedded resource from the output of terraform plan
Three total graphs were lost. The missing graphs get transformed into the next graph in the series. The missing graphs are missing in all versions of the show:

      ~ graph {
            autoscale               = false
          - events                  = [] -> null
          - group                   = [] -> null
            include_no_metric_hosts = false
            include_ungrouped_hosts = false
          - scope                   = [] -> null
            style                   = {}
          ~ title                   = "Memory Rollup" -> "Available Swap"
            viz                     = "timeseries"
            yaxis                   = {}

          ~ request {
                increase_good = false
              ~ q             = "max:system.mem.slab{$host}, max:system.mem.buffered{$host}, max:system.mem.cached{$host}, max:system.mem.free{$host}, max:system.mem.shared{$host}, max:system.mem.paged{$host}, max:system.mem.page_tables{$host}, max:system.mem.nonpaged{$host}, max:system.mem.committed{$host}" -> "avg:system.swap.free{$host}, avg:system.swap.used{$host}"
                stacked       = false
              ~ style         = {
                  - "palette" = "dog_classic" -> null
                  - "type"    = "solid" -> null
                  - "width"   = "normal" -> null
                }
              - type          = "area" -> null
            }
          - request {
              - increase_good = false -> null
              - q             = "max:system.mem.total{$host}" -> null
              - stacked       = false -> null
              - style         = {
                  - "palette" = "warm"
                  - "type"    = "dashed"
                  - "width"   = "normal"
                } -> null
              - type          = "line" -> null
            }
        }

Debug Output

Not including - no errors or complaints

Expected Behavior

terraform show, terraform show -json, and terraform state show ADDR should all show the same (complete) information

Actual Behavior

terraform show: Strips null and empty string fields, but leaves empty lists and maps
terraform show -json: Shows null, empty string, etc., but no way to differentiate a list from a repeated embedded block
terraform state show ADDR: same as terraform show
All: Occasional complete lost of embedded blocks (consistent between formats)

Steps to Reproduce

Cannot give exact steps for data loss beyond output shown above.

Discovered while using datadog, so steps will specify there

1. Create a datadog timeboard with one graph
2. Create a stub resource block to bind the import to
3. terraform init
4. terraform import datadog_dashboard.stub ID
5. terraform show, terraform show -json, and terraform state show datadog_dashboard.stub

Additional Context

Using terraform import and terraform show to generate configs that can allow management of existing resources

References

Seems similar to: #20028

[apparentlymart]

Hi @lexrj! Thanks for reporting this.

There are a few different parts to this, but I think the main one (which prompted me to label this as "bug") is that terraform state show and terraform show seem to be failing to handle some cases. Both of these would've needed to be updated for Terraform 0.12 due to changes in the underlying state models, but it seems like we didn't get all of the details correct in that process.

There is also the broader issue that it seems unnecessarily confusing to have multiple different ways to print out human-readable state. In future I expect we will consolidate both of those use-cases down into just terraform show, e.g. by adding an option to filter its output, but for this issue I think we should just focus on the aforementioned bug and leave the command separation as-is.

---

The situation with terraform show -json is a different one: that is showing what is essentially a direct JSON transcription of the state as Terraform sees it, without any adjustments made for human consumption. In the human-readable output, the command uses the resource type schema to make some distinctions that the state itself doesn't normally make, but for machine-readable output we expect the consuming program to either disregard those distinctions altogether (for most cases) or, if that external tool is also preparing the data for human consumption, to also use terraform providers schema -json to obtain the schema and thus get the same information that terraform show uses to make rendering decisions.

The schema information is intentionally not included in the terraform show -json output because many use-cases don't need it (if the data is being consumed only for internal processing, not for presentation to a user) and the schema for many providers is quite large. Having it separated means that (in situations where it's appropriate) a consumer can fetch and process the schema just once and cache the result, until the installed providers change and invalidate that cache.

The combination of terraform show -json and terraform providers schema -json should given an external program enough information to reproduce the terraform show output. Unless there's something specific that can't be determined by the combination of those two, I'd say that this is working as intended. If you do find a specific situation where there isn't enough information in the combined output of those two commands, I'd ask that you open a new issue for it so that we can address that separately from the more straightforward issue that the human-readable output is incomplete/incorrect.

Thanks!

[lexrj]

Hi @apparentlymart, thank you for the update! Thank you for the additional context!
I think my biggest concern applies to terraform show -json as well, and is this:

All: Occasional complete lost of embedded blocks (consistent between formats)
The three missing graphs mentioned were missing from all state commands, but were obviously available in state. This led to a situation where doing the following resulted in problems:

1. terraform import
2. terraform show (or variant) into a .tf file (with manipulation for the json variant)
3. terraform plan shows modification

[zachradtka]

I think I have run into a similar issue. terraform show and terraform show -json contain different information. I discovered this problem when I was trying to create an ansible inventory file using terraform show -json and I noticed that no child_modules were listed.

[alisdair]

For the discrepancy between terraform show and terraform show -json with nested modules, see #25494.

[azhurbilo]

We faced with issue (terraform 0.14 version):

terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  # module.main_dms.aws_dms_endpoint.target[0] will be updated in-place
  ~ resource "aws_dms_endpoint" "target" {
      + certificate_arn             = ""
        id                          = "XXX"
      + kms_key_arn                 = ""
        tags                        = {
            "Environment" = "XXX"
            "Team"        = "XXX"
        }
        # (6 unchanged attributes hidden)
      ~ s3_settings {
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ bucket_folder             = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ bucket_name               = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ compression_type          = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ csv_delimiter             = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ csv_row_delimiter         = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ date_partition_enabled    = (sensitive)
          + external_table_definition = (sensitive)
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ service_access_role_arn   = (sensitive)
        }
    }
Plan: 0 to add, 1 to change, 0 to destroy.

terraform show <plan_file>

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  # module.main_dms.aws_dms_endpoint.target[0] will be updated in-place
  ~ resource "aws_dms_endpoint" "target" {
        id                          = "XXX"
        tags                        = {
            "Environment" = "XXX"
            "Team"        = "XXX"
        }
        # (6 unchanged attributes hidden)
        # (1 unchanged block hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

expected behaviour for us that terraform show contains all changes as "terraform plan"