Print retryLogHook messages to stderr

[ben-z]

Terraform Version

Terraform v1.8.2
on linux_amd64

Terraform Configuration Files

terraform {
  cloud {
    organization = "..."

    workspaces {
      name = "..."
    }
  }
}

// ...

Debug Output

Not relevant

Expected Behavior

terraform -output json should output only JSON.

Actual Behavior

The output is not pure json. It contains some error messages from intermittent (retriable) errors.

> terraform -output json
There was an error connecting to HCP Terraform. Please do not exit Terraform to prevent data loss!
Trying to restore the connection...

Still trying to restore the connection... (3s elapsed)
{}

These are the culprit messages:

terraform/internal/command/views/cloud.go

Lines 51 to 56 in dc3a040

	const initialRetryError = `
	There was an error connecting to HCP Terraform. Please do not exit
	Terraform to prevent data loss! Trying to restore the connection...
	`
	const repeatedRetryError = "Still trying to restore the connection... (%s elapsed)"

This makes it nontrivial for downstream applications to parse the output.

Steps to Reproduce

1. terraform -output json
2. When the connection to the backend is suboptimal, additional messages show up.

Additional Context

No response

References

No response

[ben-z]

I'm currently using a quick and dirty workaround: next(extract_json_objects(stdout))

def extract_json_objects(text, decoder=JSONDecoder()):
    """
    Find JSON objects in text, and yield the decoded JSON data

    Does not attempt to look for JSON arrays, text, or other JSON types outside
    of a parent JSON object.

    Derived from https://stackoverflow.com/a/54235803

    This is useful for handling JSON objects embedded in a string, such as
    when this bug pollutes the output:
    https://github.com/hashicorp/terraform/issues/35159
    """
    pos = 0
    while True:
        match = text.find("{", pos)
        if match == -1:
            break
        try:
            result, index = decoder.raw_decode(text[match:])
            yield result
            pos = match + index
        except ValueError:
            pos = match + 1

[gawbul]

Any updates on this at all. We are seeing the issue with our Terraform Cloud workflows as well.
It doesn't cause failure in our Terraform Apply pipelines, but the final output steps sometimes fail and the deployments are therefore reported as failures in Jira.

[bschaatsbergen]

Hey @ben-z, thank you for reporting this.

It seems like the initial and repeated retry error log messages are not rendered through a view yet, which causes them to default to a human-readable output. Ideally, there should always be a HumanView and JsonView implementation for such operations in the views package.

I'll be picking up this issue and will share more information there as soon as it becomes available.

Thanks again!