[aws_emr_cluster] Unclear if --enable-debugging is actually implemented

[oonisim]

---

Please explain why EMR "step" can be the same with "--enable-debugging" option for EMR cluster.

I suppose there should be an option enable_debugging in addition to log_uri , and adding a step should not be the same as enabling debugging.

As in Enable the Debugging Tool, enabling debugging is an option.

To enable the debugging tool using the Amazon EMR console

Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.
1.Choose Create cluster.
2.Choose Go to advanced options.
3.In the Cluster Configuration section, in the Logging field, choose Enabled. 
You cannot enable debugging without enabling logging.
4. In the Log folder S3 location field, type an Amazon S3 path to store your logs.
5. In the Debugging field, choose Enabled. <-----

Or with CLI.

aws emr create-cluster --name "Test cluster" \\
  --release-label emr-5.25.0 \
  --log-uri s3://mybucket/logs/ \
  --enable-debugging \    <----- Optional and independent from using step or not
...

Hence, enabling debugging should be independent from using a step. However, it looks currently using a step is the same with enabling debug.

Enable Debug Logging
Debug logging in EMR is implemented as a step. <--- Why debugging is implemented as EMR step?

resource "aws_emr_cluster" "example" {
  # ... other configuration ...

  step {
    action_on_failure = "TERMINATE_CLUSTER"
    name              = "Setup Hadoop Debugging"

    hadoop_jar_step {
      jar  = "command-runner.jar"
      args = ["state-pusher-script"]
    }
  }

  # Optional: ignore outside changes to running cluster steps
  lifecycle {
    ignore_changes = ["step"]
  }
}

If this is the case, any step placed in aws_emr_cluster would be enabling debugging, if so, we cannot implement a step without enabling debugging?

PR

Add support for enabling cluster debugging to EMR #1866
resource/aws_emr_cluster: Add step support #3673

[bflad]

Hi @oonisim 👋

Why debugging is implemented as EMR step?

The EMR API does not have any special debugging parameter available: https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html#API_RunJobFlow_RequestSyntax

And the AWS CLI implementation of --enable-debugging is performing an addition of a step into the EMR Job Flow:

https://github.com/aws/aws-cli/blob/79dff4019138a58ac949c7eb2dead00dd1e30c42/awscli/customizations/emr/createcluster.py#L254-L259

https://github.com/aws/aws-cli/blob/79dff4019138a58ac949c7eb2dead00dd1e30c42/awscli/customizations/emr/createcluster.py#L462-L476

The AWS Console is likely hiding this implementation detail by calling AddJobFlowSteps itself (or via a private API), but you will need to ask AWS Support for a definitive answer if that is the case.

The Terraform documentation is reflecting the underlying EMR implementation with the step since Terraform expects to fully manage the entire steps attribute when its defined and it would be complex to introduce difference logic for a separate boolean flag with the Terraform resource.

If this is the case, any step placed in aws_emr_cluster would be enabling debugging, if so, we cannot implement a step without enabling debugging?

Only the documented step will enable debugging, if present in your Terraform configuration. If you would like to not enable debug logging, you can omit the configuration. Otherwise, to enable debug logging you will need to include this additional job flow step where it makes sense in relation to other steps.

---

If you're looking for general assistance, please note that we use GitHub issues in this repository for tracking bugs and enhancements with the Terraform AWS Provider codebase rather than for questions. While we may be able to help with certain simple problems here it's generally better to use one of the community forums where there are far more people ready to help, whereas the GitHub issues here are generally monitored only by a few maintainers and dedicated community members interested in code development of the Terraform AWS Provider itself.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!