step-functions: run ecs task when guard duty runtime monitoring is enabled

[florianbepunkt]

Describe the bug

In our org we have Guard Duty runtime monitoring for ECS enabled.

When running a task inside a step function like this:

    const runTask = new sfnt.EcsRunTask(this, "RunTask", {
      cluster,
      launchTarget: new sfnt.EcsFargateLaunchTarget(),
      integrationPattern: sfn.IntegrationPattern.RUN_JOB,
      securityGroups: [sg],
      taskDefinition: job.taskDefinition,
      comment: "Run ECS Task",
    });

The task itself is run successfully, but the state still fails due to insufficient permissions regarding GuardDuty.

    {
      "ContainerArn": "*****",
      "Cpu": "null",
      "GpuIds": [],
      "LastStatus": "STOPPED",
      "ManagedAgents": [],
      "Name": "aws-guardduty-agent-mfjGLB",
      "NetworkBindings": [],
      "NetworkInterfaces": [
        {
          "AttachmentId": "*****",
          "PrivateIpv4Address": "10.0.0.172"
        }
      ],
      "Reason": "CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref 323658145986.dkr.ecr.eu-central-1.amazonaws.com/aws-guardduty-agent-fargate:v1.5.0-Fg_arm64: unexpected status from HEAD request to https://323658145986.dkr.ec",
      "RuntimeId": "adab51249cea4b5ea2d99cede4bb257e-2987719819",
      "TaskArn": "*****"
    }

Please note that the guard duty agent is not configured in our ECS task, but it is added by AWS if guard duty is enabled.

We can work around this by adding wildcard permissions like this, but it's less than ideal

taskDefinition.addToExecutionRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
        ],
        resources: ["*"],
      }),
    );

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

CDK should add the necessary permissions.

Current Behavior

Permissions are insufficient.

Reproduction Steps

It is not possible to create a self-contained example, as the error originates from account / org specific configuration.

In our org we have a landing zone with a dedicated security account and an developer account where the ECS task and the calling step function is placed.

Steps:

1. Enable guard duty runtime monitoring
2. Wait until changed guard duty setting take effect.
3. Create an ECS task (does not matter, what it does, but should be a one-off script that executes and exits)
4. Use the cdk step function runTask task.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.173.3

Framework Version

No response

Node.js Version

22

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

When GuardDuty Runtime Monitoring is enabled for ECS Fargate tasks: [2]

1. The GuardDuty agent is automatically added as a sidecar container to new Fargate tasks
2. For each container in your task, GuardDuty attaches its monitoring container
3. The monitoring happens automatically without any changes needed to your Step Functions workflow

Looking at your provided error

323658145986.dkr.ecr.eu-central-1.amazonaws.com/aws-guardduty-agent-fargate:v1.5.0-Fg_arm64

Looks like ecs task can't pull the guardduty sidebar container due to "failed to resolve"?

Unfortunately we don't have immediately working sample and the document does not provide relevant details.

As you mentioned this works for you.

taskDefinition.addToExecutionRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
        ],
        resources: ["*"],
      }),
    );

I am guessing you probably are missing ECR permissions for this sidebar container image. I am not sure what is the best recommended permissions in your use case as the document is not having this info but I'll reach out internally and see if we can update the public documents. Some options you can consider here:

1. check your cloudtrail and see what API calls are failing and try to scope your "resources" attribute.
2. If you have AWS premium support, you can reach out to the support to help you figure out the correct IAM policies for your use case.

[pahud]

internal: V1642261665

[pahud]

Hi @florianbepunkt

According to the doc:
https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs

You will need the following in your IAM policy.

      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",

FYR

[florianbepunkt]

Hi @pahud Thank you very much for the provided help. I had searched the docs but was unable to find this. Maybe this should be included / abstracted away by the construct. Like a flag whether the task is monitored by GuardDuty or not.

I added the permissions and in my case also scoped them to the guard duty ECR repo:

    this.taskDefinition.addToExecutionRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        // @see https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs
        actions: [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
        ],
        // @see https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-ecr-repository-gdu-agent.html
        resources: ["arn:aws:ecr:eu-central-1:323658145986:repository/aws-guardduty-agent-fargate"],
      }),
    );