Cloud front: running lambda@edge sends bucket to wrong s3 region

[anthonyLock]

Describe the bug

I have a bucket in eu-west-2 and have recently created a cloudfront distribution and lambda@edge to serve the content. The lambda is a viewer request. The lambda@edge and cloudfront are in us-east-1.

For an authentication step I have added in a lambda following the following blog post https://aws.amazon.com/blogs/networking-and-content-delivery/authorizationedge-using-cookies-protect-your-amazon-cloudfront-content-from-being-downloaded-by-unauthenticated-users/

The Bucket is defined in a different CDK stack.

I have got everything working great without the lambda and cloudfront delivers the content as expected. However as soon as I add in the lambda I am getting the following message

<Error>
    <Code>PermanentRedirect</Code>
    <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
    <Endpoint>MY_BUCKET_NAME.s3.eu-west-2.amazonaws.com</Endpoint>
    <Bucket>MY_BUCKET_NAME</Bucket>
    <RequestId>WCQF2BDTAT7CYRYZ</RequestId>
  <HostId>aY57IzEVSdHy6qKTCW/v40JPyqEFAvOd3CNXGLZsr2KLxatJ8/6pAS5dBfEpr+3aVmtKDNWcCU2/hl0+m1ezMLzBdhElES0x</HostId>
</Error>

I have followed the advice in this issue #9556 and still having the same problem.

My lambda code is

export const lambdaHandler: CloudFrontRequestHandler = async (input) => {
    const event = input.Records[0].cf.request;

    // Do custom auth stuff 

    return event 
}

My CDK looks like

I also have all the permissions and bucket policies but have not added to above as the cloudfront is working without the EdgeLambdas so I am pretty sure it is not due to that.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Expect to serve the content both with and without the lambda running

Current Behavior

<Error>
    <Code>PermanentRedirect</Code>
    <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
    <Endpoint>MY_BUCKET_NAME.s3.eu-west-2.amazonaws.com</Endpoint>
    <Bucket>MY_BUCKET_NAME</Bucket>
    <RequestId>WCQF2BDTAT7CYRYZ</RequestId>
  <HostId>aY57IzEVSdHy6qKTCW/v40JPyqEFAvOd3CNXGLZsr2KLxatJ8/6pAS5dBfEpr+3aVmtKDNWcCU2/hl0+m1ezMLzBdhElES0x</HostId>
</Error>

Reproduction Steps

authLambda := awslambdanodejs.NewNodejsFunction(
		stack,
		jsii.String("cloud-front-auth-lambda"),
		&awslambdanodejs.NodejsFunctionProps{
			FunctionName:     jsii.String("cloud-front-auth-lambda"),
			Entry:            utils.SprintfPtr("%s/../../lambda/authentication-cloud-front/src/lambda.ts", dirname),
			ProjectRoot:      utils.SprintfPtr("%s/../../lambda/authentication-cloud-front", dirname),
			DepsLockFilePath: utils.SprintfPtr("%s/../../lambda/authentication-cloud-front/package-lock.json", dirname),
			Handler:          jsii.String("lambdaHandler"),
			Runtime:          awslambda.Runtime_NODEJS_18_X(),
			Architecture:     awslambda.Architecture_X86_64(),
			MemorySize:       jsii.Number(128),
			Timeout:          awscdk.Duration_Millis(jsii.Number(4000)),
			Tracing:          awslambda.Tracing_ACTIVE,
			Role:             authRole,
		})


importedBucket := awss3.Bucket_FromBucketAttributes(stack, jsii.String("cloud-front-imported-bucker"), &awss3.BucketAttributes{
		BucketName: jsii.String("my bucket name"),,
		Region:     jsii.String("eu-west-2"),
	})

cf := awscloudfront.NewDistribution(stack, jsii.String("cloud-front-distro"), &awscloudfront.DistributionProps{
		DefaultBehavior: &awscloudfront.BehaviorOptions{
			Origin: awscloudfrontorigins.S3BucketOrigin_WithOriginAccessControl(importedBucket, awscloudfrontorigins.S3BucketOriginWithOACProps{}),
			EdgeLambdas: &[]*awscloudfront.EdgeLambda{
				{
					EventType:       awscloudfront.LambdaEdgeEventType_VIEWER_REQUEST,
					FunctionVersion: authLambda.CurrentVersion(),
					IncludeBody:     jsii.Bool(true),
				},
			},
		},
	})

I have also tried changing Origin to a group

	Origin: awscloudfrontorigins.NewOriginGroup(&awscloudfrontorigins.OriginGroupProps{
				PrimaryOrigin:  awscloudfrontorigins.S3BucketOrigin_WithOriginAccessControl(importedBucket, &awscloudfrontorigins.S3BucketOriginWithOACProps{}),
				FallbackOrigin: awscloudfrontorigins.S3BucketOrigin_WithOriginAccessControl(importedBucket, &awscloudfrontorigins.S3BucketOriginWithOACProps{}),
			}),

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.160.0 (build 7a8ae02)

Framework Version

No response

Node.js Version

v18.18.2

OS

ubunbu on wsl

Language

Go

Language Version

1.23

Other information

No response

[ashishdhingra]

@anthonyLock Good afternoon. Thanks for reporting the issue. Looks like you are using the custom handler for your Lambda function. Please confirm the following:

• From where the error The bucket you are attempting to access must be addressed using the specified endpoint... is thrown? Is it your Lambda function?
• So, is Lambda function making AWS SDK S3 API calls to the configured S3 bucket?
  • Are your explicitly specifying the region while making AWS SDK call?
  • If not, have you configured AWS region to be used by AWS SDK S3 call using AWS_REGION environment variable (refer Defined runtime environment variables). If not, you could try setting this environment variable using environment property of NodejsFunctionProps as specified at NodejsFunction.

Thanks,
Ashish

[anthonyLock]

@ashishdhingra

The error message is shown when I do a request to the cloud front URL after everything is deployed. It is in the response.

I am using the typescript CloudFrontRequestHandler type to define the Handler type. This comes from import { CloudFrontRequestHandler } from "aws-lambda"; as suggested in here

The lambda is not making any SDK request but is checking a cookie that is a JWT from cognito, using CognitoJwtVerifier from import { CognitoJwtVerifier } from "aws-jwt-verify"; The lambda itself is running ok and returning the event as suggested in the blog post https://aws.amazon.com/blogs/networking-and-content-delivery/authorizationedge-using-cookies-protect-your-amazon-cloudfront-content-from-being-downloaded-by-unauthenticated-users/

I am getting cloudwatch logs saying the execution time with it running successfully.

After some further digging I tried the following in my typescript code changing returning the event to using the callback. This worked

export const lambdaHandler: CloudFrontRequestHandler = async (
  input,
  _context,
  callback
) => {
   const event = input.Records[0].cf.request;

    // Do custom auth stuff 

   callback(null, event);
   return; 
}

In all the exmples on this page https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html they use the callback function given. However according to https://docs.aws.amazon.com/lambda/latest/dg/typescript-handler.html#typescript-handler-callback it says We recommend that you use [async/await](https://docs.aws.amazon.com/lambda/latest/dg/typescript-handler.html#async-typescript) to declare the function handler instead of using callbacks

Knowing this I may have created a issue in the wrong place. Please let me know if it is more appropriate to recreate it elsewhere,

[ashishdhingra]

@anthonyLock Good morning. I'm assuming you took the Lambda handler code from https://aws.amazon.com/blogs/networking-and-content-delivery/using-amazon-cloudfront-and-amazon-s3-to-build-multi-region-active-active-geo-proximity-applications/ where it is setting S3 bucket region as below:

    # Update origin request object
    request['origin']['s3']['domainName'] = domain_name
    request['origin']['s3']['region'] = lambda_region
    request['headers']['host'] = [{'key': 'host', 'value': domain_name}]

This appears to be setting bucket region to lambda region. And as you mentioned, Lambda and S3 bucket are in different regions. So it gives the mentioned error since S3 bucket is being accessed from a different region (Lambda region).

I'm unsure if using callback as opposed to async/await should have any change in behavior.

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.