Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(stepfunctions-tasks): incorrect S3 permissions for AthenaStartQueryExecution #11203

Merged
merged 8 commits into from
Nov 3, 2020
Merged

fix(stepfunctions-tasks): incorrect S3 permissions for AthenaStartQueryExecution #11203

merged 8 commits into from
Nov 3, 2020

Conversation

Sumeet-Badyal
Copy link
Contributor

@Sumeet-Badyal Sumeet-Badyal commented Oct 29, 2020

The changes made by #11045 grant S3 scoped permissions if the optional parameter output location is passed. The output location is not parsed correctly to be attached as a resource. When the output location is correctly parsed, state machines with a valid definition and a valid S3 bucket still fail due to an Unable to verify/create output bucket error. The exact same state machine and S3 bucket pass with wildcard permissions as such the resource for Start Query Execution must be changed to *.

BREAKING CHANGE: type of outputLocation in the experimental Athena StartQueryExecution has been changed to s3.Location from string


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Oct 29, 2020

@Sumeet-Badyal Sumeet-Badyal changed the title fix(stepfunctions-tasks): Athena StartQueryExecution requires wildcard S3 permissions perf(stepfunctions-tasks): Athena StartQueryExecution requires wildcard S3 permissions Oct 29, 2020
@Sumeet-Badyal Sumeet-Badyal changed the title perf(stepfunctions-tasks): Athena StartQueryExecution requires wildcard S3 permissions fix(stepfunctions-tasks):StartQueryExecution requires wildcard S3 permissions Oct 29, 2020
@Sumeet-Badyal Sumeet-Badyal changed the title fix(stepfunctions-tasks):StartQueryExecution requires wildcard S3 permissions fix(stepfunctions-tasks): StartQueryExecution requires wildcard S3 permissions Oct 29, 2020
@shivlaks shivlaks changed the title fix(stepfunctions-tasks): StartQueryExecution requires wildcard S3 permissions fix(stepfunctions-tasks): incorrect S3 permissions for AthenaStartQueryExecution Oct 29, 2020
Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sumeet-Badyal thanks for submitting the fix - I updated the commit message to indicate the error as per our CONTRIBUTING guide.

the rationale for that is because this will show up in the CHANGELOG under bug fixes and it reads better when it describes the problem that was fixed.

@mergify mergify bot dismissed shivlaks’s stale review October 29, 2020 19:31

Pull request has been modified.

Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sumeet-Badyal I updated the commit description to include the breaking change entry so that it shows up in the changelog.

@mergify mergify bot dismissed shivlaks’s stale review November 2, 2020 23:04

Pull request has been modified.

@mergify
Copy link
Contributor

mergify bot commented Nov 3, 2020

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 3e1dbc7
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Nov 3, 2020

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit b35c423 into aws:master Nov 3, 2020
's3:ListBucketMultipartUploads',
's3:ListMultipartUploadParts',
's3:PutObject'],
resources: [this.props.resultConfiguration?.outputLocation ?? '*'], // Need S3 location where data is stored https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html
resources: [this.props.resultConfiguration?.outputLocation?.bucketName ? `arn:aws:s3:::${this.props.resultConfiguration?.outputLocation?.bucketName}/${this.props.resultConfiguration?.outputLocation?.objectKey}/*` : '*'], // Need S3 location where data is stored or Athena throws an Unable to verify/create output bucket https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sumeet-Badyal Thank you for your PR!
I've tried the changes in this PR, but since objectKey is required, it doesn't seem to work when specifying into directly under the bucket.
Another problem might be that when using CREATE TABLE AS, files are created under tables/ directory of the bucket specified with outputLocation, so we have to specify arn:aws:s3:::my-bucket/* (not including any object keys except wildcard) instead of arn:aws:s3:::my-bucket/foo/*.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants