CLI 12.10+ breaks GraphQL Queries

[teckapps]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v20.14.0

Amplify CLI Version

12.9.0

What operating system are you using?

macOS 15.1

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No. The only "manual" change I made was overriding my DynamoDB table names using amplify override api. I guess this could be the reason, because the error appends a role name like 'IAfe82d0-2aokgojt5jbq7dzxznmrvbqgwi-dev' to my TableName, just as it would be if I wouldn't use overrides.

Describe the bug

My GraphQL queries break after updating to CLI 12.10 and later (tried up to 12.13).
If I push my project using an CLI version > 12.9 and attempt to perform list or query operations from the client side, I get errors like this:
"User: arn:aws:sts::[ACCOUNT_ID]:assumed-role/PublicFeedbackItemIAfe82d0-2aokgojt5jbq7dzxznmrvbqgwi-dev/APPSYNC_ASSUME_ROLE is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:eu-central-1:[ACCOUNT_ID]:table/PublicFeedbackItem/index/byStatus because no identity-based policy allows the dynamodb:Query action (Service: DynamoDb, Status Code: 400, Request ID: KL839V8TGVEMTOOJUV75SNSOD3VV4KQNSO5AEMVJF66Q9ASUAAJG)"

Expected behavior

My queries should work without the DynamoDB exception error when using a cli version > 12.9

Reproduction steps

1.) Override table names in a current project
2.) Deploy project
3.) Update to cli >= 12.10

Queries fail
4.) Downgrade CLI to 12.9
5.) Deploy
All works as before

I think I created my project about two years ago and never had problems until now.

Project Identifier

DiagnoseReportUploadError
✖ Sending zip

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[sundersc]

@teckapps How are you overriding table names in your project? Please provide same schema of the model and the code block you have to override a table name.

[teckapps]

My schema is split in multiple files. I have e.g. these two files:

PublicSettingsUtilities.graphql

type PublicSetting @model
@auth(rules: [
{ allow: groups, groups: ["admin", "dbmanager"], operations: [create, read, update, delete] },
{ allow: groups, groups: ["appuser"], operations: [read] },
{ allow: public, operations: [read], provider: apiKey }
]) {
id: ID!
valueText: String
titleTextDE: String
titleTextEN: String
textDE: String
textEN: String
stringArrayValue: [String]
numberValue: Float
versionTag: String
notes: String
sortIndex: Int
}

PublicTutorials.graphql

type PublicTutorialTopic @model
@auth(rules: [
{ allow: groups, groups: ["admin", "dbmanager"], operations: [create, read, update, delete] },
{ allow: public, operations: [read], provider: apiKey }
]) {
id: ID!
language: String! @index(name: "byLanguage", sortKeyFields: ["sortIndex"], queryField: "tutorialTopicsByLanguage")
title: String!
descriptionText: String!
sortIndex: Int!
published: Boolean!
tutorialItems: [PublicTutorialItem] @hasmany(indexName: "byTopicID", fields: ["id"])
}

type PublicTutorialItem @model
@auth(rules: [
{ allow: groups, groups: ["admin", "dbmanager"], operations: [create, read, update, delete] },
{ allow: public, operations: [read], provider: apiKey }
]) {
id: ID!
topicID: ID! @index(name: "byTopicID", sortKeyFields: ["sortIndex"], queryField: "tutorialItemsByTopic")
language: String! @index(name: "byLanguage", sortKeyFields: ["sortIndex"], queryField: "tutorialItemsByLanguage")
staticHotLinkIdentifier: String @index(name: "byStaticHotLinkIdentifier", sortKeyFields: ["language"], queryField: "tutorialItemsByStaticHotLinkIdentifier")
title: String!
descriptionText: String!
sortIndex: Int!
published: Boolean!
mediaResourceItems: [PublicTutorialMediaResource] @hasmany(indexName: "byTutorialItemID", fields: ["id"])
}

type PublicTutorialMediaResource @model
@auth(rules: [
{ allow: groups, groups: ["admin", "dbmanager"], operations: [create, read, update, delete] },
{ allow: public, operations: [read], provider: apiKey }
]) {
id: ID!
tutorialItemID: ID! @index(name: "byTutorialItemID", sortKeyFields: ["sortIndex"], queryField: "tutorialResourcesByTutorialItemID")
title: String!
fileName: String!
resourceType: PublicTutorialMediaResourceType!
sortIndex: Int!
published: Boolean!
}

enum PublicTutorialMediaResourceType {
PDF
VIDEO
}

My overrides looks like this

override.ts

import { AmplifyApiGraphQlResourceStackTemplate } from '@aws-amplify/cli-extensibility-helper';

export function override(resources: AmplifyApiGraphQlResourceStackTemplate) {
// Public Settings Utilities
resources.models["PublicSetting"].modelDDBTable.tableName = "PublicSetting"
// Public Tutorials
resources.models["PublicTutorialTopic"].modelDDBTable.tableName = "PublicTutorialTopic"
resources.models["PublicTutorialItem"].modelDDBTable.tableName = "PublicTutorialItem"
resources.models["PublicTutorialMediaResource"].modelDDBTable.tableName = "PublicTutorialMediaResource"
}

[AnilMaktala]

Hi  @teckapps, we are working on reproducing the issue. Could you please run below command and send us the project identifier.
amplify diagnose --send-report.
please refer here

[teckapps]

Hi @AnilMaktala. I'm getting an DiagnoseReportUploadError every time I try. Is there any other way I can send you the .zip file? Or what might be the cause of the 'DiagnoseReportUploadError'? (I haven't found anything in the docs)

amplify diagnose
Learn more at https://docs.amplify.aws/cli/reference/diagnose/
✔ What would you like to do? · Generate report

✅ Report saved: /var/folders/t_/_3hxz6vx7kv0fscmqsdq017w0000gp/T/WineCloud/report-1731440812145.zip

✔ Send Report (y/N) · yes
⠹ Sending zip
DiagnoseReportUploadError
✖ Sending zip

[AnilMaktala]

Hey @teckapps, Are you on Discord by any chance? If so, could you please share your Discord username with me?

[teckapps]

Hey @AnilMaktala, yes. You can find me as danmedia93

[AnilMaktala]

@teckapps I've sent you a message on Discord. Please share the zip file there.

[palpatim]

Hi @teckapps, I'm looking at this and haven't been able to reproduce using an API created with 12.9.0 and then updated after an upgrade to 12.10.0. Client operations succeed, and the table policy looks as expected.

Can you paste the IAM policy for your PublicFeedbackItem table?

• Login to the AWS console
• Navigate to your AppSync API
• In the left nav, click Data sources
• In the search bar, enter PublicFeedbackItem to find the appropriate DynamoDB table data source
• Click the PublicFeedbackItem Data source name
• Copy the IAM role name
• Navigate to the IAM console and search for the role name you copied above

1. In the role detail page, you should see two inline policies, one that begins with DynamoDBAccess, and one beginning with PublicFeedbackItemIAMRoleDefaultPolicy. Please paste the contents of those policies here.
2. On the tab, Trust relationships, verify that the Trusted entities policy includes the appsync.amazonaws.com service principal:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "appsync.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

[teckapps]

Hi @palpatim, thanks for looking into this. Here are the (account redacted) policy statements. For what I can see, there is a problem with the DynamoBB Policy, as it does not reference the correct (override) table name but the name with the random id and environment at the end. The strange thing is, that currently all works as expected in my dev environment. The default policy references the correct table name. I guess I would have to upgrade the cli and see if the IAMRoleDefautPolicy then changes as well?

Modify permissions in DynamoDBAccess71ABE5AE

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"dynamodb:BatchGetItem",
				"dynamodb:BatchWriteItem",
				"dynamodb:PutItem",
				"dynamodb:DeleteItem",
				"dynamodb:GetItem",
				"dynamodb:Scan",
				"dynamodb:Query",
				"dynamodb:UpdateItem"
			],
			"Resource": [
				"arn:aws:dynamodb:eu-central-1:[ACCOUNT_ID]:table/PublicFeedbackItem-2aokgojt5jbq7dzxznmrvbqgwi-dev",
				"arn:aws:dynamodb:eu-central-1:[ACCOUNT_ID]:table/PublicFeedbackItem-2aokgojt5jbq7dzxznmrvbqgwi-dev/*"
			],
			"Effect": "Allow"
		}
	]
}

Modify permissions in PublicFeedbackItemIAMRoleDefaultPolicyA7CFE05B

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"dynamodb:BatchGetItem",
				"dynamodb:GetRecords",
				"dynamodb:GetShardIterator",
				"dynamodb:Query",
				"dynamodb:GetItem",
				"dynamodb:Scan",
				"dynamodb:ConditionCheckItem",
				"dynamodb:BatchWriteItem",
				"dynamodb:PutItem",
				"dynamodb:UpdateItem",
				"dynamodb:DeleteItem",
				"dynamodb:DescribeTable"
			],
			"Resource": [
				"arn:aws:dynamodb:eu-central-1:[ACCOUNT_ID]:table/PublicFeedbackItem",
				"arn:aws:dynamodb:eu-central-1:[ACCOUNT_ID]:table/PublicFeedbackItem/index/*"
			],
			"Effect": "Allow"
		}
	]
}

Trusted entities

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "appsync.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Let me know how to proceed.

[palpatim]

A couple of things are puzzling me:

1. Timeline

We made a change to the way DDB policies are handled in a86c816, which was released in Amplify CLI 12.12.0 in May 2024. However, that doesn't match up with the behavior you're reporting that any version >= 12.10.0 is causing the issue. I'm investigating this anyway because it's the most likely culprit, but the timeline inconsistency bothers me.

2. Behavior

I saw the same policy structure in my dev environment as well -- the DynamoDBAccess policy resources are named with the default Amplify naming convention, while the PublicFeedbackItemIAMRoleDefaultPolicy has the overridden name.

The thing is, it shouldn't matter! :) When AppSync adopts the role, it would gain the union of those permissions, and as long as no other permission explicitly denied access to dynamodb:Query on the PublicFeedbackItem table or PublicFeedbackItem/index/byStatus index, it should work as expected thanks to the statement in PublicFeedbackItemIAMRoleDefaultPolicy. (You'll notice that policy has a superset of actions contained in the DynamoDBAccess policy, and the resource that the error message refers to is in fact covered by the Resources clause of the PublicFeedbackItemIAMRoleDefaultPolicy.)

Followups

1. Can you confirm that the policy statements you pasted above are from the broken environment?
2. Just to be 100% sure here: Can you confirm that the DynamoDB table is in fact named PublicFeedbackItem per your override, and not the default Amplify naming convention?
3. Can you confirm that the Role ID from which you pasted the policy statements above matches the role ID in your error message above: arn:aws:sts::[ACCOUNT_ID]:assumed-role/PublicFeedbackItemIAfe82d0-2aokgojt5jbq7dzxznmrvbqgwi-dev?
4. Can you confirm that there aren't any other policy statements on that role--only the DynamoDBAccess and PublicFeedbackItemIAMRoleDefaultPolicy? (Basically, as long as AppSync is configured to assume that role, and that role has the inline policy statements (and no overriding policies with a "Deny" Effect), the query should succeed.)
5. Can you confirm that you see the same error message when you try to perform queries from the AppSync Query Editor in the console?
   • If you don't confirm this, can you specify which client you're using to make queries, and how those queries are authorized?

(In case it's not clear, at this point I'm kind of throwing things at the wall to see what sticks, because something isn't quite adding up for me. :( )

[teckapps]

Hi @palpatim. You might be right about the timeline, I might have messed up the versions when testing the issue (sorry for that). I can confirm that everything still works fine with 12.10.0. But it definitely breaks with 12.12.0 and 12.13.1.

First my follow ups you asked for:
1.) Yes, definitely
2.) Yes, 100% (I use two different AWS Accounts with the environments)
3.) Yes, confirmed (more to that later)
4.) Yes, there are now other policy statements except the two I posted previously
5.) Yes, the same errors also occur in the console as well when making queries from an iOS and react web client. It affects queries made using Cognito User Pools and public api key.

Now to my findings after more than 2hours up- & downgrading the cli:

• Deploying the project with 12.12.0 and 12.13.1 removes the permission policies 'PublicFeedbackItemIAMRoleDefaultPolicyA7CFE05B' and 'DynamoDBAccess71ABE5AE'
• Instead, there is only one 'DynamoDBAccess' policy (content is equal to the DynamoDBAccess71ABE5AE policy)
• Searchable queries still work, but all list and queries on secondary indexes fail

So the problem seems, that somehow the '...IAMRoleDefaultPolicy...' is missing.

Deployed and correct working with 12.9.0 & 12.10.0:

Deployed and not working with 12.12.0 & 12.13.1:

[palpatim]

Thanks for the additional context. This is confirming what I'm seeing, which is that the IAMRoleDefaultPolicy handling is different between the impacted versions. I'm digging into it to see if I can trace down where that policy actually gets added -- it's not obvious from the code where that role gets created in the first place. I'll updated when I have more info.

[palpatim]

Commit a86c816 optimized table roles to reduce the number of individual resources created in a stack, and to improve deployment time. One of the changes was to manually craft the table role with the same permissions as would be granted by the CDK default, and add the role to the stack with a withoutPolicyUpdates flag. That flag instructs DynamoDB CDK not to make any further changes to the table role, which is why the IAMRoleDefaultPolicy isn't being created.

Unfortunately, the routine constructs the role resources using the table name components (model name, API ID, and environment name) to craft the table name, without considering table name overrides. By contrast, the default CDK handling attaches the role directly to the table object, so the table name is resolved correctly at deploy time.

We'll need to figure out the right way to handle this. The good news is that we can scope this to just Gen 1 projects -- Gen 2 projects have different ways of handling overrides, and we're currently thinking about the right way of handling them. Even so, we need to make sure that any change we make doesn't regress any existing Gen 1 customers on CLI version >= 12.12.0.

We'll keep working on this and update here when we have more info.

In the meantime, you should be able to workaround by overriding the Amplify-created DynamoDBAccess policy. Tested with CLI@latest (12.13.1):

override.ts

import { AmplifyApiGraphQlResourceStackTemplate } from "@aws-amplify/cli-extensibility-helper";

const AWS_REGION = "YOUR AWS REGION";
const AWS_ACCOUNT_ID = "YOUR AWS ACCOUNT ID";

export function override(resources: AmplifyApiGraphQlResourceStackTemplate) {
  addTableOverrideToResources(resources, "PublicFeedbackItem", "PublicFeedbackItemOverride");
}

/**
 * Updates the resources in place by adding a table name override for the
 * specified model name, and applying appropriate policy resource updates.
 */
const addTableOverrideToResources = (
  resources: AmplifyApiGraphQlResourceStackTemplate,
  modelName: string,
  newTableName: string
): void => {
  resources.models[modelName].modelDDBTable.tableName = newTableName;
  resources.models[modelName].modelIamRole.policies = [{
    policyName: 'DynamoDBAccessOverride',
    policyDocument: {
      Version: "2012-10-17",
      Statement: [
        {
          Action: [
            "dynamodb:BatchGetItem",
            "dynamodb:BatchWriteItem",
            "dynamodb:ConditionCheckItem",
            "dynamodb:DeleteItem",
            "dynamodb:DescribeTable",
            "dynamodb:GetItem",
            "dynamodb:GetRecords",
            "dynamodb:GetShardIterator",
            "dynamodb:PutItem",
            "dynamodb:Query",
            "dynamodb:Scan",
            "dynamodb:UpdateItem",
          ],
          Resource: [
            `arn:aws:dynamodb:${AWS_REGION}:${AWS_ACCOUNT_ID}:table/${newTableName}`,
            `arn:aws:dynamodb:${AWS_REGION}:${AWS_ACCOUNT_ID}:table/${newTableName}/index/*`,
          ],
          Effect: "Allow",
        },
      ],
    }
  }];
};

[teckapps]

Thank you, just tested it and it works for me!

Just to give you heads up: I already played around with some Gen2 Projects in the last couple of months and I came across one issue with the overrides: I probably won't be able to deploy a sandbox in the same AWS Account as my dev environment using my override names.

For new projects this won't be an issue because I wouldn't use the overrides anymore. But at some point I will hopefully be able to migrate my Gen1 project to Gen2. But obviously I will still have my tables with the override names per account. So in order to deploy a sandbox environment in the same AWS account and region as my dev environment, it would be necessary to attach a suffix "-SandboxABCXYZ" to my table names - but only in the sandbox deployment, not in dev and not in prod. My project is in production for 1.5 years now, so I can't just remove the overrides. Sorry for the hassle, had I known that before I wouldn't have used the override names...

[palpatim]

No apologies necessary! We're working on a Gen 2 migration guide and hopefully we'll be able to address the Gen 2 override issue at the same time. In the meantime, I'm leaving this issue open so we can solve this in Ampilfy code rather than making you install the workaround.

[palpatim]

The fix for this (#3075) has been merged to our Gen 1 branch. It will be packaged up in the next version of the Amplify CLI.

[palpatim]

Updated Amplify Data package version has been propagated to CLI configs: aws-amplify/amplify-cli#14048 and is staged for that package's next release.