[aws-elasticsearch] Elastic search domain resource lacks permissions. #11412
Comments
laimonassutkus
added
bug
|
Anyone? |
laimonassutkus
changed the title
github-actions
bot
added
the
@aws-cdk/aws-elasticsearch
|
Same issue here. It worked before with the same configuration |
|
I was using a custom KMS key. Just created the same domain but with the default key instead and that worked. |
|
Cmon @iliapolo, help us :)) |
|
@Ruben-E using default kms keys is a bad practice. |
|
Well, For us it's an acceptable workaround for now. At least able to proceed with testing. |
|
With a custom master key you can control who has access to it and who does not. Wit default one you can't do that. Also, with custom master keys you can enable key rotation. P.S. @iliapolo, where are you? |
|
@laimonassutkus apologies for the delayed response. I will have a look at this today. |
|
Thanks! Appreciated. |
|
@laimonassutkus @Ruben-E Can you please share the code you are using exactly? |
|
@iliapolo A piece of the code: |
|
@Ruben-E I'm not able to reproduce this..we indeed didn't change anything in module for a while, especially not in the L1 resources. This is the code i'm using: const key = new kms.Key(this, 'DomainKey', { enabled: true })
new elastic.CfnDomain(this, 'Domain', {
elasticsearchVersion: '7.7',
encryptionAtRestOptions: {
kmsKeyId: key.keyId,
enabled: true,
},
ebsOptions: {
ebsEnabled: true,
volumeSize: 80
}
})I wonder if just a problem with the role you are using when deploying. Im using an administrator role. P.S just making sure you are aware that we have an L2 construct now: |
iliapolo
added
the
response-requested
|
Thanks for checking it. 10 days ago I ran into this issue, maybe its working again now. The code did work before, maybe it was a temporary hiccup? I'll test it again later today. I shared just a piece of our code, but replacing that custom kms key with the default key made it work again, so I think the example you shared is comparable with the situation and should fail as well. I know about the L2 construct, thanks for the reminder :). But I doesn't / didn't support audit logs which was recently added. Switched to the cfn resource to support that. I have it on my list to create a PR for this ;) |
|
@Ruben-E cool 👍 Regarding the error though, could it be that the role you are running doesn't have the necessary KMS permissions? |
|
Here's a reproduction case in Python from aws_cdk.core import Stack, App, RemovalPolicy
from aws_cdk.aws_kms import Key
from aws_cdk.aws_iam import PolicyDocument, PolicyStatement, Effect, AccountRootPrincipal
from aws_cdk.aws_elasticsearch import (
AdvancedSecurityOptions,
CapacityConfig,
Domain,
EbsOptions,
ElasticsearchVersion,
EncryptionAtRestOptions,
TLSSecurityPolicy,
ZoneAwarenessConfig,
)
from aws_cdk.aws_ec2 import EbsDeviceVolumeType
class Infrastructure(Stack):
def __init__(self, scope: Stack):
super().__init__(
scope=scope,
id="EsReproStack",
stack_name="EsReproStack",
)
kms_key = Key(
scope=self,
id="EsReproCmk",
alias="EsReproCmk",
enabled=True,
enable_key_rotation=True,
policy=PolicyDocument(
statements=[
PolicyStatement(
actions=[
'kms:*',
],
effect=Effect.ALLOW,
resources=['*'],
principals=[AccountRootPrincipal()]
),
]
),
removal_policy=RemovalPolicy.DESTROY,
)
domain = Domain(
scope=self,
id="EsReproDomain",
version=ElasticsearchVersion.V7_7,
access_policies=[
PolicyStatement(
actions=["es:*"],
effect=Effect.ALLOW,
resources=["*"],
principals=[AccountRootPrincipal()],
),
],
capacity=CapacityConfig(
data_node_instance_type="t3.small.elasticsearch",
data_nodes=1,
master_nodes=None,
),
ebs=EbsOptions(enabled=True, volume_size=10, volume_type=EbsDeviceVolumeType.GP2),
enforce_https=True,
encryption_at_rest=EncryptionAtRestOptions(
enabled=True,
kms_key=kms_key,
),
node_to_node_encryption=True,
tls_security_policy=TLSSecurityPolicy.TLS_1_2,
use_unsigned_basic_auth=False,
zone_awareness=ZoneAwarenessConfig(enabled=False),
)
app = App()
Infrastructure(app)
app.synth()Relevant parts of output from |
|
@iliapolo Just tested, still have the issue. I think the issue occurs when combining the cluster with an access policy. Because the creation of the cluster doesn't fail, but the creation of the custom resource to apply the access policy fails. The L2 construct creates a custom resources to set the access policy when the cluster is created: elasticsearch-access-policy.ts |
|
@ignaloidas I see you also defined an access policy and it also fails on that. This error is exactly the same as I have |
|
@Ruben-E @ignaloidas thanks for the info. I'll give that a try. |
|
@ignaloidas Are you able to consistently reproduce with the python example? Could you share which region are you deploying to? |
|
@iliapolo Yes, it is consistent, I'm deploying to |
|
For me its also consistent. Tested a couple of times today. Also eu-west-1 |
|
@Ruben-E do you have a minimal repro as well? the more the merrier |
|
@Ruben-E i mean something more detailed than what you previously posted (couldn't repro with that) |
|
@ignaloidas Are you sure the repro you posted is accurate? I'm hitting this error:
because you specify 1 data node and using 2 availability zones (the default). Not that it matters for our repro, but i just want to make sure its all accurate. Thanks |
|
I understand. I'll look at that tomorrow. |
Huh, no, I do specify to not have any availability zone awareness with |
|
@ignaloidas oh right, apologies, copy paste error 👍 |
|
It seems that in some, still unknown, conditions, the I'm reluctant to add this by default while its still unclear, but to get around this issue you can add the necessary permissions like so: import * as cr from '@aws-cdk/custom-resources';
const key = new kms.Key(...);
const domain = new elastic.Domain(...);
const policyHandler = domain.node.tryFindChild('ESAccessPolicy') as cr.AwsCustomResource;
policyHandler.grantPrincipal.addToPrincipalPolicy(new iam.PolicyStatement({
actions: ['kms:DescribeKey'],
resources: [key.keyArn],
effect: iam.Effect.ALLOW,
}))@Ruben-E this still doesn't explain why the same problem is happening in your case where only the L1 is being used. |
iliapolo
added
investigating
|
@iliapolo Thanks for the workaround, I'm going to give it a try. Although I use the L1, I still use the ElasticSearchAccessPolicy custom resource (because ARN is needed for policies which is only available after deploy). Since the problem is not in the L1 / L2, but in the custom resource, I'm also affected. This code also reproduces the issue and gives an idea how I set it up: class Elasticsearch extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const key = new Key(this, 'Key', {enabled: true});
const domain = new CfnDomain(this, 'Domain', {
domainName: 'test',
elasticsearchVersion: ElasticsearchVersion.V7_7.version,
elasticsearchClusterConfig: {
instanceCount: 1,
instanceType: 't3.small.elasticsearch'
},
ebsOptions: {ebsEnabled: true, volumeSize: 80},
nodeToNodeEncryptionOptions: {enabled: true},
encryptionAtRestOptions: {enabled: true, kmsKeyId: key.keyId},
domainEndpointOptions: {enforceHttps: true, tlsSecurityPolicy: TLSSecurityPolicy.TLS_1_2},
})
const accessPolicy = new ElasticsearchAccessPolicy(this, 'AccessPolicy', {
domainName: domain.domainName!!,
domainArn: domain.attrArn,
accessPolicies: [allowHttp(domain.attrArn)]
});
accessPolicy.node.addDependency(domain);
function allowHttp(clusterArn: string) {
return new PolicyStatement({
effect: Effect.ALLOW,
actions: ['es:ESHttp*'],
principals: [new AnyPrincipal()],
resources: [Lazy.stringValue({produce: () => `${clusterArn}/*`})],
});
}
}
}
const app = new App();
new Elasticsearch(app, 'repro', stackProperties)Output: |
|
Can confirm, the workaround works |
Awesome, i'm going to mark this as a bug and keep investigating. Thanks |
iliapolo
added
effort/small
|
@Ruben-E @ignaloidas @laimonassutkus managed to reproduce, thanks for all the cooperation :) |
iliapolo
removed
the
investigating
|
…om kms key fails to deploy (aws#11699) The problem was that we were missing the necessary kms permissions for the custom resource that applies the access policies. Fixes aws#11412 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Elastic search domain resource lacks permissions.
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<ID>:assumed-role/Testing-Mi-Main-Stack-<ID>-<ID>/Testing-Mi-Main-Stack-<ID>-<ID> is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-central-1:<ID>:key/<ID> (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: <ID>; Proxy: null)Reproduction Steps
Create elastic search Domain. (from aws_cdk.aws_elasticsearch import Domain).
What did you expect to happen?
Domain created.
What actually happened?
CloudFormation error complaining about permissions:
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<ID>:assumed-role/Testing-Mi-Main-Stack-<ID>-<ID>/Testing-Mi-Main-Stack-<ID>-<ID> is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-central-1:<ID>:key/<ID> (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: <ID>; Proxy: null)Environment
Other
The bug seems to be new. Did not experience in previous versions.
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: