Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expose AWS IAM missing param in Hashicorp secret #38536

Merged
merged 13 commits into from
Apr 25, 2024
Merged

Expose AWS IAM missing param in Hashicorp secret #38536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
31990d9
Expose aws iam missing param in hashicorp secret
pankajastro Mar 27, 2024
7e242ce
Fix
pankajastro Mar 28, 2024
6378557
Fix
pankajastro Mar 28, 2024
0e82e20
Fix
pankajastro Mar 28, 2024
4d5b0e8
Use AWS connection
pankajastro Apr 4, 2024
552dc6e
Adjust tests
pankajastro Apr 4, 2024
ea0f2a9
Apply review
pankajastro Apr 12, 2024
536d0fe
Fix auth
pankajastro Apr 20, 2024
67c384b
Fix test
pankajastro Apr 20, 2024
ad1c59d
Add docs
pankajastro Apr 21, 2024
cb82eed
Fix docs
pankajastro Apr 21, 2024
d7fb3bf
Add session auth
pankajastro Apr 22, 2024
cf70280
Fix docs
pankajastro Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 32 additions & 8 deletions airflow/providers/hashicorp/_internal_client/vault_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ class _VaultClient(LoggingMixin):
:param key_id: Key ID for Authentication (for ``aws_iam`` and ''azure`` auth_type).
:param secret_id: Secret ID for Authentication (for ``approle``, ``aws_iam`` and ``azure`` auth_types).
:param role_id: Role ID for Authentication (for ``approle``, ``aws_iam`` auth_types).
:param role_arn: AWS arn role (for ``aws_iam`` auth_type)
Lee-W marked this conversation as resolved.
Show resolved Hide resolved
:param kubernetes_role: Role for Authentication (for ``kubernetes`` auth_type).
:param kubernetes_jwt_path: Path for kubernetes jwt token (for ``kubernetes`` auth_type, default:
``/var/run/secrets/kubernetes.io/serviceaccount/token``).
Expand Down Expand Up @@ -103,6 +104,7 @@ def __init__(
password: str | None = None,
key_id: str | None = None,
secret_id: str | None = None,
role_arn: str | None = None,
role_id: str | None = None,
kubernetes_role: str | None = None,
kubernetes_jwt_path: str | None = "/var/run/secrets/kubernetes.io/serviceaccount/token",
Expand Down Expand Up @@ -161,6 +163,7 @@ def __init__(
self.key_id = key_id
self.secret_id = secret_id
self.role_id = role_id
self.role_arn = role_arn
self.kubernetes_role = kubernetes_role
self.kubernetes_jwt_path = kubernetes_jwt_path
self.gcp_key_path = gcp_key_path
Expand Down Expand Up @@ -318,15 +321,36 @@ def _auth_azure(self, _client: hvac.Client) -> None:
)

def _auth_aws_iam(self, _client: hvac.Client) -> None:
if self.auth_mount_point:
_client.auth.aws.iam_login(
access_key=self.key_id,
secret_key=self.secret_id,
role=self.role_id,
mount_point=self.auth_mount_point,
)
if self.key_id and self.secret_id:
auth_args = {
"access_key": self.key_id,
"secret_key": self.secret_id,
"role": self.role_id,
}
else:
_client.auth.aws.iam_login(access_key=self.key_id, secret_key=self.secret_id, role=self.role_id)
import boto3

if self.role_arn:
sts_client = boto3.client("sts")
credentials = sts_client.assume_role(RoleArn=self.role_arn, RoleSessionName="airflow")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pankajastro - we should consider exposing assume_role_kwargs as a backend_kwarg.

This is currently available as an AWS Connection Extra parameter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please share an example of AIRFLOW__SECRETS__BACKEND_KWARGS?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see the Request Syntax section for example kwargs.

I imagine it would be a dictionary that you would pass to AIRFLOW__SECRETS__BACKEND_KWARGS, like one does today in the AWS connection.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdyt #39279

AIRFLOW__SECRETS__BACKEND_KWARGS='{"kv_engine_version": 1, "mount_point": "kv", "variables_path": "airflow", "url": "http://127.0.0.0:8200/", "auth_type": "aws_iam", "assume_role_kwargs": {"RoleArn": "arn:aws:iam::1234567890000:role/hashicorp-aws-iam", "RoleSessionName": "airflow", "DurationSeconds": 900}}'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that looks good (assuming that it works). You wouldn't need a role_arn arg in this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's included in the assume_role_kwargs parameter. I've shared an example for reference ^^

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I am saying you no longer need a dedicated role_arn backend kwarg. It would be a part of assume_role_kwargs.

auth_args = {
"access_key": credentials["Credentials"]["AccessKeyId"],
"secret_key": credentials["Credentials"]["SecretAccessKey"],
"session_token": credentials["Credentials"]["SessionToken"],
}
else:
session = boto3.Session()
credentials = session.get_credentials()
auth_args = {
"access_key": credentials.access_key,
"secret_key": credentials.secret_key,
"session_token": credentials.token,
}

if self.auth_mount_point:
auth_args["mount_point"] = self.auth_mount_point

_client.auth.aws.iam_login(**auth_args)

def _auth_approle(self, _client: hvac.Client) -> None:
if self.auth_mount_point:
Expand Down
6 changes: 6 additions & 0 deletions airflow/providers/hashicorp/provider.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,9 @@ connection-types:

secrets-backends:
- airflow.providers.hashicorp.secrets.vault.VaultBackend

additional-extras:
- name: boto3
dependencies:
# Require for AWS assume role authentication
- boto3>=1.33.0
3 changes: 3 additions & 0 deletions airflow/providers/hashicorp/secrets/vault.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ class VaultBackend(BaseSecretsBackend, LoggingMixin):
:param key_id: Key ID for Authentication (for ``aws_iam`` and ''azure`` auth_type).
:param secret_id: Secret ID for Authentication (for ``approle``, ``aws_iam`` and ``azure`` auth_types).
:param role_id: Role ID for Authentication (for ``approle``, ``aws_iam`` auth_types).
:param role_arn: AWS arn role,
:param kubernetes_role: Role for Authentication (for ``kubernetes`` auth_type).
:param kubernetes_jwt_path: Path for kubernetes jwt token (for ``kubernetes`` auth_type, default:
``/var/run/secrets/kubernetes.io/serviceaccount/token``).
Expand Down Expand Up @@ -107,6 +108,7 @@ def __init__(
key_id: str | None = None,
secret_id: str | None = None,
role_id: str | None = None,
role_arn: str | None = None,
kubernetes_role: str | None = None,
kubernetes_jwt_path: str = "/var/run/secrets/kubernetes.io/serviceaccount/token",
gcp_key_path: str | None = None,
Expand Down Expand Up @@ -147,6 +149,7 @@ def __init__(
key_id=key_id,
secret_id=secret_id,
role_id=role_id,
role_arn=role_arn,
kubernetes_role=kubernetes_role,
kubernetes_jwt_path=kubernetes_jwt_path,
gcp_key_path=gcp_key_path,
Expand Down
12 changes: 12 additions & 0 deletions docs/apache-airflow-providers-hashicorp/secrets-backends/hashicorp-vault.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,18 @@ Add "verify": "absolute path to ca-certificate file"
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"connections_path": "airflow-connections", "variables_path": null, "mount_point": "airflow", "url": "http://127.0.0.1:8200", "verify": "/etc/ssl/certs/ca-certificates"}

Vault authentication with AWS Assume Role STS
"""""""""""""""""""""""""""""""""""""""""""""

Add parameter "role_arn": "The AWS ARN of the role to assume"

.. code-block:: ini

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"connections_path": "airflow-connections", "variables_path": null, "mount_point": "airflow", "url": "http://127.0.0.1:8200", "auth_type": "aws_iam", "role_arn": "arn:aws:iam::123456789000:role/hashicorp-aws-iam-role"}


Using multiple mount points
"""""""""""""""""""""""""""

Expand Down