From 67ad683a41d421d1edad35b11a0ee49824469dd8 Mon Sep 17 00:00:00 2001 From: Elad Kalif <45845474+eladkal@users.noreply.github.com> Date: Fri, 1 Mar 2024 11:52:16 +0200 Subject: [PATCH 1/3] docs: Write to secrets store is not supported by design --- .../core-extensions/secrets-backends.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst index 26ee3ce882a02..61b91f5edf483 100644 --- a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst +++ b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst @@ -26,6 +26,22 @@ than from its own Database. While storing such information in Airflow's database enterprise customers already have some secret managers storing secrets, and Airflow can tap into those via providers that implement secrets backends for services Airflow integrates with. +.. note:: +Secret Backend integration do not allow writes to the secret backend. +This is a design choice as normally secret stores require elevated permissions to write as it is a protected resource. +That means ``Variable.set(...)`` will write to the Airflow metastore. +If you need to update a value of a secret stored in the secret backend you must do it explicitly. That can be done +by using operator that writes to the secret backend of your choice. + +.. warning:: +If you have key ``foo`` in secret backend and you will do ``Variable.set(key='foo',...)`` it will create +Airflow Variable with key ``foo`` in the Airflow metastore. It means you will have 2 secrets with key ``foo``. +While this is possible, Airflow detects that this situation is likely wrong and output to the task log a warning that +explains while the write request is honored it will be ignored with the next read. The reason for this is when executing +``Variable.get('foo')``, it will read the value from the secret backend. The value stored in Airflow metastore will be +ignored due to priority given to the secret backend. + + You can also take a look at Secret backends available in the core Airflow in :doc:`apache-airflow:security/secrets/secrets-backend/index` and here you can see the ones From 3dcae819686010c5bf8761176e7ca418b8727620 Mon Sep 17 00:00:00 2001 From: Elad Kalif <45845474+eladkal@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:14:05 +0200 Subject: [PATCH 2/3] fix build docs --- .../core-extensions/secrets-backends.rst | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst index 61b91f5edf483..bb582c6898c83 100644 --- a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst +++ b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst @@ -27,19 +27,21 @@ enterprise customers already have some secret managers storing secrets, and Airf via providers that implement secrets backends for services Airflow integrates with. .. note:: -Secret Backend integration do not allow writes to the secret backend. -This is a design choice as normally secret stores require elevated permissions to write as it is a protected resource. -That means ``Variable.set(...)`` will write to the Airflow metastore. -If you need to update a value of a secret stored in the secret backend you must do it explicitly. That can be done -by using operator that writes to the secret backend of your choice. + + Secret Backend integration do not allow writes to the secret backend. + This is a design choice as normally secret stores require elevated permissions to write as it is a protected resource. + That means ``Variable.set(...)`` will write to the Airflow metastore. + If you need to update a value of a secret stored in the secret backend you must do it explicitly. That can be done + by using operator that writes to the secret backend of your choice. .. warning:: -If you have key ``foo`` in secret backend and you will do ``Variable.set(key='foo',...)`` it will create -Airflow Variable with key ``foo`` in the Airflow metastore. It means you will have 2 secrets with key ``foo``. -While this is possible, Airflow detects that this situation is likely wrong and output to the task log a warning that -explains while the write request is honored it will be ignored with the next read. The reason for this is when executing -``Variable.get('foo')``, it will read the value from the secret backend. The value stored in Airflow metastore will be -ignored due to priority given to the secret backend. + + If you have key ``foo`` in secret backend and you will do ``Variable.set(key='foo',...)`` it will create + Airflow Variable with key ``foo`` in the Airflow metastore. It means you will have 2 secrets with key ``foo``. + While this is possible, Airflow detects that this situation is likely wrong and output to the task log a warning that + explains while the write request is honored it will be ignored with the next read. The reason for this is when + executing ``Variable.get('foo')``, it will read the value from the secret backend. The value stored in Airflow + metastore will be ignored due to priority given to the secret backend. You can also take a From 6d34a62f761edc4f4f7fc5a826678aafc3a55fb2 Mon Sep 17 00:00:00 2001 From: Elad Kalif <45845474+eladkal@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:33:21 +0200 Subject: [PATCH 3/3] small clarification --- .../core-extensions/secrets-backends.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst index bb582c6898c83..9aa2e7d367fec 100644 --- a/docs/apache-airflow-providers/core-extensions/secrets-backends.rst +++ b/docs/apache-airflow-providers/core-extensions/secrets-backends.rst @@ -30,7 +30,7 @@ via providers that implement secrets backends for services Airflow integrates wi Secret Backend integration do not allow writes to the secret backend. This is a design choice as normally secret stores require elevated permissions to write as it is a protected resource. - That means ``Variable.set(...)`` will write to the Airflow metastore. + That means ``Variable.set(...)`` will write to the Airflow metastore even if you use secret backend. If you need to update a value of a secret stored in the secret backend you must do it explicitly. That can be done by using operator that writes to the secret backend of your choice.