Fix EKS token generation #51333
Fix EKS token generation #51333
Conversation
Tasks now executing via TaskSDK may not have access to the Database and so initting the orm will fail. During the creation of the eks token, we import the EksHook, which triggered the airflow init to run (specifically in settings where the orm was initted). Export the env var PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1 so that this does not happen. Also update the output processing to: 1) To be fully bash compliant (drop the use of `grep` with options that are not supported on some platforms) 2) Handle the new struct logging output/formatting of Airflow (this was also causing the token generation code to fail)
boring-cyborg
bot
added
area:providers
provider:amazon
|
ah good find this :)
|
|
wow, thanks for the quick fix on this! |
Tasks now executing via TaskSDK may not have access to the Database and so initting the orm will fail. During the creation of the eks token, we import the EksHook, which triggered the airflow init to run (specifically in settings where the orm was initted). Export the env var PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1 so that this does not happen. Also update the output processing to: 1) To be fully bash compliant (drop the use of `grep` with options that are not supported on some platforms) 2) Handle the new struct logging output/formatting of Airflow (this was also causing the token generation code to fail)
|
Hello @o-nikolas , But, we are using the EksPodOperator from inside our DAG to execute some Pod with image. We were using that system in version 2.x without issue, but since we use version 3 we face some trouble. Main point is that we use a aws_conn_id="xxx" in the setup of our EksPodOperator allowing us to assume a different role when the dag need to comunicate with EKS. But that system seems broken in v3 and i'am thinking if it's not linked to the TaskSDK, if the ORM wasn't able to start, i suppose that my DAG have no access to Ariflow database neither, and by so no access to the list of configured connection id ? Or did i misunderstood the TaskSDK impact? |
@valereColleville Do you mind creating a new Github Issue for this? With all the details you can find that are relevant (more explanation, code samples, etc). I don't think this resolved PR is the right place to start this new discussion. Thanks! |
5 participants
Tasks now executing via TaskSDK may not have access to the Database and so initting the orm will fail. During the creation of the eks token, we import the EksHook, which triggered the airflow init to run (specifically in settings where the orm was initted). Export the env var PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1 so that this does not happen.
Also update the output processing to:
grepwith options thatare not supported on some platforms)
also causing the token generation code to fail)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.