Feature/add token authentication to internal api

[jscheffl]

During the implementation of authentication protection for AIP-69 I realized that Internal API does not carry support for authentication access.

Due to review of #40897 - alternative PR using token authentication on internal API

Noteworthy: Compared to other use I just added the token to the JSON request body dict, thought this is "simpler" than adding a POST/request parameter to the call, API is internal anyway.

[potiuk]

I looked in detail, and (sorry for that I was just quickly looking it up - but the concept is similar).

The actual code I think we should model it from is similar but adds a bit more protection is the one we use to retrieve logs (because It also adds expiry-date for the request) - and it uses JWTSigner Instead of itsdangerous.

Here is the server side:

airflow/airflow/utils/serve_logs.py

Line 80 in c2a54ef

@flask_app.before_request

And here is the client side:

https://github.com/apache/airflow/blob/main/airflow/utils/log/file_task_handler.py#L94

Conceptually it's very similar to signer - you can sign the whole payload of the request - and pass it as "Authorization:" header.

This way:

• the signature cannot be re-used in another request
• it has expiry date
• also it can have salt added (the audience is effectively salt)

[potiuk]

(and I am quickly fixing the selective check error)

[potiuk]

Fix to selective checks failing here: #40904

[jscheffl]

@potiuk Thanks for the fix in selective checks. Have re-worked now with JWT approach - this is how you wanted to have it?

Note I am off on a cycling tour through switzerland tomorrow, will check probably in the evening, will not be online frequently.

[potiuk]

I hope you have a good tour!

Yes that's what I thought about.

[potiuk]

One small comment on that. I think the mechanism is cool. But we should use different secret key - not the webserver one. Otherwise DAG authors will have access to webserver key which is somewhat sensitive. the Internal API key should be different.

[jscheffl]

One small comment on that. I think the mechanism is cool. But we should use different secret key - not the webserver one. Otherwise DAG authors will have access to webserver key which is somewhat sensitive. the Internal API key should be different.

Yes, DAG authors could have access to the secret key. But all is internal API components need to have a shared secret. Even if we use any other means (adding another different key, using a specific keyfile, even if public/private keypair is used) the DAG code has access to these details and could grab the key.

This might be a very more complex mechanism needed if the key should be secured and this might impose a more complex implementation. I'd propose to take this into consideration when we go to Airflow 3, as ash also described a few mechanisms in his AIP-72 that might be used.

So, WDYT:
a) okay, LGTM
b) please use one additional key for internal API, replace webserver-key - gut a static shared key in Airflow conf is okay in general
c) We need to re-think the key distribution also in Airflow 2.10 for this

[potiuk]

b)