AIP-72: Add support to get Variables in task sdk outside of context

[amoghrajesh]

Body

To write dags using the task sdk, we need to add support to get variables using client side definitions and the task sdk machinery.

Move Secrets Backend on the client side!

Currently, Secrets Backends are only supported (and configured) on the API Server. The Python Task SDK client does not do any lookup locally and relies on Server.

This is for supporting the following use-case:

Deploying tasks in transient or ephemeral environments (e.g., GPU cloud services) where secrets need to be retrieved dynamically via the API

This allows for Secrets to be defined once and allows central management with fewer configurations on the worker.

However, there is a complementary use case as below

An organization processes sensitive customer data for financial transactions. Regulatory and security policies mandate that certain credentials (e.g., database credentials, API keys, encryption keys) cannot leave a specific network zone. This includes ensuring that the Airflow scheduler, API server, or other components outside this zone cannot access these secrets.

This use-case means we need a way where secrets are already provisioned locally on the worker.

Secrets Backend will be available to be configured on the Client by default. Packaging-wise: The External Secrets Backend will be part of Providers, which depend on Task SDK.

For the API-Server, by default it will only look at Env Variable and then Database by default.

1. Optionally, if users want the API server to fetch secrets from the External Secrets backend, they will have to install the Task SDK (and relevant providers like Hashicorp) with the scheduler/Airflow core code.
2. Longer term: we might be able to remove requirements on Task SDK by implementing some protocol but this will be discussed again post AF 3.0.