Add autoscaling subcluster support and remove defaults

[peter-volkov]

Yandexcloud package dependency is updated.
New autoscaling feature is supported by the operator.
Default values are removed, so they can be set at the server side of the API.
closes: #17031

I have tested the PR using the related example DAG for both airflow 2.0 and 1.10.

[peter-volkov]

Build Images / Build PROD images 3.6 (pull_request_target) CI check:
ERROR: Cannot install apache-airflow-providers-sftp==2.1.0.dev0 and apache-airflow-providers-ssh 2.1.0.dev0 (from /docker-context-files/apache_airflow_providers_ssh-2.1.0.dev0-py3-none-any.whl) because these package versions have conflicting dependencies.
does not seem to be related to my PR.

[potiuk]

yep. already fixed. rebase to main will fix it.

[peter-volkov]

yep. already fixed. rebase to main will fix it.

Alright. And what should I do with Mysql test crashes?

[uranusjr]

The test failures seem irralevant to your change. (The google and aws providers are broken; we should fix them…)

[potiuk]

@uranusjr r FYI. The problem is that new dependency versions got released during the last few days and they broke our tests. Thanks to our constraint mechanism, this - luckily by design - only affects changes that make change in setup.py (as then we eagerly upgrade constraints). Due to the summit and personal stuff I had no time to fix those (but I will fix them soon - unless you would like to pick those up. They seem to be rather just fixing the tests so that they use 'stable' imports from the libraries.

[potiuk]

BTW @uranusjr . Before constraints mechanism was in place - such a "third-party-release" was a disastrous event breaking everyone PRs and we needed to scramble to fix those as fast as we could. right now, it will only affect a small subset of PRs (only those that change setup.py/scripts etc.) and we have time to react. This is all by design :).

[potiuk]

Can you please rebase to latest main @peter-volkov ?

[potiuk]

Closed/re-opened to re-trigger build.

[peter-volkov]

Can you please rebase to latest main @peter-volkov ?

Was on a vacation. Rebased now

[peter-volkov]

@potiuk is main branch supposed to be green now?

[potiuk]

Sort-of. We had fixed some of the problems but there are som flaky tests still - please rebase and let's see where it gets. We can merge it still if we see only flaky tests failure, this is not a blocker .

[uranusjr]

All failures here are out-of-resource of some kind, so probably not related to this PR.

[potiuk]

LGTM

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Indeed. Merging. Do not know about yandex too much, but it looks good.

[peter-volkov]

@potiuk how do we make a release of
https://pypi.org/project/apache-airflow-providers-yandex/
?

[potiuk]

We release all providers in waves in ~ 1 month cadence. Once this is merged it will be included. The next wave is going to happen in ~ 1.5 - 2 weeks.