feat: Upgrade Airflow from 2.9.3 to 2.10.5 (DENG-5881)

[sean-rose]

Description

Some notable changes...

Airflow (upgrading from 2.9.3 to 2.10.5)

• 2.10.0
  • try_number is no longer incremented during task execution (#39336).
    • I think this may allow reattach_on_restart to work more reliably.
  • OpenTelemetry traces for Airflow system and DAG runs (#37948).
  • Ability to use multiple executors (#40701).
  • Airflow now collects usage telemetry by default (#39510).
    • We're going to opt out.
  • Optional dark mode for Airflow UI (#39355).
  • Filters for active/paused and running/failed DAGs now work independently (#39701).
  • Log lines containing "error" or "exception" will be displayed in red text, and log lines containing "warn" will be displayed in yellow text (#39006).
    • The terms that trigger red/yellow log text highlighting can also be customized.
  • The rendered k8s pod spec is now available under a new tab in the task instance details (#39141).
  • Important executor logs are sent to task logs (#40468).
• 2.10.1
  • n/a
• 2.10.2
  • n/a
• 2.10.3
  • When marking a DAG run as failed any non-running, non-finished tasks will be marked as skipped (#43572).
• 2.10.4
  • Tasks stuck in a queued state will be re-queued up to two times by default (#44158).
• 2.10.5
  • Fix endless sensor rescheduling (#45250).

apache-airflow-providers-amazon (upgrading from 8.25.0 to 9.2.0)

• 9.2.0
  • Removed deprecated code (#42450).

apache-airflow-providers-cncf-kubernetes (upgrading from 8.3.3 to 10.1.0)

• 8.3.4
  • Fix Kubernetes executor task leak (#36998).
    • Oddly the ostensible fix for this was listed in the v8.3.0 release notes, but we definitely had to upgrade this provider to v8.3.4 specifically to fix this bug.
• 8.4.1
  • Changes to do_xcom_push and get_logs functionality for KubernetesJobOperator (#40814).
• 10.0.0
  • Removed deprecated code (#43689).
• 10.0.1
  • Allow for retry when tasks are stuck in queued (#43520).
• 10.1.0
  • Added logging support for init containers in KubernetesPodOperator (#43853).
  • Support grouping of log lines for KubernetesPodOperator (#44428).

apache-airflow-providers-google (upgrading from 10.21.0 to 14.0.0)

• 11.0.0
  • Removed deprecated code (#43953).
    • In particular this removed some Dataproc job operators which we were using.
  • Fixed incorrect query in BigQueryAsyncHook.create_job_for_partition_get (#44225).
    • Due to the incorrect query we had to temporarily change BigQueryTablePartitionExistenceSensors to run in reschedule mode, and this will allow us to revert to running them in deferrable mode.
• 12.0.0
  • Removed more deprecated code (#45084).
• 14.0.0
  • Removed even more deprecated code (#46235).
  • Fixed Dataproc operators failing to import without OpenLineage installed (#46561).
    • Fixing this bug is why we've force-upgraded this provider to v14.0.0.

apache-airflow-providers-http (upgrading from 4.12.0 to 5.0.0)

• 5.0.0
  • Removed deprecated code (#44542).
    • In particular this removed SimpleHttpOperator, which one of our DAGs was using.

apache-airflow-providers-postgres (upgrading from 5.11.2 to 6.0.0)

• 6.0.0
  • Removed deprecated code (#44705).

apache-airflow-providers-slack (upgrading from 8.7.1 to 9.0.0)

• 8.8.0
  • Added unfurl options to Slack notifier (#40694).
• 9.0.0
  • Removed deprecated code (#44693).

Related Tickets & Documents

• DENG-5881: Upgrade WTMO to Airflow 2.10
• https://github.com/mozilla-it/dataservices-infra/pull/644

[akkomar]

Why is this changing?

[sean-rose]

I mentioned that in the (lengthy) PR description:

• 14.0.0
  • Fixed Dataproc operators failing to import without OpenLineage installed (#46561).
    • Fixing this bug is why we've force-upgraded this provider to v14.0.0.

And that's mentioned in the commit message for b0bc472 (which should also end up in the final squashed merge commit message).

However, I agree it would be helpful for the reason for the override to be more obvious so you don't have to Git blame to find the reason. I'll add an explanatory comment in requirements-override.txt (and IMO going forward we should have a convention of always adding comments for such overrides).

[sean-rose]

Also FYI, there are some additional package dependency changes caused by forcing this upgrade (though none seem consequential):

-apache-airflow-providers-common-compat==1.3.0
+apache-airflow-providers-common-compat==1.5.1
-apache-airflow-providers-google==12.0.0
+apache-airflow-providers-google==14.0.0
-google-cloud-build==3.29.0
+google-cloud-build==3.31.1
+google-cloud-managedkafka==0.1.9
+joblib==1.4.2
+scikit-learn==1.6.1
+scipy==1.15.2
+threadpoolctl==3.6.0
+tqdm==4.67.1

[sean-rose]

Why is this changing?

In case you were also wondering why the apache-airflow-providers-cncf-kubernetes==8.3.4 override is being removed, we had forced an upgrade to that version to get a bug fix, and now Airflow 2.10.5 comes with apache-airflow-providers-cncf-kubernetes 10.1.0 by default, which includes that bug fix.

[akkomar]

I'll add an explanatory comment in requirements-override.txt (and IMO going forward we should have a convention of always adding comments for such overrides).

Great idea

[sean-rose]

With this PR deployed to staging I successfully tested the following DAGs:

• clean-gke-pods (GKE pod operator)
• bqetl_cjms_nonprod (Fivetran operators, bigquery-etl operators)
• bqetl_google_search_console (BigQuery sensors, bigquery-etl operators)
• socorro_import.crash_report_parquet (Dataproc operators)