Fix the write-to-es feature #53821

Owen-CH-Leung · 2025-07-28T15:11:42Z

ElasticsearchTaskHandler has a feature that allows task log to be directly written to elasticsearch. Such feature is broken in airflow 3.

The root cause is that in Airflow 3, the write path for remote logging has changed. There's a detailed description here by @jason810496 which illustrates how it works in airflow 3. ( Thank you Jason ! )

In summary, the solution is to add RemoteLogIO for Elasticsearch also. When write_to_es is set to true, airflow will initiate a ElasticsearchRemoteLogIO which will handle the task log writing to elasticsearch

…uild-doc CI.

jason810496

Nice! Big thanks for helping out!

Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.
LGTM overall.

providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py

providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

ashb · 2025-07-30T08:21:23Z

providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py

+    base_log_folder: Path = attrs.field(converter=Path)
+    delete_local_copy: bool
+
+    processors = ()


This should probably be set up as a processor so that log messages get sent to ElasticSearch as soon as they are written, just once when the log is "closed". See CloudWatch logging handler for the only example we have right now.

That can be a future PR though.

@ashb I took a quick look at how we'd make this a processor in the future and I got stuck on the fact that while the upload() method takes the ti as an argument, the process function takes only the event_dict, and we need the ti to construct the log_id which needs to be on every log event that gets indexed to ES/OS.

Can we expect each log event from the task logger to contain the ti or otherwise give us the log_id components? (Or is that a necessary future change?)

Sorry for the late reply. Here are some updates that might answer your question:

The LogTemplate DB Model will be removed from ES, which also means "we need the ti to construct the log_id which needs to be on every log event that gets indexed to ES/OS." will never need anymore. We will retrieve log_id_template directly from airflow.conf instead of fetching from LogTemplate DB Model.

Detail in https://lists.apache.org/thread/nlmhs1plo77qnlp7rqk27mkb2hs41f1p

ashb

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.

providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py

airflow-core/src/airflow/config_templates/airflow_local_settings.py

…rite on RemoteIO class

…endency

Owen-CH-Leung · 2025-08-07T15:05:23Z

@jason810496 @ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks

providers/elasticsearch/pyproject.toml

jason810496

Nice! Thanks for the update!
It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!

providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

providers/elasticsearch/tests/conftest.py

Owen-CH-Leung · 2025-08-08T06:15:15Z

Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!

~~I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase~~

No worries I think I found it. Let me expand the testing logic

jason810496 · 2025-08-10T15:00:23Z

~~I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase~~

No worries I think I found it. Let me expand the testing logic

Sorry for the late reply.

The following command will be helpful to setup Airflow with ElasticSearch in Breeze.

breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>

It would be great to test with the following version matrix for testing compatibility.
List of the <version> to be replaced with:

3.0.3
3.0.4
main ( for unreleased 3.1.x version, but this one is a bit more complicated )
1. Switch to main
2. Run rm -rf dist/*
3. Run breeze release-management prepare-airflow-distributions
4. Run breeze release-management prepare-task-sdk-distributions
5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.

Thanks a lot!

…ard to be ready

jason810496

Nice! The integration tests with test container LGTM.

It would be nice to test the change by start-airflow command with different Airflow core versions mentioned above to ensure the user behavior. Thanks!

Owen-CH-Leung · 2025-10-19T02:47:14Z

@jason810496 Thanks. I can confirm that the write-to-es feature works for version 3.0.6 and 3.1.0. the log-reading feature in airflow 2.11 also works as before.

@jhgoebbert Sorry for my late reply ! I've adopted your suggestion to capture the BulkIndexError exception & change the value file also

eladkal · 2025-10-28T09:23:02Z

@Owen-CH-Leung can you rebase and resolve conflicts?

eladkal · 2025-10-28T09:25:26Z

chart/values.yaml

  elasticsearch:
    json_format: 'True'
-    log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
+    log_id_template: "{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}"


Is this a breaking change for the chart?
cc @jedcunningham

Yes, but we do not have execution date anymore.

So maybe we need to make it based on AF2/3 as string?

Can someone refresh my memory - does this config item control writing, reading, or both?

IOW is there a risk here that if the chart default changes, or changes based on 2.x vs 3.x, that valid logs written from Airflow 2.x tasks won't be readable when a stack goes to Airflow 3.x?

(I think I recall that the task run table has a column for its log_id template, so the correct log_id can always be reconstructed for any task run...however I also remember somebody mentioning this column might be removed? I just don't remember exact details here.)

Anyway earlier logs vanishing in the UI because of changing chart defaults would be an uncomfortable surprise 😅

Owen-CH-Leung · 2025-10-29T07:19:30Z

@Owen-CH-Leung can you rebase and resolve conflicts?

Rebased. There're errors at main though for roviders/standard/tests/unit/standard/operators/test_python.py::TestPythonVirtualenvOperator

topherinternational · 2026-01-13T12:35:19Z

@Owen-CH-Leung et al, any reason we can't get this rebased and ready for merge soon?

My organization (and many others I expect) is still blocked from going to Airflow 3 since last summer because of ES logging, this PR should unblock us so we are very excited to see it release 🤗

Introduce ElasticsearchRemoteLogIO

2233ddc

boring-cyborg bot added area:ConfigTemplates area:logging area:providers provider:elasticsearch labels Jul 28, 2025

Owen-CH-Leung added 4 commits July 29, 2025 13:15

Fix mypy error. Fix ruff format error. Remove test_write_to_es. Fix b…

9bf1cf4

…uild-doc CI.

Fix mypy error. Fix ruff format error. Fix spelling mistakes

68c5546

Fix ruff check

b407075

Fix ruff check on testEsTaskhandler

65f78c5

Owen-CH-Leung marked this pull request as ready for review July 29, 2025 12:48

jason810496 reviewed Jul 30, 2025

View reviewed changes

jason810496 requested review from ashb and dstandish July 30, 2025 03:55

ashb reviewed Jul 30, 2025

View reviewed changes

ashb requested changes Jul 30, 2025

View reviewed changes

providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py Outdated Show resolved Hide resolved

ashb reviewed Jul 30, 2025

View reviewed changes

airflow-core/src/airflow/config_templates/airflow_local_settings.py Outdated Show resolved Hide resolved

Owen-CH-Leung added 7 commits July 31, 2025 22:29

Merge branch 'main' into fix-write-to-es-feature

960166b

Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/w…

3e59474

…rite on RemoteIO class

Fix existing unit tests

a60a315

Add unit test for ElasticsearchRemoteIO. Add testcontainer as new dep…

aff9a30

…endency

Fix provider test

b1cb760

Fix typo

f3ce8e7

Fix long running provider test

d28d40a

potiuk reviewed Aug 7, 2025

View reviewed changes

providers/elasticsearch/pyproject.toml Outdated Show resolved Hide resolved

jason810496 reviewed Aug 7, 2025

View reviewed changes

providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py Outdated Show resolved Hide resolved

providers/elasticsearch/tests/conftest.py Outdated Show resolved Hide resolved

Remove leftover comments. Move testcontainer dependency to devel-common

998aeee

jason810496 mentioned this pull request Aug 10, 2025

Simplify installing airflow from GitHub repo in breeze #54070

Merged

Owen-CH-Leung added 7 commits October 10, 2025 14:59

Merge branch 'main' into fix-write-to-es-feature

f0a893e

Increase timeout to 120. Disable unnecessary modules of ES

059906e

Extend pytest execution and setup timeout to 300

132922d

Combine read write es into one test. Pre-create index and wait for sh…

f822f86

…ard to be ready

Print container stats and indices to debug

e4731ed

Disable disk-based allocation of shard

aec7374

Remove debug lines

1ad3376

jason810496 reviewed Oct 13, 2025

View reviewed changes

Owen-CH-Leung added 7 commits October 15, 2025 21:41

Merge branch 'main' into fix-write-to-es-feature

e598b4e

Merge branch 'apache:main' into fix-write-to-es-feature

0711079

Fix infinite read

b5d7e01

Fix failing CI

765e7bc

Fix failing CI

18d39a6

Merge branch 'main' into fix-write-to-es-feature

3694219

Merge branch 'main' into fix-write-to-es-feature

49fca58

potiuk requested a review from ashb October 18, 2025 23:06

Update default helm value file

d9b2444

Owen-CH-Leung requested review from hussein-awala and jedcunningham as code owners October 19, 2025 01:44

eladkal reviewed Oct 28, 2025

View reviewed changes

Owen-CH-Leung added 2 commits October 29, 2025 12:21

Merge branch 'main' into fix-write-to-es-feature

dcc8926

Fix failing ruff

6a0e355

jason810496 mentioned this pull request Dec 4, 2025

Drop LogTemplate DB Model #59027

Draft

Merge branch 'main' into fix-write-to-es-feature

4b648e4

jason810496 requested a review from jscheffl as a code owner December 4, 2025 06:29

joshgamache mentioned this pull request Dec 15, 2025

Task: Send Airflow logs to Elastic rather than GCS bcgov/cas-compliance#440

Open

Fix the write-to-es feature #53821

Are you sure you want to change the base?

Fix the write-to-es feature #53821

Uh oh!

Conversation

Owen-CH-Leung commented Jul 28, 2025 • edited by eladkal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashb Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

topherinternational Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

jason810496 Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Owen-CH-Leung commented Aug 7, 2025

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Owen-CH-Leung commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jason810496 commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Owen-CH-Leung commented Oct 19, 2025

Uh oh!

eladkal commented Oct 28, 2025

Uh oh!

eladkal Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

jscheffl Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

topherinternational Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Owen-CH-Leung commented Oct 29, 2025

Uh oh!

topherinternational commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Owen-CH-Leung commented Jul 28, 2025 •

edited by eladkal

Loading

Owen-CH-Leung commented Aug 8, 2025 •

edited

Loading

jason810496 commented Aug 10, 2025 •

edited

Loading