Rework remote task log handling for the structlog era. #48491
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:logging
area:providers
area:task-sdk
provider:amazon
ashb
changed the title
ashb
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
ashb
force-pushed
the
overhaul-task-logging
branch
2 times, most recently
from
e25bd96 to
e43c7bb
Compare
ashb
changed the title
ashb
requested review from
amoghrajesh,
eladkal,
gopidesupavan,
kaxil,
o-nikolas and
potiuk
as code owners
kaxil
approved these changes
Mar 31, 2025
ashb
commented
Mar 31, 2025
ashb
commented
Mar 31, 2025
ashb
commented
Mar 31, 2025
jedcunningham
approved these changes
Apr 1, 2025
o-nikolas
reviewed
Apr 1, 2025
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
Show resolved
Hide resolved
providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
Show resolved
Hide resolved
Contributor
|
Does this affect elasticsearch / opensearch logging? asking because I don't see changes in this PR for these providers |
Member
Author
Oh yes, I forgot to deal with this. My plan for now is to make it fall back to the existing airflow.task handler. (I also really don't like we have two almost identical providers for ES and OS. Thanks a lot Elastic. |
ashb
force-pushed
the
overhaul-task-logging
branch
from
b6cea26 to
1d0c3eb
Compare
ashb
force-pushed
the
overhaul-task-logging
branch
3 times, most recently
from
6600ce2 to
6c2a9f3
Compare
This comment was marked as resolved.
This comment was marked as resolved.
ashb
force-pushed
the
overhaul-task-logging
branch
2 times, most recently
from
5baf022 to
062fb4b
Compare
Previously this feature was built on top of the stdlib logging.Handler interface, and it worked but had a few issues (even before we switched to structlog for Task SDK): - we had to use what is in many ways a hack with the "set_context" to get information down in to the task handler. - Discovering of the configured task handler was somewhat baroque - The whole thing is just complex due to the features of stdlib logging (loggers, propagate, handler levels etc etc.) - The upload was triggered somewhat "automatically" inside close, which from an abstraction point of view is messy. This changes things to have a more explicit interface purpose made for uploading task log files and for reading them, and perhaps more crucially for things like CloudWatch Logs, it (re)adds the ability to install a structlog processor that will receive every log message as it happens. The return types for the read et al functions were confusing the living daylights out of me, so I've created type alias to give the return types explicit names to reduce (my) confusion.
ashb
force-pushed
the
overhaul-task-logging
branch
from
062fb4b to
6191c13
Compare
eladkal
approved these changes
Apr 2, 2025
1 task
Contributor
|
We are missing handling for HdfsTaskHandler but I believe this is a niche one and we can do it later. created #48685 to followup |
Member
Author
|
mypy providers failure is fixed in #48686 |
Member
Author
|
I haven't changed deps, everything but lowest provider deps is passing, so I'm merging this now. |
Adaverse
reviewed
Apr 2, 2025
providers/microsoft/azure/src/airflow/providers/microsoft/azure/log/wasb_task_handler.py
Show resolved
Hide resolved
2 tasks
Member
I think |
This was referenced Apr 3, 2025
Member
Author
|
@jason810496 ES and OS are a bit of a mess tbh. They are entirely too specialized, and 95% of that isn't needed anymore now that Task SDK writes out JSON logs natively. |
Contributor
|
We can decide that these two providers (Elastic, open search) would be Airflow 3+ compatible from next release. |
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Apr 4, 2025
Previously this feature was built on top of the stdlib logging.Handler interface, and it worked but had a few issues (even before we switched to structlog for Task SDK): - we had to use what is in many ways a hack with the "set_context" to get information down in to the task handler. - Discovering of the configured task handler was somewhat baroque - The whole thing is just complex due to the features of stdlib logging (loggers, propagate, handler levels etc etc.) - The upload was triggered somewhat "automatically" inside close, which from an abstraction point of view is messy. This changes things to have a more explicit interface purpose made for uploading task log files and for reading them, and perhaps more crucially for things like CloudWatch Logs, it (re)adds the ability to install a structlog processor that will receive every log message as it happens. The return types for the read et al functions were confusing the living daylights out of me, so I've created type alias to give the return types explicit names to reduce (my) confusion.
diogotrodrigues
pushed a commit
to diogotrodrigues/airflow
that referenced
this pull request
Apr 6, 2025
Previously this feature was built on top of the stdlib logging.Handler interface, and it worked but had a few issues (even before we switched to structlog for Task SDK): - we had to use what is in many ways a hack with the "set_context" to get information down in to the task handler. - Discovering of the configured task handler was somewhat baroque - The whole thing is just complex due to the features of stdlib logging (loggers, propagate, handler levels etc etc.) - The upload was triggered somewhat "automatically" inside close, which from an abstraction point of view is messy. This changes things to have a more explicit interface purpose made for uploading task log files and for reading them, and perhaps more crucially for things like CloudWatch Logs, it (re)adds the ability to install a structlog processor that will receive every log message as it happens. The return types for the read et al functions were confusing the living daylights out of me, so I've created type alias to give the return types explicit names to reduce (my) confusion.
simonprydden
pushed a commit
to simonprydden/airflow
that referenced
this pull request
Apr 8, 2025
Previously this feature was built on top of the stdlib logging.Handler interface, and it worked but had a few issues (even before we switched to structlog for Task SDK): - we had to use what is in many ways a hack with the "set_context" to get information down in to the task handler. - Discovering of the configured task handler was somewhat baroque - The whole thing is just complex due to the features of stdlib logging (loggers, propagate, handler levels etc etc.) - The upload was triggered somewhat "automatically" inside close, which from an abstraction point of view is messy. This changes things to have a more explicit interface purpose made for uploading task log files and for reading them, and perhaps more crucially for things like CloudWatch Logs, it (re)adds the ability to install a structlog processor that will receive every log message as it happens. The return types for the read et al functions were confusing the living daylights out of me, so I've created type alias to give the return types explicit names to reduce (my) confusion.
18 tasks
2 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously this feature was built on top of the stdlib logging.Handler
interface, and it worked but had a few issues (even before we switched to
structlog for Task SDK):
information down in to the task handler.
(loggers, propagate, handler levels etc etc.)
an abstraction point of view is messy.
This changes things to have a more explicit interface purpose made for
uploading task log files and for reading them, and perhaps more crucially for
things like CloudWatch Logs, it (re)adds the ability to install a structlog
processor that will recieve every log message as it happens.
The return types for the read et al functions were confusing the living
daylights out of me, so I've created type alias to give the return types
explicit names to reduce (my) confusion.