Add task context logging feature to allow forwarding messages to task logs #32646
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:logging
area:Scheduler
pankajkoti
force-pushed
the
arbitrary-log-shipper
branch
from
405191c
to
e490d8c
Compare
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
dstandish
reviewed
Jul 18, 2023
pankajkoti
force-pushed
the
arbitrary-log-shipper
branch
from
33f5611
to
641ff14
Compare
|
What does ship means here? I don’t see it explained anywhere and it isn’t a commonly used word in this context, as far as I know. |
Totally open to better names here. By ship i just mean send. We're sending from executor etc to task instance logs, which is something up to now we can't do. |
|
Also, if you have any other design suggestions, that is welcome |
pankajkoti
changed the title
|
The failing test is unrelated and seen in other PRs too. It was reported here getmoto/moto#7031 (comment) and likely supposed to get fixed |
i created #35687 which seems to fix it on my machine |
potiuk
approved these changes
Nov 16, 2023
potiuk
reviewed
Nov 16, 2023
o-nikolas
approved these changes
Nov 16, 2023
pankajkoti
added a commit
to astronomer/airflow
that referenced
this pull request
Nov 17, 2023
With the addition of taxt context logging feature in PR apache#32646, this PR extends the feature to AWS S3 when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core.
pankajkoti
added a commit
to astronomer/airflow
that referenced
this pull request
Nov 17, 2023
With the addition of taxt context logging feature in PR apache#32646, this PR extends the feature to GCP Cloud storage when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core.
pankajkoti
added a commit
that referenced
this pull request
Nov 17, 2023
…ure Blob Storage) (#32972) With the addition of task context logging feature in PR #32646, this PR extends the feature to Azure Blob Storage (WASB) when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core.
dstandish
pushed a commit
to astronomer/airflow
that referenced
this pull request
Nov 17, 2023
…earch With the addition of task context logging feature in PR apache#32646, this PR extends the feature to Elasticsearch when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core.
pankajkoti
added a commit
that referenced
this pull request
Nov 21, 2023
…earch (#32977) * Extend task context logging support for remote logging using Elasticsearch With the addition of task context logging feature in PR #32646, this PR extends the feature to Elasticsearch when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core. * update ensure_ti --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
ephraimbuddy
pushed a commit
that referenced
this pull request
Nov 23, 2023
…earch (#32977) * Extend task context logging support for remote logging using Elasticsearch With the addition of task context logging feature in PR #32646, this PR extends the feature to Elasticsearch when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core. * update ensure_ti --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
69 tasks
pankajkoti
added a commit
to astronomer/airflow
that referenced
this pull request
Nov 26, 2023
Using the feature built in apache#32646, when the scheduler marks tasks stuck in queued as failed, send such an explicit log indicating the action to the task logs so that it helps users identify why exactly the task was marked failed in such a case.
ephraimbuddy
pushed a commit
that referenced
this pull request
Nov 26, 2023
…earch (#32977) * Extend task context logging support for remote logging using Elasticsearch With the addition of task context logging feature in PR #32646, this PR extends the feature to Elasticsearch when is it set as remote logging store. Here, backward compatibility is ensured for older versions of Airflow that do not have the feature included in Airflow Core. * update ensure_ti --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
pankajkoti
added a commit
that referenced
this pull request
Nov 26, 2023
…35857) Using the feature built in #32646, when the scheduler marks tasks stuck in queued as failed, send such an explicit log indicating the action to the task logs so that it helps users identify why exactly the task was marked failed in such a case. --------- Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
27 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PR adds a feature by adding the
TaskContextLoggerclass thatcan forward messages from Airflow components like Scheduler,
Executor, etc to the task logs. This is helpful when in exception
scenarios the task is marked as failed from these components
e.g. when task times out because it remains stuck in the queue or
when a zombie task is killed. In such scenarios, currently, no task
logs are available in the UI and the user does not have a clue why
the task failed. Forwarding such event messages to the task logs
will give a clear idea to the user.
The PR also adds a config param to disable this feature in case
something is observed to be not working well in the Airflow
components due to the addition of this feature.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.