Close the logger's file descriptor once the process is complete.#47574
Merged
tirkarthi merged 3 commits intoapache:mainfrom Mar 12, 2025
Merged
Close the logger's file descriptor once the process is complete.#47574tirkarthi merged 3 commits intoapache:mainfrom
tirkarthi merged 3 commits intoapache:mainfrom
Conversation
4ab52d8 to
562591d
Compare
2 tasks
phanikumv
approved these changes
Mar 12, 2025
Contributor
Author
|
Thanks @phanikumv and @jedcunningham for the review. |
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Apr 4, 2025
…s complete. (apache#47574) Close the logger's file descriptor once the DagFileProcessorProcess is complete. This also includes closing the file descriptor in case of timeouts.
Member
|
Coming back to this many months later, I now worry that there is a memory leak in the dag processsor service, since otherwise popping the processor object out of the dict should have removed all references to the open filehandle, meaning the GC would close the file for us? |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initially I thought of accessing the filehandle through
proc.log._logger._filewherelogis theBoundLoggerbut they are private and mypy doesn't like it. The other approach I thought was to store the filehandle as part ofDagFileProcessorProcessitself. Thus whenever the processor is popped out after everything is complete the filehandle associated with theDagFileProcessorProcessis also closed.filehandlecould be renamed tolog_file_handleor something. I opened the PR for discussion to see if there are other approaches.Closes #46048
Tests can be improved by asserting for close call but wanted to check first if there are any better approaches.