Skip to content

Close the logger's file descriptor once the process is complete.#47574

Merged
tirkarthi merged 3 commits intoapache:mainfrom
tirkarthi:dag-processor-file-leak
Mar 12, 2025
Merged

Close the logger's file descriptor once the process is complete.#47574
tirkarthi merged 3 commits intoapache:mainfrom
tirkarthi:dag-processor-file-leak

Conversation

@tirkarthi
Copy link
Contributor

@tirkarthi tirkarthi commented Mar 10, 2025

Initially I thought of accessing the filehandle through proc.log._logger._file where log is the BoundLogger but they are private and mypy doesn't like it. The other approach I thought was to store the filehandle as part of DagFileProcessorProcess itself. Thus whenever the processor is popped out after everything is complete the filehandle associated with the DagFileProcessorProcess is also closed. filehandle could be renamed to log_file_handle or something. I opened the PR for discussion to see if there are other approaches.

Closes #46048

Tests can be improved by asserting for close call but wanted to check first if there are any better approaches.

@tirkarthi tirkarthi force-pushed the dag-processor-file-leak branch from 4ab52d8 to 562591d Compare March 11, 2025 10:18
@tirkarthi tirkarthi merged commit aeec6ef into apache:main Mar 12, 2025
44 checks passed
@tirkarthi
Copy link
Contributor Author

Thanks @phanikumv and @jedcunningham for the review.

nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
…s complete. (apache#47574)

Close the logger's file descriptor once the DagFileProcessorProcess is complete. This also includes closing the file descriptor in case of timeouts.
@ashb
Copy link
Member

ashb commented Sep 24, 2025

Coming back to this many months later, I now worry that there is a memory leak in the dag processsor service, since otherwise popping the processor object out of the dict should have removed all references to the open filehandle, meaning the GC would close the file for us?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow dag processor exits with too many open files after sometime

4 participants