[AIRFLOW-4248] Fix 'FileExistsError' makedirs() race condition #5047
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make sure you have checked all steps below.
Jira
Description
The
FileProcessorHandler
's_init_file
method can trigger an unhandled exception due to a race condition when multiple processes attempt to create the same directory on a shared filesystem. When this occurs aFileExistsError
is raised, and if it is the scheduler process, can prevent tasks from starting until the scheduler is restarted.This PR catches the exception and handles it similarly to how it's done when using the
exists_ok=True
argument of Python v3.2'sos.makedirs
function , which does anisdir()
call instead of checking for theEEXIST
error code (comments inos.makedirs
state that checking for this error code is unreliable because the OS can return other error codes instead). This PR also replaces the existingEEXIST
check in theFileProcessorHandler.__init__
method to instead useisdir()
.Note: It'd be slightly nicer to just use the
os.makdirs(exists_ok=True)
parameter instead, which can be done if this PR becomes dependent uponAIP-3
.Tests
Commits
Documentation
Code Quality
flake8