Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: eliminate race condition in cache for SHASUMS256 #294

Closed
wants to merge 2 commits into from

Conversation

erickzhao
Copy link
Member

@erickzhao erickzhao commented Jul 11, 2024

Fixes electron/packager#1552

When downloading the same version across multiple architectures at once (e.g. for a Universal macOS build), it's possible for fs.move to run into a race condition because the same SHASUMS256 file is being written to the cache for each arch.

@dsanders11 kindly pointed out that the code we have in putFileInCache is trying to mimic the overwrite option in fs.move, but follows an antipattern in Node.js:

Using fs.exists() to check for the existence of a file before calling fs.open(), fs.readFile(), or fs.writeFile() is not recommended. Doing so introduces a race condition, since other processes may change the file’s state between the two calls. Instead, user code should open/read/write the file directly and handle the error raised if the file does not exist.

@erickzhao erickzhao marked this pull request as ready for review July 11, 2024 20:11
@erickzhao erickzhao requested a review from a team as a code owner July 11, 2024 20:11
@erickzhao erickzhao closed this Jul 11, 2024
@erickzhao erickzhao reopened this Jul 11, 2024
Copy link
Member

@erikian erikian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@erickzhao erickzhao changed the title fix: allow overwriting SHASUMS256 in cache fix: eliminate race condition in cache for SHASUMS256 Jul 11, 2024
Copy link
Member

@MarshallOfSound MarshallOfSound left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is semantically correct as it allows one download job to override the other downloads checksums in flight.

We probably want to simply not put checksum files in the cache if we're never going to use them from the cache

@erickzhao
Copy link
Member Author

I don't think this is semantically correct as it allows one download job to override the other downloads checksums in flight.

Is that creating a separate race condition where we might have a limbo state where we attempt to read from SHASUMS256 mid-overwrite? Both files are otherwise be 100% identical since this is an arch/platform-agnostic SHASUMS256.txt file.

@dsanders11
Copy link
Member

I don't think this is semantically correct as it allows one download job to override the other downloads checksums in flight.

Isn't that the current behavior? If the file already exists it removes it before putting the new one in the cache. It's effectively already an overwrite, just one that's prone to race conditions by checking first.

@erickzhao
Copy link
Member Author

Regardless, going to close this in favour of #267

@erickzhao erickzhao closed this Jul 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

parallel download race condition errors out with: dest already exists
4 participants