fix(gatsby-source-filesystem): Retry stalled remote file downloads #20843
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
df34bb6 to
d673e9a
Compare
pieh
reviewed
Jan 24, 2020
pieh
reviewed
Jan 24, 2020
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
837654d to
5304fc9
Compare
sidharthachatterjee
approved these changes
Jan 31, 2020
sidharthachatterjee
suggested changes
Jan 31, 2020
packages/gatsby-source-filesystem/src/create-remote-file-node.js
Outdated
Show resolved
Hide resolved
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
56d11bd to
f02a80c
Compare
sidharthachatterjee
approved these changes
Jan 31, 2020
sidharthachatterjee
added
the
bot: merge on green
ehowey
pushed a commit
to ehowey/gatsby
that referenced
this pull request
Jan 31, 2020
…atsbyjs#20843) * fix(gatsby-source-filesystem): Retry stalled remote file downloads * Switch to got@8 * Await the recursive callback * Add tests for loading fix * Don't reset on data. Start timout after `response` * Add got timeout * Don't await the recursive call * Use constants for connection timeouts
muescha
reviewed
Feb 15, 2020
| }) | ||
| const fsWriteStream = fs.createWriteStream(tmpFilename) | ||
| responseStream.pipe(fsWriteStream) | ||
| responseStream.on(`downloadProgress`, pro => console.log(pro)) |
There was a problem hiding this comment.
the console.log is removed accidentally in resetTimeout?
|
I think this is breaking my build (at least locally with Don't have time to grok the code, but what I think is happening: I'm using Ultimately, what I'm sure of is that increasing the timeout to 20 min fixes my issues. |
mwfrost
pushed a commit
to mwfrost/gatsby
that referenced
this pull request
Apr 20, 2023
…atsbyjs#20843) * fix(gatsby-source-filesystem): Retry stalled remote file downloads * Switch to got@8 * Await the recursive callback * Add tests for loading fix * Don't reset on data. Start timout after `response` * Add got timeout * Don't await the recursive call * Use constants for connection timeouts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
gatsby-source-filesystem has a problem where downloads would stall and never timeout or error. Eventually all the concurrent downloads were stalled, preventing any more starting. This was causing builds to stall indefinitely.
This PR adds a timeout in
createRemoteFileNode. If nodataevents are received in the download stream for 30 seconds then the download is cancelled and retried up to three times. In tests this unblocks the downloads on the first attempt.WIP while I'm writing unit tests.
Documentation
Related Issues