feat(gatsby-source-filesystem): add an environment variable to control concurrent queue size

[violy]

related to #13073

[pieh]

Thanks for changes @violy!

Just curious what were values you used for timeout / retries - maybe better approach would be to change defaults there?

[violy]

as I explain on the issue #13073 — during my troubleshooting, I observed this warning, for different reasons (server overload, network / local overload) related to my stack and my work environment, and my successive attempts to valid locally my build.

During my troubleshooting, I've try to change timeout & retries values, but with no effect. The only success was changing concurrent to lower value (10 instead of 200, in my case) the original question in my issue / PR.
But I think to other case where troubleshooting could be painful, access to const via .env variables or equivalent, is a good way to adjust the stack with no pain.

When I finally deployed my app, via netlify, there was no problem (no need to fix with custom fork 😅)!
So initial default static values may be not so bad!

[pieh]

So if changing those didn't have any effect I'm not sure it makes sense to add yet more moving parts. I would prefer to just add GATSBY_CONCURRENT_DOWNLOAD and more detailed error message here.

[muescha]

the new possible env variables should be anywhere documented:

• GATSBY_CONCURRENT_DOWNLOAD
• GATSBY_DOWNLOAD_TIMEOUT
• GATSBY_DOWNLOAD_RETRIES

[violy]

Thank you, according your comment @pieh
I've update my PR, by removing the variables GATSBY_DOWNLOAD_TIMEOUT & GATSBY_DOWNLOAD_RETRIES maybe useless.

[muescha]

Maybe it would be nice to have an "main entry point" for all "magic numbers" and a default solution how to overwrite it.

[LekoArts]

@violy Hi 👋 Do you have time to update your PR (adding information to README) and resolve the conflicts? Would be cool to have this in!

[violy]

@LekoArts thanks for your request,
so I've resolve conflict and add a small word in the README Configuration paragraph and force-push all the change in one commit.

[antoinerousseau]

LGTM

[violy]

Any news / reviewer ?

[DSchau]

Seems reasonable.

Longer term, we very much prefer to bake in optimized, smart defaults (or a system that adapts, e.g. will back off the queue if failures arise) but until #10049 lands this is an OK approach that we can offer to people who are blocked in some way.

Thank you!

[muescha]

This download manager should also include an cache and some strategies how not redownload media assets when nothing is changed.

[DSchau]

@muescha I think the issue (#10049) is a better place for that comment; thanks! Otherwise it's very possible we'll lose track of your suggestions.

[muescha]

Is there also a stragegy only to download the used media assets (when maybe more assets are saved in wordpress but only some used in the used articles)?

[muescha]

Thanks for your hint. I was thinking i am already in the other issue. #commentOnMobileProblems ;)

[KyleAMathews]

Yeah, env variables seem appropriate now as a temporary fix until we can work on #10049