Contributing to this repository is HARD

[eyalroth]

I just had a small fix I wanted to contribute to this repository, but the process was difficult, time consuming and daunting, and all because of the extremely cumbersome process of setting up the repository locally.

My setup process

1. Upon cloning the repository I encountered a git problem I've never seen before: "The remote end hung up unexpectedly". Eventually I managed to workaround the problem by increasing http.postBuffer and using the HTTPS clone link instead of the SSH one, but that obviously took a few attempts and quite some time to resolve.

2. I had to wait for the entire repo to download, which is a bit more than 600MB. On a slower-than-normal connection (5-10 Mbps) this could take up to 20 minutes.

3. After installing Yarn, which was pretty straightforward, I encountered a problem which seems to take a variety of different forms. Troubleshooting this problem took an immense amount of time, since I had to clean the Yarn cache and yarn install multiple times for each workaround attempt. Also there was the --network-concurrency 1 attempt which took forever.
   I eventually resorted to upgrade my Windows from 10.0.17134.799 to 10.0.18362.295 in a final desperate attempt to resolve these issues by upgrading WSL. Lucky for me, that worked, but took quite a long time (couple of hours).

4. On to the next Yarn error: "There appears to be trouble with your network connection. Retrying..." (+#5259). This was luckily resolved faster than the previous error with the --network-timeout workaround, but still took more than just a few minutes to fix.

5. Finally, yarn install finished without any error, but that alone took more than 10 minutes.

6. yarn run bootstrap takes a few more minutes to complete.

7. Finally, some tests (less than 20) fail. I didn't bother making them pass since they were not related to the package/plugin I was intending to fix.

Proposal

As I said in the beginning, this is one hell of a bad user experience for anyone who is just casual contributor, and I assume these problems have led others to eventually give up on contributing to this repository.

I believe there are two major steps that can be taken to dramatically improve this experience.

1. Reduce the repository size

This was very recently discussed in #16722, but I believe the discussion there was missing the fact that the reason this repository is so big is not because it is a mono-repo, but because more than half of it includes the docs directory, and especially the docs/blog directory:

(.git and node_modules can obviously be ignored in this demonstration)

There are two possible ways of remedying this: (a) Move the documentation / blog out of the repository, or (b) Make use of Git LFS to basically remove all the documentation / blog images from local repositories; though, I have no previous experience with this tool so I cannot attest to how well it works.

2. Move away from Yarn

The vast majority of my time consumed in the setup process was invested in solving Yarn problems. I can't say how common are these problems, especially since I am quite new to node and javascript, but from what little research I made I gather that some of these problems are indeed common in large Yarn projects.

The official documentation explains that the reason this repository is using Yarn instead of NPM is mainly due to the "workspaces" feature, but from what I gather that feature is possible to achieve in NPM via Lerna (and some say it is even better).

I understand this is probably quite a major step and requires a lot of work, but if Yarn is going to keep causing problems, perhaps it's better making that transition earlier so it will be easier.

[georgiee]

Hi @eyalroth,
I did not go through setting up Gatsby yet but I did so with other larger repositories like Angular & Ruby on Rails. You will almost always leave your comfort zone because you enter a large piece of software that isn't crafted by you. This will hurt on the first day. But it seems that you reached the point that Gatsby is almost running locally and this within a day — I would say that's good enough. That time it took for you might be related to your still great potential to learn things in the Node & JavaScript world, other people might have a better time setting Gatsby up.

Here two things that might help you the next time you want to contribute to any larger project:

1. git clone size & docs
To reduce the download size of the clone I would have used a shallow clone:

git clone --depth 1 your-repo-url

That way you will save A LOT of files that are not relevant if you are only relying on the current state as it drops all the files from the past. I'm a fan of including the docs and blog posts. It's so handy to have all those information at hands if you dig deeper into the framework.

2. yarn vs npm
I love yarn, I love their workspace feature and I can tell you as a user of the workspace functionality: It feels faster and more convenient if the package manager itself takes care of the package hoisting (think of updating all your packages). Lerna itself is using the yarn workspace feature directly (if you use yarn) nowadays and is only a small shell on top of yarn.

--
As said, you encountered problems that are pretty common when you touch an unfamiliar large piece of software. You should not take the easy way to propose a switch from yarn to npm just because of your problems during the setup and your comfort with npm. The time after the setup, the time your work with yarn is the more important part and I guess that's working pretty good if no one else from the contribution team complains about yarn.

I hope you are able to finish your contribution. I'm still looking forward to my own first code contribution as Gatsby is such an awesome piece of software 🙏

[eyalroth]

@georgiee I believe my experience has very little to do with my unfamiliarity with these tools, as the vast majority of my time was invested in troubleshooting unexpected problems with the tools (aka bugs). Other than these problems, most of the process was pretty straight-forward as the guides are pretty good and really there isn't much to it.

I obviously tried cloning only with minimal depth, but that doesn't help much as even the latest revision without the history is pretty big, as you can see in the image I attached (or simply du on your local clone).

Again, I have very little experience with both Yarn and NPM alike, so I cannot really say which gives a better user experience; however, having the repository setup in "less than a day" is not an achievement in my opinion, and I believe it will further alienate contributors from this repository over time. I would at least expect a warning in the documentation about the potential difficulties in setting up a local clone.

[wardpeet]

Thank you for bringing this up. I'm closing this as a duplicate of #13699

[eyalroth]

@wardpeet What about Yarn? Reducing the repo size won't solve the cumbersomeness of the setup process, which is caused mostly due to bugs in Yarn.

[Muizuddeen]

{
"_from": "gatsby-cli",
"_id": "gatsby-cli@2.8.21",
"_inBundle": false,
"_integrity": "sha512-lRf9xm9qa0Zzp6N/sfPXot01WLm6ABjREeKQgZ8pxNYngVZFtR3N51/+NIEY4VMEGvaALBanhAD5MiOcPYoiOQ==",
"_location": "/gatsby-cli",
"_phantomChildren": {},
"_requested": {
"type": "tag",
"registry": true,
"raw": "gatsby-cli",
"name": "gatsby-cli",
"escapedName": "gatsby-cli",
"rawSpec": "",
"saveSpec": null,
"fetchSpec": "latest"
},
"_requiredBy": [
"#USER",
"/"
],
"_resolved": "https://registry.npmjs.org/gatsby-cli/-/gatsby-cli-2.8.21.tgz",
"_shasum": "456277538b983545be5dc2ce036616f5a5b43fdf",
"_spec": "gatsby-cli",
"_where": "C:\Users\user",
"author": {
"name": "Kyle Mathews",
"email": "mathews.kyle@gmail.com"
},
"bin": {
"gatsby": "lib/index.js"
},
"bugs": {
"url": "https://github.com/gatsbyjs/gatsby/issues"
},
"bundleDependencies": false,
"dependencies": {
"@babel/code-frame": "^7.5.5",
"@babel/runtime": "^7.7.6",
"@hapi/joi": "^15.1.1",
"better-opn": "^1.0.0",
"bluebird": "^3.7.2",
"chalk": "^2.4.2",
"clipboardy": "^2.1.0",
"common-tags": "^1.8.0",
"configstore": "^5.0.0",
"convert-hrtime": "^3.0.0",
"core-js": "^2.6.11",
"envinfo": "^7.5.0",
"execa": "^3.4.0",
"fs-exists-cached": "^1.0.0",
"fs-extra": "^8.1.0",
"gatsby-core-utils": "^1.0.25",
"gatsby-telemetry": "^1.1.46",
"hosted-git-info": "^3.0.2",
"ink": "^2.6.0",
"ink-spinner": "^3.0.1",
"is-valid-path": "^0.1.1",
"lodash": "^4.17.15",
"meant": "^1.0.1",
"node-fetch": "^2.6.0",
"object.entries": "^1.1.0",
"opentracing": "^0.14.4",
"pretty-error": "^2.1.1",
"progress": "^2.0.3",
"prompts": "^2.3.0",
"react": "^16.12.0",
"redux": "^4.0.4",
"resolve-cwd": "^2.0.0",
"semver": "^6.3.0",
"signal-exit": "^3.0.2",
"source-map": "0.7.3",
"stack-trace": "^0.0.10",
"strip-ansi": "^5.2.0",
"update-notifier": "^3.0.1",
"uuid": "3.3.3",
"yargs": "^12.0.5",
"yurnalist": "^1.1.1"
},
"deprecated": false,
"description": "Gatsby command-line interface for creating new sites and running Gatsby commands",
"devDependencies": {
"@babel/cli": "^7.7.5",
"@babel/core": "^7.7.5",
"babel-preset-gatsby-package": "^0.2.14",
"cross-env": "^5.2.1"
},
"files": [
"lib",
"scripts"
],
"gitHead": "f5f0c9b91d914cc86f9ce483e062357ed7ebbb7d",
"homepage": "https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby-cli#readme",
"keywords": [
"gatsby"
],
"license": "MIT",
"main": "lib/index.js",
"name": "gatsby-cli",
"optionalDependencies": {
"ink": "^2.6.0",
"ink-spinner": "^3.0.1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/gatsbyjs/gatsby.git",
"directory": "packages/gatsby-cli"
},
"scripts": {
"build": "babel src --out-dir lib --ignore "/tests" --extensions ".ts,.js"",
"postinstall": "node scripts/postinstall.js",
"prepare": "cross-env NODE_ENV=production npm run build",
"watch": "babel -w src --out-dir lib --ignore "/tests" --extensions ".ts,.js""
},
"version": "2.8.21",
"yargs": {
"boolean-negation": false
}
}

**Microsoft Windows [Version 10.0.16299.371]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\Users\user>gatsby -v
Gatsby CLI version: 2.8.21

C:\Users\user>npm install gatsby-cli
^C^CTerminate batch job (Y/N)? Y

C:\Users\user>npm cache clean --force
npm WARN using --force I sure hope you know what you are doing.

C:\Users\user>npm install -g gatsby-cli
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
Terminate batch job (Y/N)? Y

C:\Users\user>core-js -v
'core-js' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\user>npm install -g core-js

core-js@3.6.0 postinstall C:\Users\user\AppData\Roaming\npm\node_modules\core-js
node -e "try{require('./postinstall')}catch(e){}"

Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!

The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:

https://opencollective.com/core-js
https://www.patreon.com/zloirock

Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)

• core-js@3.6.0
  added 1 package in 6.944s

C:\Users\user>core-js -v
'core-js' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\user>npm install -g gatsby-cli
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
C:\Users\user\AppData\Roaming\npm\gatsby -> C:\Users\user\AppData\Roaming\npm\node_modules\gatsby-cli\lib\index.js

gatsby-cli@2.8.21 postinstall C:\Users\user\AppData\Roaming\npm\node_modules\gatsby-cli
node scripts/postinstall.js

Success!

Welcome to the Gatsby CLI! Please visit https://www.gatsbyjs.org/docs/gatsby-cli/ for more information.

Usage: gatsby [options]

Commands:
gatsby develop Start development server. Watches files, rebuilds, and hot reloads if something changes
gatsby build Build a Gatsby project.
gatsby serve Serve previously built Gatsby site.
gatsby info Get environment information for debugging and issue reporting
gatsby clean Wipe the local gatsby environment including built assets and cache
gatsby repl Get a node repl with context of Gatsby environment, see (https://www.gatsbyjs.org/docs/gatsby-repl/)
gatsby new [rootPath] [starter] Create new Gatsby project.
gatsby plugin Useful commands relating to Gatsby plugins
gatsby telemetry Enable or disable Gatsby anonymous analytics collection.

Options:
--verbose Turn on verbose output [boolean] [default: false]
--no-color, --no-colors Turn off the color in output [boolean] [default: false]
--json Turn on the JSON logger [boolean] [default: false]
-h, --help Show help [boolean]
-v, --version Show the version of the Gatsby CLI and the Gatsby package in the current project [boolean]
npm WARN ink@2.6.0 requires a peer of @types/react@>=16.8.0 but none is installed. You must install peer dependencies yourself.

• gatsby-cli@2.8.21
  updated 2 packages in 372.649s

C:\Users\user>cd desktop

C:\Users\user\Desktop>gatsby new gatsby-site
info Creating new site from git: https://github.com/gatsbyjs/gatsby-starter-default.git
Cloning into 'gatsby-site'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 28 (delta 2), reused 17 (delta 2), pack-reused 0
Unpacking objects: 100% (28/28), done.
success Created starter directory layout
info Installing packages...
npm WARN tarball tarball data for terser-webpack-plugin@1.4.3 (sha512-QMxecFz/gHQwteWwSo5nTc6UaICqN1bMedC5sMtUc7y3Ha3Q8y6ZO0iCR8pq4RJC8Hjf0FEPEHZqcMB/+DFCrA==) seems to be corrupted. Trying one more time.
Terminate batch job (Y/N)? Y

C:\Users\user\Desktop>**

how can i solve this problem?