Framework: use npm scripts instead of make for build system

[Pomax]

Steps to reproduce

1. Starting at URL: https://github.com/Automattic/wp-calypso
2. this is clearly an npm-managed project, with a package.json and even a webpack build
3. installation instructions rely on "make"
4. everything make does can be done with npm scripts instead

What I expected

1. git clone
2. npm install
3. npm start

Like any npm-managed project. Nice and OS agnostic, guaranteed to work on unix, linus, OSX, windows, and pretty much anything else that lets you install node and npm, with all bootstrapping managed via npm scripts triggered by the npm "postinstall" hook.

What happened instead

The docs tell me to use make, something that is notoriously unreliable and absolutely not usable cross-platform. To make matters worse, a windows user is doubly-impacted, because they are also instructed to install MSYS2, a tool chain that breaks when there are spaces in paths, but also tell them to use the windows git client which installs in the "program files (x86)" folder, with spaces. The instructions are literally set up such that windows users can't install Calypso unless they go out on their own and figure out how to get "make" to work.

This is super-silly, just turn that makefile into a set of npm scripts instead. If you've not heard of npm scripts, or you only have passing knowledge of them: npm scripts are scripts that run as CLI commands, just like MAKE instructions, except they run with execution access to any and all CLI utilities installed in the node_modules dir, which means that instead of unix-only commands like rm or cp, you can use universal packages like rimraf or cp for node and be guaranteed that no additional tool chains are needed: things just work. (Need environment variables? don't use ENV on the command line, use an .env file and habitat)

This will work, for instance:

  "scripts": {
    ...
    "postinstall": "npm run bootstrap",
    "bootstrap": "rimraf build && mkdirp build/assets",
    "start": "npm-run --parallel 'npm run webpack:dev' 'node build/bundle-dev.js'",
    "webpack:dev": "webpack-dev-server --hot --inline --progress --colors dev.config.js"
    "webpack:prod": "webpack --progress --colors",
    ...
  }

It's 2016, we have a universally supported system installed already (node+npm), there is no need for really-only-unixy-systems make anymore. Everyone wins.

[rralian]

@Pomax Thank you for the issue. We are aware of the additional complexity of running wp-calypso on windows. It has been pointed out in a number of other issues (#592, #1193, #2029, #2066). There's some discussion in #1193 about why we had originally chosen make for our build system in that issue. The choice is a bit of an artifact from two years ago when calypso was still an experimental internal project with just a handful of developers.

I think you're right that npm scripts would be an improvement. I will change the title of your issue to more simply state your request. We don't have anyone actively working on this right now, but of course pull requests are welcome if you're eager.

In the meantime, I would suggest you take a look at #2124 where we have put some effort into making our current Makefile system installable and runnable on windows. If the instructions in windows.md do not work for you, you could comment on that PR or open a new issue and ping @klimeryk with questions. Or you may want to try using our wp-calypso-bootstrap project which uses vagrant and virtualbox to create a portable development environment for calypso.

[Pomax]

Unfortunately those instructions only work by violating the basic concept of "where programs install" on windows, by telling people to install things on the drive root: something that UAC normally forbids (the system drive root is a protected location, programs go in "program files" and "program files (x86)").

I'll have a look at how much work it is to turn your makefile into npm scripts, because make was finally on its way out 2 years ago, and seeing it still in use for a modern Node-based project is really sadmaking, especially since node+npm are unquestionably properly-cross platform, where make is horribly deficient in that respect =)

(also, I'm using a VM solution at the moment. Calypso is rather intensive, and even on an i7 4790 with 32GB of ram it runs quite slowly. It's a solution, but not one to be used for too long)

[folletto]

For reference, I'd also add #1345 and #100 (which is a personal issue 😜 ) to the issues that might disappear if we stop using make. :)

[klimeryk]

@Pomax thank you for your comments and suggestions. I completely agree that make is not the perfect build tool and it has it's quirks and drawbacks. But it's definitely not "on its way out" - it's a de facto standard for building many *nix programs.

node+npm are unquestionably properly-cross platform

I wouldn't call a project "unquestionably properly cross-platform" that has a huuuuuuuge issue titled "Windows users are not happy" for node-gyp, node itself has problems with quotes on Windows, etc. I don't want to sound defensive - I just want to you to understand that no build tool is perfect. We know the quirks and drawbacks of ours. Substituting it all for npm scripts might simplify things for Windows users in the long run, but could introduce new bugs, regressions, etc. It's hard to properly test a new build framework. So we've tried our best to support Windows users, as @rralian mentioned.

Also, please don't lump every problem on make - issues with running Calypso from folders with spaces are bugs in our scripts, not something inherently broken in make. Bugs like this can be just as easily introduced in npm scripts: case in point from an ongoing PR. As a side note, it seems that Calypso is building fine in such folders now 😄 So thanks for prompting me to test that again 😁

[folletto]

Also, please don't lump every problem on make - issues with running Calypso from folders with spaces are bugs in our scripts, not something inherently broken in make.

LOL sorry, I tried to write it in a way to make clear that I was cross-linking for reference, not to say that it's a make problem per se. Clearly I failed in that. :D

[klimeryk]

@folletto - no problem, I wasn't actually addressing that at you 😃 It's just that @Pomax's comments suggest that every problem we have with regards to our build would be solved by switching to npm scripts - which I believe is not true. I used the "folders with spaces" argument just as an example of blaming everything on make.

[Pomax]

I'm not blaming everything on make, it's indeed still used a lot for compling unix/linux programs, but for projects that don't need compilation, it was definitely on the way out. Since this seems a predominantly node codebase, a lot of the issues around getting calypso installed on windows seemed to be because make was still being used even though a node codebase doesn't really need it (it's already more cross-platform and with npm has a powerful make/autoconf equivalent).

If there are other places in the code itself that can't actually run when there are spaces in paths, then that is unfortunate, and hopefully all of those places are known and files as issues to tackle in the future. I'm going to see if I can find some time to turn as much of the make script into npm management instead, and if that works I'll be happy to file a PR.

[stephanethomas]

(also, I'm using a VM solution at the moment. Calypso is rather intensive, and even on an i7 4790 with 32GB of ram it runs quite slowly. It's a solution, but not one to be used for too long)

Just wanted to mention that it might be caused by synced folders. We moved away from using VirtualBox shared folders syncing mechanism in Calypso Bootstrap because it was killing performance.

[Pomax]

Unfortunately, I'm already using VMware rather than VirtualBox for that very reason (there's a longstanding bug around syncing that breaks synced folders for windows users because Oracle's not updated its win calls to use unicode paths, so you're stuck with 255 instead of 32000+ characters for a path. Which is pretty disastrous for OSS given that a lot of potential contributors tend to be on windows machines)

[Pomax]

I've filed a PR with refactors that moves some things from the makefile into package.json as npm scripts, such that a make run for windows users (as well as not-bash-compatible shell users?) now gets them all the way to http://localhost:3000 without errors (that I can tell, at least)

[pdfernhout]

@Pomax Perhaps ShellJS could help with this?

From: https://github.com/shelljs/shelljs
"ShellJS is a portable (Windows/Linux/OS X) implementation of Unix shell commands on top of the Node.js API. You can use it to eliminate your shell script's dependency on Unix while still keeping its familiar and powerful commands. You can also install it globally so you can run it from outside Node projects - say goodbye to those gnarly Bash scripts! ...
A convenience script shelljs/make is also provided to mimic the behavior of a Unix Makefile. In this case all shell objects are global, and command line arguments will cause the script to execute only the corresponding function in the global target object. To avoid redundant calls, target functions are executed only once per script."

See also: http://blog.npmjs.org/post/118810260230/building-a-simple-command-line-tool-with-npm
"npm run scripts are convenient because you can simply use the commands that you use on the command line. Running those scripts inside a reusable module can be just as convenient, and we want it to be convenient… otherwise our teammates won’t make their scripts reusable.
We’ll be using the shelljs module to do this. You could also use Node’s child_process for this, but one nice thing about shelljs is that it makes it possible for you to use a lot of Unix commands in Windows. See the docs for a full list of supported commands."

[Pomax]

feel free to also file a PR that uses ShellJS instead. In the mean time I've filed one that works quite well for windows users, without adversely affecting 'nix users, while also setting up a path to move away from the makefile approach in small steps, for universal cross-platform support at some future point in time.

[rralian]

@Pomax thank you very much for submitting the PR. I'm not sure I'll get a chance to look today myself, but I'll try to get some eyes on it.

[gziolo]

With #4596 we fully migrated our test runner to npm scripts. You can run tests nows with npm test :)

[Pomax]

oh nice, great job!

[ockham]

Fixed by #13536.

[ockham]

FYI, PR to remove Makefile: #17033