feat: rebuild native dependencies for any project structure

BREAKING CHANGE: rebuild native dependencies for any project structure

README.md

@@ -2,7 +2,7 @@
 A complete solution to package and build a ready for distribution Electron app for MacOS, Windows and Linux with “auto update” support out of the box.
 
 * NPM packages management:
-  * [Native application dependencies](http://electron.atom.io/docs/latest/tutorial/using-native-node-modules/) compilation (only if the [two-package.json project structure](#two-packagejson-structure) is used).
+  * [Native application dependencies](http://electron.atom.io/docs/latest/tutorial/using-native-node-modules/) compilation.
   * Development dependencies are never included. You don't need to ignore them explicitly.
 * [Code Signing](https://github.com/electron-userland/electron-builder/wiki/Code-Signing) on a CI server or development machine.
 * [Auto Update](#auto-update) ready application packaging.
@@ -12,6 +12,7 @@ A complete solution to package and build a ready for distribution Electron app f
   * [MacOS](https://github.com/electron-userland/electron-builder/wiki/Options#MacOptions-target): `dmg`, `pkg`, `mas`.
   * [Linux](https://github.com/electron-userland/electron-builder/wiki/Options#LinuxBuildOptions-target): `AppImage`, `deb`, `rpm`, `freebsd`, `pacman`, `p5p`, `apk`.
   * [Windows](https://github.com/electron-userland/electron-builder/wiki/Options#WinBuildOptions-target): NSIS, Squirrel.Windows.
+* [Two package.json Structure](https://github.com/electron-userland/electron-builder/wiki/Two-package.json-Structure) is supported, but you are not forced to use it even if you have native production dependencies.  
 * [Publishing artifacts](https://github.com/electron-userland/electron-builder/wiki/Publishing-Artifacts) to GitHub Releases and Bintray.
 
 _Note: Platform specific `7zip-bin-*` packages are `optionalDependencies`, which may require manual install if you have npm configured to [not install optional deps by default](https://docs.npmjs.com/misc/config#optional)._
@@ -20,25 +21,6 @@ Real project example — [onshape-desktop-shell](https://github.com/develar/onsh
 
 Consider to use `nsis` target for Windows ([auto-update](https://github.com/electron-userland/electron-builder/issues/529) will be implemented this month) for new projects.
 
-# Two package.json structure
-
-We recommend to use two package.json files (it is not required, you can build your project with any structure).
-
-1. For development (`./package.json`)
-
-   The `package.json` resides in the root of your project. Here you declare the dependencies for your development environment and build scripts (`devDependencies`).
-
-2. For your application (`./app/package.json`)
-
-   The `package.json` resides in the `app` directory. Declare your application dependencies (`depencencies`) here. *Only this directory is distributed with the final, packaged application.*
-
-Why?
-
-1. Native npm modules (those written in C, not JavaScript) need to be compiled and here we have two different compilation targets for them. Those used within the application need to be compiled against the electron runtime and all `devDependencies` need to be compiled against your local node.js environment. Thanks to the two `package.json` structure, this is trivial (see [#39](https://github.com/electron-userland/electron-builder/issues/39)).
-2. No need to specify which [files](https://github.com/electron-userland/electron-builder/wiki/Options#BuildMetadata-files) to include in the app (because development files reside outside the `app` directory).
-
-Please see [Loading App Dependencies Manually](https://github.com/electron-userland/electron-builder/wiki/Loading-App-Dependencies-Manually) and [#379](https://github.com/electron-userland/electron-builder/issues/379#issuecomment-218503881).
-
 # Configuration
 
 See [options](https://github.com/electron-userland/electron-builder/wiki/Options) for a full reference but consider following the simple guide outlined below first.

docker/7/Dockerfile

@@ -1,6 +1,6 @@
 FROM electronuserland/electron-builder:base
 
-ENV NODE_VERSION 7.0.0
+ENV NODE_VERSION 7.1.0
 
 # https://github.com/npm/npm/issues/4531
 RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz" \

docs/Two package.json Structure.md

@@ -0,0 +1,16 @@
+Since version 8 electron-builder rebuilds only production dependencies, so, you are not forced to use two package.json structure.
+
+1. For development (`./package.json`)
+
+   The `package.json` resides in the root of your project. Here you declare the dependencies for your development environment and build scripts (`devDependencies`).
+
+2. For your application (`./app/package.json`)
+
+   The `package.json` resides in the `app` directory. Declare your application dependencies (`depencencies`) here. *Only this directory is distributed with the final, packaged application.*
+
+Why?
+
+1. Native npm modules (those written in C, not JavaScript) need to be compiled and here we have two different compilation targets for them. Those used within the application need to be compiled against the electron runtime and all `devDependencies` need to be compiled against your local node.js environment. Thanks to the two `package.json` structure, this is trivial (see [#39](https://github.com/electron-userland/electron-builder/issues/39)).
+2. No need to specify which [files](https://github.com/electron-userland/electron-builder/wiki/Options#BuildMetadata-files) to include in the app (because development files reside outside the `app` directory).
+
+Please see [Loading App Dependencies Manually](https://github.com/electron-userland/electron-builder/wiki/Loading-App-Dependencies-Manually) and [#379](https://github.com/electron-userland/electron-builder/issues/379#issuecomment-218503881).

package.json

@@ -16,7 +16,7 @@
   },
   "scripts": {
     "compile": "ts-babel . nsis-auto-updater test",
-    "lint": "tslint 'src/**/*.ts' 'test/src/**/*.ts' 'nsis-auto-updater/src/**/*.ts'",
+    "lint": "node ./test/lint.js",
     "pretest": "yarn run compile && yarn run lint",
     "check-deps": "node ./test/out/helpers/checkDeps.js",
     "test": "node ./test/out/helpers/runTests.js",

src/install-app-deps.ts

@@ -7,7 +7,7 @@ import BluebirdPromise from "bluebird-lst-c"
 import { DevMetadata } from "./metadata"
 import yargs from "yargs"
 import { readPackageJson } from "./util/readPackageJson"
-import { installDependencies } from "./yarn"
+import { installDependencies, computeExtraArgs } from "./yarn"
 
 const args: any = yargs
   .option("arch", {
@@ -28,7 +28,7 @@ async function main() {
     throw new Error("install-app-deps is only useful for two package.json structure")
   }
 
-  await installDependencies(results[0], results[1], args.arch, devMetadata.build.npmSkipBuildFromSource !== true)
+  await installDependencies(results[0], results[1], args.arch, computeExtraArgs(devMetadata.build))
 }
 
 main()

src/packager.ts

@@ -1,8 +1,5 @@
 import * as path from "path"
-import {
-  computeDefaultAppDirectory, getElectronVersion, use, exec, isEmptyOrSpaces, exists,
-  asArray
-} from "./util/util"
+import { computeDefaultAppDirectory, getElectronVersion, use, exec, isEmptyOrSpaces, exists } from "./util/util"
 import { all, executeFinally } from "./util/promise"
 import { EventEmitter } from "events"
 import BluebirdPromise from "bluebird-lst-c"
@@ -20,7 +17,7 @@ import { createTargets } from "./targets/targetFactory"
 import { readPackageJson } from "./util/readPackageJson"
 import { TmpDir } from "./util/tmp"
 import { BuildOptions } from "./builder"
-import { getGypEnv, installDependencies, rebuild } from "./yarn"
+import { getGypEnv, installDependencies, rebuild, computeExtraArgs } from "./yarn"
 
 function addHandler(emitter: EventEmitter, event: string, handler: Function) {
   emitter.on(event, handler)
@@ -236,31 +233,22 @@ export class Packager implements BuildInfo {
       })
     }
 
-    if (this.isTwoPackageJsonProjectLayoutUsed) {
-      if (this.devMetadata.build.npmRebuild === false) {
-        log("Skip app dependencies rebuild because npmRebuild is set to false")
-      }
-      else {
-        const forceBuildFromSource = this.devMetadata.build.npmSkipBuildFromSource !== true
-        if (forceBuildFromSource && platform.nodeName !== process.platform) {
-          log("Skip app dependencies rebuild because platform is different")
-        }
-        else {
-          if (await exists(path.join(this.appDir, "node_modules"))) {
-            const args = asArray(this.devMetadata.build.npmArgs)
-            if (forceBuildFromSource) {
-              args.push("--build-from-source")
-            }
-            await rebuild(this.appDir, this.electronVersion, Arch[arch], args)
-          }
-          else {
-            await installDependencies(this.appDir, this.electronVersion, Arch[arch], forceBuildFromSource, this.devMetadata.build.npmArgs)
-          }
-        }
-      }
+    if (this.devMetadata.build.npmRebuild === false) {
+      log("Skip app dependencies rebuild because npmRebuild is set to false")
+      return
+    }
+
+    if (this.devMetadata.build.npmSkipBuildFromSource !== true && platform.nodeName !== process.platform) {
+      log("Skip app dependencies rebuild because platform is different")
     }
     else {
-      log("Skip app dependencies rebuild because dev and app dependencies are not separated")
+      const args = computeExtraArgs(this.devMetadata.build)
+      if (await exists(path.join(this.appDir, "node_modules"))) {
+        await rebuild(this.appDir, this.electronVersion, Arch[arch], args)
+      }
+      else if (this.isTwoPackageJsonProjectLayoutUsed) {
+        await installDependencies(this.appDir, this.electronVersion, Arch[arch], args)
+      }
     }
   }
 }

src/targets/dmg.ts

@@ -4,7 +4,7 @@ import { log, warn } from "../util/log"
 import { PlatformPackager, TargetEx } from "../platformPackager"
 import { MacOptions, DmgOptions, DmgContent } from "../options/macOptions"
 import BluebirdPromise from "bluebird-lst-c"
-import { debug, use, exec, statOrNull, isEmptyOrSpaces, spawn } from "../util/util"
+import { debug, use, exec, statOrNull, isEmptyOrSpaces, spawn, exists } from "../util/util"
 import { copy, unlink, outputFile, remove } from "fs-extra-p"
 import { executeFinally } from "../util/promise"
 import sanitizeFileName from "sanitize-filename"
@@ -54,7 +54,7 @@ export class DmgTarget extends TargetEx {
     ]).concat(tempDmg))
 
     const volumePath = path.join("/Volumes", volumeName)
-    if (await statOrNull(volumePath) != null) {
+    if (await exists(volumePath)) {
       debug("Unmounting previous disk image")
       await detach(volumePath)
     }

src/yarn.ts

@@ -2,10 +2,11 @@ import BluebirdPromise from "bluebird-lst-c"
 import * as path from "path"
 import { task, log } from "./util/log"
 import { homedir } from "os"
-import { spawn, exists } from "./util/util"
+import { spawn, exists, asArray } from "./util/util"
+import { BuildMetadata } from "./metadata"
 
-export function installDependencies(appDir: string, electronVersion: string, arch: string = process.arch, forceBuildFromSource: boolean, additionalArgs?: any): Promise<any> {
-  return task(`Installing app dependencies for arch ${arch} to ${appDir}`, spawnNpmProduction(appDir, forceBuildFromSource, getGypEnv(electronVersion, arch), additionalArgs))
+export function installDependencies(appDir: string, electronVersion: string, arch: string = process.arch, additionalArgs: Array<string>): Promise<any> {
+  return task(`Installing app dependencies for arch ${arch} to ${appDir}`, spawnNpmProduction(appDir, getGypEnv(electronVersion, arch), additionalArgs))
 }
 
 export function getGypEnv(electronVersion: string, arch: string): any {
@@ -20,41 +21,44 @@ export function getGypEnv(electronVersion: string, arch: string): any {
   })
 }
 
-function spawnNpmProduction(appDir: string, forceBuildFromSource: boolean, env?: any, additionalArgs?: any): Promise<any> {
+export function computeExtraArgs(options: BuildMetadata) {
+  const args = asArray(options.npmArgs)
+  if (options.npmSkipBuildFromSource !== true) {
+    args.push("--build-from-source")
+  }
+  return args
+}
+
+function spawnNpmProduction(appDir: string, env: any, additionalArgs: Array<string>): Promise<any> {
   let npmExecPath = process.env.npm_execpath || process.env.NPM_CLI_JS
   const npmExecArgs = ["install", "--production"]
 
-  const isNotYarn = npmExecPath == null || !npmExecPath.includes("yarn")
-  if (isNotYarn) {
+  const isYarn = npmExecPath != null || npmExecPath.includes("yarn")
+  if (!isYarn) {
     if (process.env.NPM_NO_BIN_LINKS === "true") {
       npmExecArgs.push("--no-bin-links")
     }
     npmExecArgs.push("--cache-min", "999999999")
   }
 
   if (npmExecPath == null) {
-    npmExecPath = process.platform === "win32" ? "npm.cmd" : "npm"
+    npmExecPath = getPackageToolPath()
   }
   else {
     npmExecArgs.unshift(npmExecPath)
     npmExecPath = process.env.npm_node_execpath || process.env.NODE_EXE || "node"
   }
-  if (isNotYarn && forceBuildFromSource) {
-    npmExecArgs.push("--build-from-source")
-  }
 
-  if (additionalArgs) {
-    if (Array.isArray(additionalArgs)) {
-      npmExecArgs.push(...additionalArgs)
-    }
-    else {
-      npmExecArgs.push(additionalArgs)
+  for (let a of additionalArgs) {
+    if (!isYarn || a !== "--build-from-source") {
+      npmExecArgs.push(a)
     }
   }
 
+  console.log("AAA " + npmExecPath + " " + npmExecArgs.join(" "))
   return spawn(npmExecPath, npmExecArgs, {
     cwd: appDir,
-    env: env || process.env
+    env: env
   })
 }
 
@@ -89,6 +93,15 @@ function flatDependencies(data: any, result: Set<string>, seen: Set<string>, ext
   }
 }
 
+function getPackageToolPath() {
+  if (process.env.FORCE_YARN === "true") {
+    return process.platform === "win32" ? "yarn.cmd" : "yarn"
+  }
+  else {
+    return process.platform === "win32" ? "npm.cmd" : "npm"
+  }
+}
+
 export async function rebuild(appDir: string, electronVersion: string, arch: string = process.arch, additionalArgs: Array<string>) {
   const deps = new Set<string>()
   await dependencies(appDir, false, deps)
@@ -98,18 +111,13 @@ export async function rebuild(appDir: string, electronVersion: string, arch: str
     return
   }
 
-  log(`Rebuilding native app dependencies for arch ${arch} to ${appDir}`)
+  log(`Rebuilding native production dependencies for arch ${arch}`)
 
   let execPath = process.env.npm_execpath || process.env.NPM_CLI_JS
   const execArgs = ["run", "install", "--"]
 
   if (execPath == null) {
-    if (process.env.FORCE_YARN === "true") {
-      execPath = process.platform === "win32" ? "yarn.cmd" : "yarn"
-    }
-    else {
-      execPath = process.platform === "win32" ? "npm.cmd" : "npm"
-    }
+    execPath = getPackageToolPath()
   }
   else {
     execArgs.unshift(execPath)
@@ -128,10 +136,5 @@ export async function rebuild(appDir: string, electronVersion: string, arch: str
   execArgs.push(`--arch=${arch}`)
   execArgs.push(...additionalArgs)
 
-  for (let dir of nativeDeps) {
-    await spawn(execPath, execArgs, {
-      cwd: dir,
-      env: env
-    })
-  }
+  await BluebirdPromise.each(nativeDeps, it => spawn(execPath, execArgs, {cwd: it, env: env}))
 }

tslint.json → test/lint.js

@@ -1,4 +1,7 @@
-{
+const Linter = require("tslint")
+const path = require("path")
+
+const configuration = {
   "extends": "tslint:recommended",
   "rules": {
     "member-ordering": [
@@ -51,6 +54,22 @@
       "check-type"
     ],
     "no-bitwise": false,
-    "jsdoc-format": false
+    "jsdoc-format": false,
+    "no-for-in-array": true
+  }
+}
+const options = {
+  configuration: configuration,
+}
+
+for (let projectDir of [path.join(__dirname, ".."), path.join(__dirname, "..", "nsis-auto-updater"), __dirname]) {
+  const program = Linter.createProgram("tsconfig.json", projectDir)
+  for (let file of Linter.getFileNames(program)) {
+    const fileContents = program.getSourceFile(file).getFullText()
+    const linter = new Linter(file, fileContents, options, program)
+    const out = linter.lint().output
+    if (out.length > 0) {
+      process.stdout.write(out)
+    }
   }
 }

yarn.lock

@@ -1536,8 +1536,8 @@ dateformat@^1.0.11, dateformat@^1.0.12:
     meow "^3.3.0"
 
 debug@^2.1.1, debug@^2.1.3, debug@^2.2.0, debug@2:
-  version "2.3.0"
-  resolved "https://registry.yarnpkg.com/debug/-/debug-2.3.0.tgz#3912dc55d7167fc3af17d2b85c13f93deaedaa43"
+  version "2.3.2"
+  resolved "https://registry.yarnpkg.com/debug/-/debug-2.3.2.tgz#94cb466ef7d6d2c7e5245cdd6e4104f2d0d70d30"
   dependencies:
     ms "0.7.2"