Skip to content

Commit

Permalink
Change build system to hereby
Browse files Browse the repository at this point in the history
This eliminates a significant number of dependencies, eliminating all
npm audit issues, speeding up `npm ci` by 20%, and overall making the
build faster (faster startup, direct code is faster than streams, etc)
and clearer to understand.

I'm finding it much easier to make build changes for the module
transform with this; I can more clearly indicate task dependencies and
prevent running tasks that don't need to be run.

Given we're changing our build process entirely (new deps, new steps),
it seems like this is a good time to change things up.
  • Loading branch information
jakebailey committed Oct 24, 2022
1 parent d11a9bf commit 57a42fd
Show file tree
Hide file tree
Showing 31 changed files with 3,664 additions and 12,095 deletions.
2 changes: 1 addition & 1 deletion .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
ARG VARIANT="14-buster"
FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}

RUN sudo -u node npm install -g gulp-cli
RUN sudo -u node npm install -g hereby
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Thank you for submitting a pull request!
Please verify that:
* [ ] There is an associated issue in the `Backlog` milestone (**required**)
* [ ] Code is up-to-date with the `main` branch
* [ ] You've successfully run `gulp runtests` locally
* [ ] You've successfully run `hereby runtests` locally
* [ ] There are new or updated unit tests validating the change
Refer to CONTRIBUTING.MD for more details.
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/accept-baselines-fix-lints.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ jobs:
git config user.name "TypeScript Bot"
npm install
git rm -r --quiet tests/baselines/reference :^tests/baselines/reference/docker :^tests/baselines/reference/user
gulp runtests-parallel --ci --fix || true
gulp baseline-accept
npx hereby runtests-parallel --ci --fix || true
npx hereby baseline-accept
git add ./src
git add ./tests/baselines/reference
git diff --cached
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,10 @@ jobs:
run: npm install --no-save --no-package-lock playwright

- name: Build local
run: gulp local
run: npx hereby local

- name: Validate the browser can import TypeScript
run: gulp test-browser-integration
run: npx hereby test-browser-integration

typecheck:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -102,10 +102,10 @@ jobs:
- run: npm ci

- name: Build scripts
run: gulp scripts
run: npx hereby scripts

- name: ESLint tests
run: gulp run-eslint-rules-tests
run: npx hereby run-eslint-rules-tests

self-check:
runs-on: ubuntu-latest
Expand All @@ -119,10 +119,10 @@ jobs:
- run: npm ci

- name: Build tsc
run: gulp tsc
run: npx hereby tsc

- name: Clean
run: gulp clean-src
run: npx hereby clean-src

- name: Self build
run: gulp build-src --built
run: npx hereby build-src --built
2 changes: 1 addition & 1 deletion .github/workflows/new-release-branch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
sed -i -e 's/const versionMajorMinor = ".*"/const versionMajorMinor = "${{ github.event.client_payload.core_major_minor }}"/g' tests/baselines/reference/api/tsserverlibrary.d.ts
sed -i -e 's/const version\(: string\)\{0,1\} = `${versionMajorMinor}.0-.*`/const version = `${versionMajorMinor}.0-${{ github.event.client_payload.core_tag || 'dev' }}`/g' src/compiler/corePublic.ts
npm ci
gulp LKG
npx hereby LKG
npm test
git diff
git add package.json
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/nightly.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ jobs:
run: |
npm whoami
npm ci
gulp configure-nightly
gulp LKG
gulp runtests-parallel
gulp clean
npx hereby configure-nightly
npx hereby LKG
npx hereby runtests-parallel
npx hereby clean
npm publish --tag next
env:
NODE_AUTH_TOKEN: ${{secrets.npm_token}}
6 changes: 3 additions & 3 deletions .github/workflows/release-branch-artifact.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ jobs:
- name: Adding playwright
run: npm install --no-save --no-package-lock playwright
- name: Validate the browser can import TypeScript
run: gulp test-browser-integration
run: npx hereby test-browser-integration
- name: LKG, clean, and pack
run: |
gulp LKG
gulp clean
npx hereby LKG
npx hereby clean
npm pack ./
mv typescript-*.tgz typescript.tgz
- name: Upload built tarfile
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/set-version.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
sed -i -e 's/const versionMajorMinor = ".*"/const versionMajorMinor = "${{ github.event.client_payload.core_major_minor }}"/g' tests/baselines/reference/api/tsserverlibrary.d.ts
sed -i -e 's/const version\(: string\)\{0,1\} = .*;/const version = "${{ github.event.client_payload.package_version }}" as string;/g' src/compiler/corePublic.ts
npm ci
gulp LKG
npx hereby LKG
npm test
git diff
git add package.json
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/update-lkg.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
git config user.email "typescriptbot@microsoft.com"
git config user.name "TypeScript Bot"
npm ci
gulp LKG
npx hereby LKG
npm test
git diff
git add ./lib
Expand Down
18 changes: 18 additions & 0 deletions .gulp.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
const cp = require("child_process");
const path = require("path");
const chalk = require("chalk");

const argv = process.argv.slice(2);

if (!argv.includes("--tasks-simple")) {
console.error(chalk.red("Warning: using gulp shim; please run hereby directly."));
}

const args = [
...process.execArgv,
path.join(__dirname, "node_modules", ".bin", "hereby"),
...argv,
];

const { status } = cp.spawnSync(process.execPath, args, { stdio: "inherit" });
process.exit(status ?? 1);
7 changes: 3 additions & 4 deletions .vscode/launch.template.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
"configurations": [
{
"type": "node",
"protocol": "inspector",
"request": "launch",
"name": "Mocha Tests (currently opened test)",
"runtimeArgs": ["--nolazy"],
Expand All @@ -43,20 +42,20 @@
},
"sourceMaps": true,
"smartStep": true,
"preLaunchTask": "gulp: tests",
"preLaunchTask": "npm: build:tests",
"console": "integratedTerminal",
"outFiles": [
"${workspaceRoot}/built/local/run.js"
],
"customDescriptionGenerator": "'__tsDebuggerDisplay' in this ? this.__tsDebuggerDisplay(defaultValue) : defaultValue",
"customDescriptionGenerator": "'__tsDebuggerDisplay' in this ? this.__tsDebuggerDisplay(defaultValue) : defaultValue"
},
{
// See: https://github.com/microsoft/TypeScript/wiki/Debugging-Language-Service-in-VS-Code
"type": "node",
"request": "attach",
"name": "Attach to VS Code TS Server via Port",
"processId": "${command:PickProcess}",
"customDescriptionGenerator": "'__tsDebuggerDisplay' in this ? this.__tsDebuggerDisplay(defaultValue) : defaultValue",
"customDescriptionGenerator": "'__tsDebuggerDisplay' in this ? this.__tsDebuggerDisplay(defaultValue) : defaultValue"
}
]
}
8 changes: 8 additions & 0 deletions .vscode/settings.template.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,12 @@
// "--ignore-revs-file",
// ".git-blame-ignore-revs"
// ]

// These options search the repo recursively and slow down
// the build task menu. We define our own in tasks.json.
"typescript.tsc.autoDetect": "off",
"npm.autoDetect": "off",
"grunt.autoDetect": "off",
"jake.autoDetect": "off",
"gulp.autoDetect": "off"
}
40 changes: 20 additions & 20 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,42 +4,42 @@
"version": "2.0.0",
"tasks": [
{
"type": "gulp",
"label": "gulp: local",
"task": "local",
"group": {
"kind": "build",
"isDefault": true
},
"label": "tsc --build ./src --watch",
"type": "shell",
"command": "node",
"args": ["${workspaceFolder}/lib/tsc.js", "--build", "${workspaceFolder}/src", "--watch"],
"group": "build",
"isBackground": true,
"problemMatcher": [
"$tsc"
"$tsc-watch"
]
},
{
"type": "gulp",
"label": "gulp: tsc",
"task": "tsc",
"label": "npm: build:compiler",
"type": "npm",
"script": "build:compiler",
"group": "build",
"problemMatcher": [
"$tsc"
]
},
{
"type": "gulp",
"label": "gulp: tests",
"task": "tests",
"label": "npm: build:tests",
"type": "npm",
"script": "build:tests",
"group": "build",
"problemMatcher": [
"$tsc"
]
},
{
"type": "gulp",
"task": "services",
"label": "gulp: services",
"label": "gulp: tests", // Kept for backwards compat for old launch.json files.
"type": "npm",
"script": "build:tests",
"group": "build",
"problemMatcher": [
"$tsc"
],
}
]
},
]
}
}
66 changes: 33 additions & 33 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,42 +54,42 @@ In general, things we find useful when reviewing suggestions are:
2. A copy of the TypeScript code. See the next steps for instructions.
3. [Node](https://nodejs.org), which runs JavaScript locally. Current or LTS will both work.
4. An editor. [VS Code](https://code.visualstudio.com) is the best place to start for TypeScript.
5. The gulp command line tool, for building and testing changes. See the next steps for how to install it.
5. The hereby command line tool, for building and testing changes. See the next steps for how to install it.

## Get Started

1. Install node using the version you downloaded from [nodejs.org](https://nodejs.org).
2. Open a terminal.
3. Make a fork—your own copy—of TypeScript on your GitHub account, then make a clone—a local copy—on your computer. ([Here are some step-by-step instructions](https://github.com/anitab-org/mentorship-android/wiki/Fork%2C-Clone-%26-Remote)). Add `--depth=1` to the end of the `git clone` command to save time.
4. Install the gulp command line tool: `npm install -g gulp-cli`
4. Install the hereby command line tool: `npm install -g hereby`
5. Change to the TypeScript folder you made: `cd TypeScript`
6. Install dependencies: `npm ci`
7. Make sure everything builds and tests pass: `gulp runtests-parallel`
7. Make sure everything builds and tests pass: `hereby runtests-parallel`
8. Open the TypeScript folder in your editor.
9. Follow the directions below to add and debug a test.

## Helpful tasks

Running `gulp --tasks --depth 1` provides the full listing, but here are a few common tasks you might use.
Running `hereby --tasks` provides the full listing, but here are a few common tasks you might use.

```
gulp local # Build the compiler into built/local.
gulp clean # Delete the built compiler.
gulp LKG # Replace the last known good with the built one.
# Bootstrapping step to be executed when the built compiler reaches a stable state.
gulp tests # Build the test infrastructure using the built compiler.
gulp runtests # Run tests using the built compiler and test infrastructure.
# You can override the specific suite runner used or specify a test for this command.
# Use --tests=<testPath> for a specific test and/or --runner=<runnerName> for a specific suite.
# Valid runners include conformance, compiler, fourslash, project, user, and docker
# The user and docker runners are extended test suite runners - the user runner
# works on disk in the tests/cases/user directory, while the docker runner works in containers.
# You'll need to have the docker executable in your system path for the docker runner to work.
gulp runtests-parallel # Like runtests, but split across multiple threads. Uses a number of threads equal to the system
# core count by default. Use --workers=<number> to adjust this.
gulp baseline-accept # This replaces the baseline test results with the results obtained from gulp runtests.
gulp lint # Runs eslint on the TypeScript source.
gulp help # List the above commands.
hereby local # Build the compiler into built/local.
hereby clean # Delete the built compiler.
hereby LKG # Replace the last known good with the built one.
# Bootstrapping step to be executed when the built compiler reaches a stable state.
hereby tests # Build the test infrastructure using the built compiler.
hereby runtests # Run tests using the built compiler and test infrastructure.
# You can override the specific suite runner used or specify a test for this command.
# Use --tests=<testPath> for a specific test and/or --runner=<runnerName> for a specific suite.
# Valid runners include conformance, compiler, fourslash, project, user, and docker
# The user and docker runners are extended test suite runners - the user runner
# works on disk in the tests/cases/user directory, while the docker runner works in containers.
# You'll need to have the docker executable in your system path for the docker runner to work.
hereby runtests-parallel # Like runtests, but split across multiple threads. Uses a number of threads equal to the system
# core count by default. Use --workers=<number> to adjust this.
hereby baseline-accept # This replaces the baseline test results with the results obtained from hereby runtests.
hereby lint # Runs eslint on the TypeScript source.
hereby help # List the above commands.
```

## Tips
Expand All @@ -111,7 +111,7 @@ You might need to run `git config --global core.longpaths true` before cloning T

### Using local builds

Run `gulp` to build a version of the compiler/language service that reflects changes you've made. You can then run `node <repo-root>/built/local/tsc.js` in place of `tsc` in your project. For example, to run `tsc --watch` from within the root of the repository on a file called `test.ts`, you can run `node ./built/local/tsc.js --watch test.ts`.
Run `hereby` to build a version of the compiler/language service that reflects changes you've made. You can then run `node <repo-root>/built/local/tsc.js` in place of `tsc` in your project. For example, to run `tsc --watch` from within the root of the repository on a file called `test.ts`, you can run `node ./built/local/tsc.js --watch test.ts`.

## Contributing bug fixes

Expand Down Expand Up @@ -163,7 +163,7 @@ Any changes should be made to [src/lib](https://github.com/Microsoft/TypeScript/
Library files in `built/local/` are updated automatically by running the standard build task:

```sh
gulp
hereby
```

The files in `lib/` are used to bootstrap compilation and usually **should not** be updated unless publishing a new version or updating the LKG.
Expand All @@ -178,36 +178,36 @@ If you need a head start understanding how the compiler works, or how the code i

## Running the Tests

To run all tests, invoke the `runtests-parallel` target using gulp:
To run all tests, invoke the `runtests-parallel` target using hereby:

```Shell
gulp runtests-parallel
hereby runtests-parallel
```

This will run all tests; to run only a specific subset of tests, use:

```Shell
gulp runtests --tests=<regex>
hereby runtests --tests=<regex>
```

e.g. to run all compiler baseline tests:

```Shell
gulp runtests --tests=compiler
hereby runtests --tests=compiler
```

or to run a specific test: `tests\cases\compiler\2dArrays.ts`

```Shell
gulp runtests --tests=2dArrays
hereby runtests --tests=2dArrays
```

## Debugging the tests

You can debug with VS Code or Node instead with `gulp runtests -i`:
You can debug with VS Code or Node instead with `hereby runtests -i`:

```Shell
gulp runtests --tests=2dArrays -i
hereby runtests --tests=2dArrays -i
```

You can also use the [provided VS Code launch configuration](./.vscode/launch.template.json) to launch a debug session for an open test file. Rename the file 'launch.json', open the test file of interest, and launch the debugger from the debug panel (or press F5).
Expand Down Expand Up @@ -257,12 +257,12 @@ When a change in the baselines is detected, the test will fail. To inspect chang
git diff --diff-filter=AM --no-index ./tests/baselines/reference ./tests/baselines/local
```

Alternatively, you can set the `DIFF` environment variable and run `gulp diff`, or manually run your favorite folder diffing tool between `tests/baselines/reference` and `tests/baselines/local`. Our team largely uses Beyond Compare and WinMerge.
Alternatively, you can set the `DIFF` environment variable and run `hereby diff`, or manually run your favorite folder diffing tool between `tests/baselines/reference` and `tests/baselines/local`. Our team largely uses Beyond Compare and WinMerge.

After verifying that the changes in the baselines are correct, run

```Shell
gulp baseline-accept
hereby baseline-accept
```

This will change the files in `tests\baselines\reference`, which should be included as part of your commit.
Expand All @@ -271,6 +271,6 @@ Be sure to validate the changes carefully -- apparently unrelated changes to bas
## Localization

All strings the user may see are stored in [`diagnosticMessages.json`](./src/compiler/diagnosticMessages.json).
If you make changes to it, run `gulp generate-diagnostics` to push them to the `Diagnostic` interface in `diagnosticInformationMap.generated.ts`.
If you make changes to it, run `hereby generate-diagnostics` to push them to the `Diagnostic` interface in `diagnosticInformationMap.generated.ts`.

See [coding guidelines on diagnostic messages](https://github.com/Microsoft/TypeScript/wiki/Coding-guidelines#diagnostic-messages).
Loading

0 comments on commit 57a42fd

Please sign in to comment.