Prepare bazel integration tests #17304

filipesilva · 2020-03-25T13:45:09Z

Groundwork for #16976

clydin · 2020-03-25T18:19:54Z

packages/angular_devkit/core/src/workspace/json/reader_spec.ts

@@ -28,7 +28,7 @@ const basicFile = stripIndent`
  "x-bar": 5,
 }`;

-const representativeFile = readFileSync(__dirname + '/test/angular.json', 'utf8');
+const representativeFile = readFileSync(require.resolve(__dirname + '/test/angular.json'), 'utf8');


__dirname is an absolute path. require.resolve should be a noop in this case. Can you provide some background on what's happening here on windows?

In general, any sort of node file lookup on Bazel should be subject to require.resolve. This is how rules_nodejs resolves paths using the Bazel runfiles mechanism, where a given Bazel target only has access to outputs from its dependencies.

In practice, this does not make a lot of difference on Linux. A symlink forest is laid down where the target is going to actually run, and mostly the files are resolved correctly whether you use require.resolve or not because the files are there.

On windows though, that's a stricter. Bazel does not lay down a symlink forest on windows by default. If you don't use require.resolve, it's still possible to correctly resolve some files, like outputs from other rules. But other files, like node modules dependencies and data files, need to be looked up in the runfiles.

Since the requirement is quite lax on Linux but quite strict on windows, what ends up happening is that lack of require.resolve calls go unnoticed until someone tries to run things on Windows. Then it breaks.

Ah. ok. so this is really a workaround for Bazel Windows support?

Since this is not needed on any OS outside of Bazel nor with Bazel on linux/Mac, might be good to add a comment so someone reading through knows why it's there and doesn't try to remove it unintentionally.

@filipesilva, this doesn't seem correct for Windows either. Under Bazel to resolve files from the manifest require.resolve need to contain a full path including the workspace name, similar to rlocation.

Example;

require.resolve('workspace_name/tsconfig.json');

Will resolve the file from manifest to an absolute path:

C:/git/workspace_name/tsconfig.json

In the above case, the path provided will never be in the MANIFEST, because Bazel paths/keys are never absolute.

MANIFEST extract

angular_bazel_architect/src/polyfills.ts C:/git/rules_nodejs/examples/angular_bazel_architect/src/polyfills.ts angular_bazel_architect/src/styles.scss C:/git/rules_nodejs/examples/angular_bazel_architect/src/styles.scss angular_bazel_architect/tsconfig.app.json C:/git/rules_nodejs/examples/angular_bazel_architect/tsconfig.app.json angular_bazel_architect/tsconfig.json C:/git/rules_nodejs/examples/angular_bazel_architect/tsconfig.json

That's a good point. I wonder why this worked then... I assumed the paths were in the manifest, but didn't verify.

I know rules_nodejs' require.resolve has fallbacks. The absolute path must hit the fallback then? But then this still gets confusing because the data files can only be looked up by runfiles...

Went looking at the require patches in rules_nodejs and found this:
https://github.com/bazelbuild/rules_nodejs/blob/cd8520d870abe490d7a154f7e1d9ccac32ea04d8/internal/node/require_patch.js#L141-L148

// Determine bin and gen root to convert absolute paths into runfile paths. const binRootIdx = manifestPath.indexOf(BIN_DIR); let binRoot, genRoot; if (binRootIdx !== -1) { const execRoot = manifestPath.slice(0, binRootIdx); binRoot = `${execRoot}${BIN_DIR}/`; genRoot = `${execRoot}${GEN_DIR}/`; }

So that makes sense now: the absolute paths are converted.

Thanks for checking @filipesilva.

Here's where the conversion happens:
https://github.com/bazelbuild/rules_nodejs/blob/cd8520d870abe490d7a154f7e1d9ccac32ea04d8/internal/node/require_patch.js#L307-L315

runfilesEntry.startsWith(binRoot) || runfilesEntry.startsWith(genRoot) || runfilesEntry.startsWith(localWorkspacePath)) { // For absolute paths, replace binRoot, genRoot or localWorkspacePath with // USER_WORKSPACE_NAME to enable lookups. // It's OK to do multiple replacements because all of these are absolute paths with drive // names (e.g. C:\), and on Windows you can't have drive names in the middle of paths. runfilesEntry = runfilesEntry.replace(binRoot, `${USER_WORKSPACE_NAME}/`) .replace(genRoot, `${USER_WORKSPACE_NAME}/`) .replace(localWorkspacePath, `${USER_WORKSPACE_NAME}/`);

@clydin I'm a bit wary of adding a comment here specifically, because this is just standard practice on rules_nodejs. I agree it would be helpful to have that information somewhere though.

I added it to docs/process/bazel.md. We already had some general bazel information there. Does that sound good to you?

packages/angular_devkit/architect/node/node-modules-architect-host.ts

packages/ngtools/webpack/BUILD

clydin · 2020-03-25T19:00:05Z

package.json

-    "verdaccio": "4.5.1"
+    "verdaccio": "4.5.1",
+    "webpack": "4.42.0",
+    "webpack-sources": "1.4.3"


Having to put all the dependencies in two places is really awkward and a potential maintenance burden. Hopefully this can be rectified in the future.

At its core, this is needed because rules_nodejs doesn't understand yarn workspaces. It's not super clear to me whether it should though, or how yarn pnp plays into that either.

We could reduce this burden with some health checks on our repository, like checking that these versions are the same everywhere they are mentioned, plus some exceptions. Maybe at that point what we need to to also loop in dev-infra to share that tooling between fw/comp/cli. We're the only ones that use yarn workspaces afaik but we're not the only ones that have versions sprinkled around that need to be matched.

For now, I think extra validation checks are going to be paramount. Especially to ensure that the tests are using the same dependency versions that the users are using when they install the package.

I think this is also problematic if we need to use a different version of a dependency for a particular package (recent build optimizer TS issue, for example) or is there a mechanism for that situation? I don't think there are any of these currently though.

cc @josephperrott

We should also update renovate configuration now, so that it doesn't create separate PRs for dependencies and dev deps, as otherwise we'll have 2 PRs for each dependency update.

clydin · 2020-03-25T23:40:22Z

packages/angular_devkit/schematics/tasks/index.ts

+// TODO: This folder should not be prefixed with `not-`, but its's needed to workaround a Bazel bug:
+// https://github.com/bazelbuild/rules_nodejs/issues/313
+export { NodePackageInstallTask } from './not-node-package/install-task';
+export { NodePackageLinkTask } from './not-node-package/link-task';


This is quite invasive. Is there any chance we can prioritize a fix for this upstream?

Looping in @gregmagolan to see what he thinks.

Depending on the timeframe for an upstream fix, can we change not-node-package to package-manager instead? This could potentially be a permanent change since the name still reflects the purpose of the directory contents.

SGTM, changed it.

packages/angular_devkit/schematics_cli/BUILD.bazel

In most cases, this setting up creating more PRs instead of reducing them. This is because if a depedency is listed as both a `devDependencies` and `dependencies`, Renovate will create 2 PRs; Example: #17380 and #17379 This will happen more often now when adding Bazel integration tests see: #17304 (comment)

…UMDs The TS bug is microsoft/TypeScript#36780. The workaround is needed because `ts_library` emits UMDs currently. This will change with bazelbuild/rules_typescript#492 and bazel-contrib/rules_nodejs#1687.

Using an extension gives editors a chance to figure out the correct language syntax to use.

This PR changes ts_json_schema to not create a ts_library rule for the json schema, and instead let consuming ts_libraries use and compile the resulting .ts files themselves. This is needed in order to maintain the module_name of the consuming libraries on the .ts files resulting from json compilation.

Without this change, module names will be computed with incorrect paths containing duplicate path fragments (e.g. `@angular-devkit/architect/testing/testing/test-logger`).

See bazel-contrib/rules_nodejs#313 for details.

The build-webpack large spec uses it, but it was toplevel only because it was hoisted.

angular-automatic-lock-bot · 2020-05-08T08:03:16Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

filipesilva requested a review from clydin March 25, 2020 13:45

googlebot added the cla: yes label Mar 25, 2020

filipesilva added the target: major This PR is targeted for the next major release label Mar 25, 2020

filipesilva mentioned this pull request Mar 25, 2020

More bazel tests #16976

Merged

filipesilva force-pushed the prepare-bazel-integration-tests branch 2 times, most recently from 89223a1 to 78cae69 Compare March 25, 2020 16:50

clydin reviewed Mar 25, 2020

View reviewed changes

filipesilva force-pushed the prepare-bazel-integration-tests branch 2 times, most recently from 0d5f938 to e38d9b1 Compare March 26, 2020 15:04

alan-agius4 approved these changes Mar 31, 2020

View reviewed changes

alan-agius4 mentioned this pull request Apr 3, 2020

ci: disable separate renovate PRs for devDependencies and dependencies #17383

Merged

filipesilva added 18 commits April 7, 2020 12:11

build: fix bazel tests to run on windows

457b046

build: use runfiles on windows

9790ad3

fix(@angular-devkit/architect): temporary workaround for TS bug with …

3235d07

…UMDs The TS bug is microsoft/TypeScript#36780. The workaround is needed because `ts_library` emits UMDs currently. This will change with bazelbuild/rules_typescript#492 and bazel-contrib/rules_nodejs#1687.

build: build and test @ngtools/webpack with Bazel

fe680df

build: add module_name to @angular-devkit/build-optimizer Bazel rule

bdda757

refactor: use BUILD.bazel instead of BUILD

0fc9189

Using an extension gives editors a chance to figure out the correct language syntax to use.

build: also produce tar.gz for bazel npm_package rules

ef82fc9

build: move BUILD.bazel files for secondary entry points into own folder

2513487

Without this change, module names will be computed with incorrect paths containing duplicate path fragments (e.g. `@angular-devkit/architect/testing/testing/test-logger`).

build: include json files in @angular-devkit/build-optimizer Bazel build

425de7d

build: update com_google_protobuf shallow_since arg

c807ebd

build: workaround bazel module resolution bug

96c3874

See bazel-contrib/rules_nodejs#313 for details.

test(@angular-devkit/architect): run tests in Bazel

6c0ff8b

fix(@schematics/schematics): add module_name and data files

3f2392b

fix(@angular-devkit/schematics): add module_name and data files

e1cb441

test(@angular-devkit/schematics-cli): test with Bazel

d09c6d5

fix(@schematics/angular): include assets in ts library

51fc2ea

test(@angular/pwa): test with Bazel

9fcd794

filipesilva added 5 commits April 7, 2020 12:14

test(@angular/cli): test with Bazel

b283747

build: update yarn lock

5dfa64e

style: reformat bazel files

ac384fd

build: add missing raw-loader dependency

29137eb

The build-webpack large spec uses it, but it was toplevel only because it was hoisted.

docs: add windows note to bazel process doc

5dba778

filipesilva force-pushed the prepare-bazel-integration-tests branch from e38d9b1 to 5dba778 Compare April 7, 2020 11:17

clydin approved these changes Apr 7, 2020

View reviewed changes

filipesilva added the action: merge The PR is ready for merge by the caretaker label Apr 7, 2020

mgechev merged commit c8d5c3f into angular:master Apr 7, 2020

angular-automatic-lock-bot bot locked and limited conversation to collaborators May 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare bazel integration tests #17304

Prepare bazel integration tests #17304

filipesilva commented Mar 25, 2020

clydin Mar 25, 2020

filipesilva Mar 26, 2020

clydin Mar 26, 2020

alan-agius4 Mar 31, 2020

filipesilva Mar 31, 2020

alan-agius4 Mar 31, 2020 •

edited

Loading

filipesilva Apr 7, 2020

clydin Mar 25, 2020

filipesilva Mar 26, 2020

clydin Mar 26, 2020

filipesilva Mar 26, 2020

alan-agius4 Mar 31, 2020

clydin Mar 25, 2020

filipesilva Mar 26, 2020

clydin Apr 6, 2020

filipesilva Apr 7, 2020

angular-automatic-lock-bot bot commented May 8, 2020

Prepare bazel integration tests #17304

Prepare bazel integration tests #17304

Conversation

filipesilva commented Mar 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alan-agius4 Mar 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angular-automatic-lock-bot bot commented May 8, 2020

alan-agius4 Mar 31, 2020 •

edited

Loading