Validate NodeJS repo's yarn.lock is in sync with package.json

[flyingsilverfin]

Issue to Solve

Goal: make the developer re-generate package-lock.json from scratch every time package.json is changed.

In the NPM environment, it's possible that a package.json, which is the source of truth, is updated with new versions. However, the package-lock.json, which should update when running npm install (this is also run via bazel build in our node-based repositories), may not necessarily change - notably when the installed package in the lock file do not contradict the new versions required in the package.json [note: this is our experience testing with NPM version 7.16.0].

This happens often in the Node environment, unlike in Maven, because using range dependencies is conventional especially in transitive dependencies, for example:

"strip-ansi": "^4.0.0"

Continously regenerating the lock file from scratch negates the point of having a lock file, as two weeks later, for the same commit, newer transitive dependencies could be available and installed and the commit is no longer hermitic.

Thus, the best solution we have is to completely regenerate the lock file every time the package.json file is modified by the developer, allowing us to have hermitic commits, but also use the latest version of every dependency whenever the package.json is updated.

Proposed Solution

Write two new bazel rules to address this problem, which can be utilised locally and in CI:

1. check: ensure that the hash of package.json contents matches recorded hash in package-lock.json
2. sync: re-generate the package-lock.json from scratch (delete - run npm install), and add the hash of the source package.json into this file.

The intended workflow in CI is to:

• Require check passes. If check fails, tell the user to run sync to regenerate the package-lock.json WITH the hash of the package.json contents (ie. using npm install will not be sufficient) and make a new commit.

Other

• we may want to apply something similar in Python!

• we probably don't see this issue with dependencies coming up when constantly re-generating the maven snapshots because the Java convention appears to be to not use dependency version ranges nearly as much!

[lolski]

👍

We definitely want to have similar check for Python's pip package.

[alexjpwalker]

In my experience, @flyingsilverfin , running bazel build //... after updating package.json always results in the package-lock.json (or in YARN's case, yarn.lock) updating.

That being said, this issue still holds value because we want to verify that the developer has actually run bazel build //... or yarn to regenerate yarn.lock. If they haven't, their lockfile will be out of sync with the package.json, which may produce odd, potentially incorrect behaviour in CI.

Also, I don't see how this is relevant in pip. pip doesn't use a lockfile. cc @lolski