Unable to build Nomad with Nodejs v16 LTS

[dermotbradley]

Nomad version

1.1.6, 1.1.9 and 1.2.3

Operating system and Environment details

Alpine Linux v3.15 and Edge

Issue

Unable to build Nomad using Nodejs v16 - this version of Nodejs became LTS on 26th October as per https://nodejs.org/en/about/releases/

Appears to be a known Yuidoc issue: yui/yuidoc#453

Related Alpine issues:

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28610
https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28611

Reproduction steps

Expected Result

Actual Result

error: An uncaught YUIDoc error has occurred, stack trace given below
error: -------------------------------------------------------------------------
-
error: TypeError [ERR_INVALID_ARG_TYPE]: The "options" argument must be of type object. Received an instance of Array
    at ChildProcess.target.send (node:internal/child_process:713:7)
    at Array.forEach (<anonymous>)
    at dispatchQueuedRequests (/home/builder/package/src/nomad-1.1.6/ui/node_modules/ember-cli-test-loader/node_modules/workerpool/lib/WorkerHandler.js:174:21)
    at ChildProcess.<anonymous> (/home/builder/package/src/nomad-1.1.6/ui/node_modules/ember-cli-test-loader/node_modules/workerpool/lib/WorkerHandler.js:129:7)
    at ChildProcess.emit (node:events:390:28)
    at ChildProcess.emit (node:domain:475:12)
    at emit (node:internal/child_process:917:12)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)
error: --------------------------------------------------------------------------
error: Node.js version: v16.13.1
error: YUI version: 3.18.1
error: YUIDoc version: 0.10.2
error: Please file all tickets here: http://github.com/yui/yuidoc/issues
error: --------------------------------------------------------------------------

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[DerekStrickland]

Thanks for filing this issue! We'll take a look at it as soon as possible.

[ChaiWithJai]

Hey, this is likely because we're using ember-cli: 3.20. We need to upgrade 2 LTS's to ember-cli: 3.28. This requires dependency upgrades that would need to be tackled in a major release 1.3.

For now, we can only support Node v12. This issue has been on our radar and a candidate to tackle in 1.3.

[dermotbradley]

@ChaiWithJai The Alpine current release (3.15.x) has Nomad 1.1.1 and NodeJS 16.13.1 packaged so we're stuck as Nomad 1.1.9 cannot be packaged for the recent security fixes due to this issue...

[dermotbradley]

@ChaiWithJai

Any indication of timescales for a Nomad 1.3 release containing support for the current NodeJS LTS?

Alpine is still stuck with Nomad 1.1.1 in its current "stable" release (v3.15.0) and this Nomad version obviously does not have the security fixes from 1.1.2-1.1.9.

Should we remove Nomad completely from the current stable Alpine release? I see no other way to address the security implications of shipping a version with known CVEs.

We cannot switch NodeJs back to the previous LTS release as other packages in Alpine v3.15.0 may be relying on NodeJS 16.x.

[tgross]

Hi @dermotbradley!

Any indication of timescales for a Nomad 1.3 release containing support for the current NodeJS LTS?

Generally speaking we don't give firm timelines for releases, because lots can happen between now and then. But our rough plan is on the scale of "a couple months" rather than "a couple days".

Alpine is still stuck with Nomad 1.1.1 in its current "stable" release (v3.15.0) and this Nomad version obviously does not have the security fixes from 1.1.2-1.1.9.

While I'm definitely sympathetic to this... it doesn't seem like security patches are really relevant to this issue, are they?

Here's where you may need to fill in for us a little bit in terms of Alpine processes around stale builds of packages, but my understanding is that the distro won't ship packages that it can't currently build. (Which is good!) So the problem isn't that you can't package newer versions of Nomad, but that the NodeJS package in Alpine stable has been bumped out from under us. So no versions of Nomad can be packaged until we bump the dependency. Even if there were no security patches or you were to backport them, you can't continue to package Nomad with our required version of NodeJS.

Should we remove Nomad completely from the current stable Alpine release? I see no other way to address the security implications of shipping a version with known CVEs.

I see a few options here:

• As far as I can tell YUIDoc isn't a runtime dependency (@ChaiWithJai may need to check me on this). You could conceivably patch it out.
• NodeJS isn't a runtime dependency at all! Alpine's build infrastructure could in theory provide arbitrary NodeJS versions with isolated build environments, that don't get packaged with the final package.

But I totally understand that either of those options might not work with Alpine's packaging policies! In which case, yes, you might need to remove Nomad from Alpine stable until we've released 1.3.0. That's unfortunate but we'll definitely make sure we get a newer NodeJS LTS by then (Node 12 is towards the end of maintenance anyways).

---

As an aside: Nomad has some CGO dependencies and we frankly don't test with musl at all. I'd love to hear more about how Alpine validates the package after the build, as that might help influence us to put the work in to do the same validation and that'll get Nomad into the hands of more Alpine users without them doing terrible things like installing glibc in Alpine. 😀

[loqs]

I encountered a similar failure with vault. The change in nodejs causing the issue is 1. The fix for workerpool is 2. nomad 1.2.6 uses workerpool 2.3.3 3 the same as vault 1.9.4. I was able to get vault to build with nodejs 16 by vendoring a patched workerpool 2.3.3, I did not have to patch workerpool 3.1.2. @dermotbradley perhaps that can provide you with a solution until 1.3.0 is released.

[dermotbradley]

@tgross Sorry for taking so long to respond again to this.

So Alpine does package more than one version of NodeJS - the current LTS ("Active LTS" as NodeJS call it: https://nodejs.org/en/about/releases/), which is v16, and 1 other newer non-LTS version, currently v18 as it just came out but was v17 up until a couple of days ago.

The underlying issue appears to be that Nomad (and also Vault) in general are lagging in the LTS version that they support - v12 goes out of support in 9 days time, whilst v14 is still in support for another year I see no reason to select it rather than v16 which has been the current LTS for almost 5 months now.

The same issue was raised by someone else with the Vault team some time ago: hashicorp/vault#8504

I see someone in that Vault issue ticket pointing out that RHEL/CentOS 9 also only provides NodeJS v16 so the underlying issue is not specific to Alpine.

Currently the Nomad and Vault packages for Alpine have been built without a UI to avoid this issue. However Consul does not appear to have the same issue - it seems that Consul is built for Alpine, complete with UI, without NodeJS being used.

It is not clear if Nomad v1.3.x will officially support NodeJS v16 - I have compiled v1.3.0-beta.1 successfully with that but have not yet had a chance to test it. Can you indicate what the status is regarding supported NodeJS version(s) for v1.3.x?

NodeJS v18 will be the next LTS in 6 months time so I suspect the same problem will reoccur then - v18 was released a couple of days ago and so the next 6 months should give ample time for projects to prepare for it becoming the new Active LTS in October. As Alpine v3.17 is due to be released approx 1 month afterwards then I expect that Alpine version to ship with NodeJS v18.

So I'm not sure whether we should keep the UI disabled or not going forward. I'd hate to re-enable it for Nomad 1.3.x and then have to disable it again after the next NodeJS LTS comes out.

One potential solution I think someone suggested is whether the Nomad source tarball could ship with the UI artifacts already present, so avoiding the need to use NodeJS when building from source.

As an aside: Nomad has some CGO dependencies and we frankly don't test with musl at all. I'd love to hear more about how Alpine validates the package after the build

Basically there is no validation apart from whether it builds on all the supported archs. In the specific case of Nomad it hasn't actually had an active maintainer since it first was packaged back in late 2019 - unfortunately it sometimes happens that someone creates a package, updates it a few times, and then stops maintaining it (others have updated the Nomad package during the past 2 years but not taken over active maintenance of it). I am intended to change this for the Alpine Nomad package.

You can see the 6 archs that it is currently packaged for here: https://pkgs.alpinelinux.org/packages?name=nomad&branch=edge

[tgross]

Hi @dermotbradley! Just for some background, we're currently in the process of migrating our release infrastructure to a new HashiCorp-wide centrally managed release tooling. So there's some stuff in flux here as we migrate away from each product team having unique processes and you're getting caught up in that a bit. In the near future you should start seeing more consistency across products.

Also note that we support two previous major versions of Nomad at a time (so with Nomad 1.3.0's release we'll continue support for Nomad 1.2.x and Nomad 1.1.x). Although sometimes we have to upgrade our oldest supported Nomad's toolchain because of the lifecycle of the toolchains, we really try to avoid it if we can. So unless a NodeJS version goes entirely out of support and we need to ship a backported bug fix, we won't upgrade it in the older Nomad versions.

The underlying issue appears to be that Nomad (and also Vault) in general are lagging in the LTS version that they support - v12 goes out of support in 9 days time, whilst v14 is still in support for another year I see no reason to select it rather than v16 which has been the current LTS for almost 5 months now.

Agreed! Unfortunately we just didn't manage to get to the upgrade for 1.3.0-beta.1. There's always a bunch of decisions around priorities we have to make and this didn't quite get above the line.

It is not clear if Nomad v1.3.x will officially support NodeJS v16 - I have compiled v1.3.0-beta.1 successfully with that but have not yet had a chance to test it. Can you indicate what the status is regarding supported NodeJS version(s) for v1.3.x?

We built Nomad 1.3.0 with NodeJS v14 and that'll most likely be the version we ship with for the entire 1.3.x cycle. Our test infrastructure doesn't currently test the UI with different versions of the toolchains (go or NodeJS), so we can't really speak to whether NodeJS v16 will actually work until we're ready to do the upgrade ourselves. We can only actively support the toolchains we build on but if you have patches to contribute for forwards compatibility that don't break the current builds, we're always more than happy to get those.

One potential solution I think someone suggested is whether the Nomad source tarball could ship with the UI artifacts already present, so avoiding the need to use NodeJS when building from source.

The source tarball under the GitHub release includes the bundled assets. You can find that at ./command/agent/bindata_assetfs.go within the tarball for a given release (ex [v1.3.0-beta.1]https://github.com/hashicorp/nomad/releases/tag/v1.3.0-beta.1)). I suspect some other package maintainers maybe won't care for that as a solution because of their distribution's policies. But if that works for you, great!