Kibana 8.7.0-snapshot with Node 18.x does not install on Ubuntu 18.04

[marius-dr]

The current Node 18 version used in Kibana main requires glibc_2.28. This is not available on ubuntu 18.04 (there are unstable hacks with building it from source, but it's not officially supported on the kernel shipped with ubuntu 18.04).
This makes the deb packages fail to install in ubuntu 18.04. Has to be fixed because Ubuntu 18.04 is on our support matrix as a supported OS.
#144012

usr/share/kibana/bin/ ../node/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28` not found (required by usr/share/kibana/bin/ ../node/bin/node)
 dpkg: error processing package kibana (--install):
installed kibana package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 kibana

This will probably also be true for other OSs that do not have GLIBC_2.28 (do not have a comprehensive list right now, but this one is the largest one in terms of userbase)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[jbudz]

Context on upgrade: Node 16 is EOL on 2023-09-11 https://nodejs.org/en/blog/announcements/nodejs16-eol/

[droberts195]

RHEL/CentOS 7 is also on the support matrix, and that uses glibc 2.17. That's the oldest supported distribution. So, to avoid removing platform support in a minor, a build of Node that uses glibc 2.17 would be required.

It's deliberate that Node 18 requires glibc 2.28: https://nodejs.org/de/blog/announcements/v18-release-announce/#toolchain-and-compiler-upgrades

• Prebuilt binaries for Linux are now built on Red Hat Enterprise Linux (RHEL) 8 and are compatible with Linux distributions based on glibc 2.28 or later, for example, Debian 10, RHEL 8, Ubuntu 20.04.

Interestingly that doesn't say it won't work on older distributions, just implies you'd have to build it yourself.

[lukeelmers]

to avoid removing platform support in a minor

This is going to be the key question if we can't identify any stable workarounds. Even if we tried to revert and delay the Node upgrade, we (almost certainly) won't have a major stack release before the Node 16 EOL anyway. Removing platform support in a minor isn't without precedent; it looks like we dropped Ubuntu 14.04 support in 6.7.0. But that was just one platform, and I don't know what our reasoning was for it at the time.

cc @elastic/kibana-security

[marius-dr]

Regarding Ubuntu 20.04 I should note that only 20.04.5 has glibc newer than 2.27 (2.31) so most people will require an upgrade there as well.

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes/ChangeSummary/20.04.5

glibc	1951033	Update for 20.04.

glibc

1951033

Update for 20.04.

[marius-dr]

I will go through our Kibana support matrix and update this comment with the full list of impacted OSs and if there is an upgrade path that the users can take.
LE:

OS	glibc version	upgrade path	status
CentOS/RHEL 7.x	2.17	none	Not OK
RHEL 8.x	2.28		OK
RHEL 9.x	2.34		OK
Oracle Enterprise Linux 7	2.17	none	Not OK
Oracle Enterprise Linux 8	2.28		OK
Almalinux 8	2.28		OK
Rocky Linux 8	2.28		OK
Ubuntu 18.04	2.27	Ubuntu 18.10	Maybe
Ubuntu 20.04	2.27	Ubuntu 20.04.5	OK
Ubuntu 22.04	2.35		OK
OpenSUSE 15	2.26	OpenSUSE 15.4	OK
SLES 12	2.22	none	Not OK
SLES 15	2.31		OK
Debian 10	2.28		OK
Debian 11	2.31		OK
Amazon Linux 2	2.26	Amazon Linux 2022	Maybe

[watson]

I've been thinking about how to get ahead of this issue in the future. The best I think we can do is to check the list of supported Node.js platforms every time there's a new major release of Node.js. For example, here's the list of supported platforms for:

• Node.js 16: https://github.com/nodejs/node/blob/v16.x/BUILDING.md#platform-list
• Node.js 17: https://github.com/nodejs/node/blob/v17.x/BUILDING.md#platform-list
• Node.js 18: https://github.com/nodejs/node/blob/v18.x/BUILDING.md#platform-list
• Node.js 19: https://github.com/nodejs/node/blob/v19.x/BUILDING.md#platform-list

If we had checked this list once Node.js 18.0.0 was released (2022-04-19) we would have caught this issue around 17 months before EoL of Node.js 16 (2023-09-11) - which would almost satisfy our 18 months deprecation rule.

[watson]

I think in any case we'll most likely want to revert the upgrade to Node.js 18 in main until we have a solution right? I can go ahead and do that if y'all agree (UPDATE: I've been told that this is what we need to do, so I'll make that my top priority). I would prefer not to simply revert the commit, but to make a new PR which reverts the upgrade. That way we can keep some of the improvements that was also introduced in the PR which would make upgrading in the future that much easier.