node-gyp ignores(?) node-rdkafka binding.gyp cflags settings when included via a library when building for node 20 or 22, yet works on node 18

[argh]

Environment Information

• OS [e.g. Mac, Arch, Windows 10]: mac local, docker alpine, docker node:20
• Node Version [e.g. 8.2.1]: 20, 22
• NPM Version [e.g. 5.4.2]: 10.7.0
• C++ Toolchain [e.g. Visual Studio, llvm, g++]: g++
• node-rdkafka version [e.g. 2.3.3]: 2.18.0, 3.0.x

Steps to Reproduce

node-rdkafka is used internally by a custom library, which is built and npm published to our repo. (I've tried these steps with a couple different versions of node-rdkafka

This custom library is then included as a dependency of an application project. When the application project runs npm install, and it includes the dependency, it runs node-gyp, but the cflags setting doesn't correctly get set (c++-17) so it bombs out with lots of errors.

Here's the sequence of steps that trigger the issue...

nvm use 20
(cd library-dependency)
npm install (works)
publish library, all good.
(also, going to node-modules/node-rdkafka and manually running node-gyp rebuild works.

(cd application-project)
nvm use 20
npm install (fails)

npm error In file included from /root/.cache/node-gyp/20.15.1/include/node/v8-object.h:9,
npm error                  from /root/.cache/node-gyp/20.15.1/include/node/v8-array-buffer.h:13,
npm error                  from /root/.cache/node-gyp/20.15.1/include/node/v8.h:24,
npm error                  from /root/.cache/node-gyp/20.15.1/include/node/node.h:73,
npm error                  from ../../nan/nan.h:58,
npm error                  from /build/node_modules/node-rdkafka/src/binding.h:13,
npm error                  from ../src/binding.cc:11:
npm error /root/.cache/node-gyp/20.15.1/include/node/v8-maybe.h:106:45: error: 'is_lvalue_reference_v' is not a member of 'std'; did you mean 'is_lvalue_reference'?
npm error   106 |   template <class U, std::enable_if_t<!std::is_lvalue_reference_v<U>>*>
npm error       |                                             ^~~~~~~~~~~~~~~~~~~~~
npm error       |                                             is_lvalue_reference
npm error In file included from ../../nan/nan.h:58:
npm error Library/Caches/node-gyp/20.15.1/include/node/node.h:692:8: error: no template named 'optional' in namespace 'std'
npm error   std::optional<std::string> builder_script_path;

nvm use 22
npm install (fails, same as above)

nvm use 18
npm install (works!)

nvm use 20
add

  "overrides": {
    "node-rdkafka": "3.x.x"
  },

to application package.json...
npm install (works!)

Additional context

The application doesn't touch node-rdkafka directly, it uses the implementation from the included library only. But adding the dependency as an override in the parent app makes everything start to work.

This is code that has worked for years without being touched, all I was trying to do was use a more modern version of node. I'd nuked package-lock and node_modules, so this was a clean install. It seems really weird to me that the "node-gyp a dependency" works in node 18 but not on node 20-22 with the exact same code. None of our other libraries that have node-gyp built code require this kind of override in the application.