default variable ignored for `openssl_fips` #2750

TheJJ · 2022-10-19T13:44:28Z

I can't build element-desktop with node18 due to the error message:

gyp: name 'openssl_fips' is not defined while evaluating condition 'openssl_fips != ""' in binding.gyp while trying to load binding.gyp

openssl_fips used in /tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp/20.1.4/include/node/common.gypi,
which is fetched from https://electronjs.org/headers/v20.1.4/node-v20.1.4-headers.tar.gz and put in npm_config_devdir, i.e. ~/.electron-gyp.

include/node/common.gypi looks like this:

{
  'variables': {
    ...,
    'openssl_fips%': '',
    'openssl_no_asm%': 0,
    'conditions': [
      ...,
      ['openssl_fips != ""', {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }, {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }],
      ...
    ]
}

The definition of openssl_fips% = '' is also in the file, so according to https://gyp.gsrc.io/docs/InputFormatReference.md variables ending with % should be used as default value if not set previously.

But that mechanism doesn't seem to kick in:

first, to be able to debug and see a backtrace, edit node-gyp/gyp/pylib/gyp/__init__.py so def main(args): only does return gyp_main(args) without swallowing the very useful backtrace.
when inspecting the variables dictionary in node_modules/node-gyp/gyp/pylib/gyp/input.py's EvalSingleCondition just before the crash 'openssl_fips%': '', but openssl_fips is unset. The condition expression from the node-headers then can't be evaluated since the variable is not available when EvalSingleCondition runs eval(ast_code, ..., variables).

-> either, the condition has to lookup the variable with a % appended, or it needs to be in the variables dict without a %. I guess the latter should be the case.

I haven't found yet where exactly the bug is, or if it even is a bug in node-gyp. But maybe you already have some idea :)

Node Version: node v18.10.0 and npm 8.19.2
Platform: Linux 5.19.8-JJ
Compiler: gcc version 12.2.0
Module: keytar, for element-desktop

Verbose output (from npm or node-gyp):

yarn run hak
neon info generating native/index.node
hak build: keytar
Running yarn with env {
  [... many env variables],
  npm_config_init_version: '1.0.0',
  npm_config_arch: 'x64',
  npm_config_target_arch: 'x64',
  npm_config_disturl: 'https://electronjs.org/headers',
  npm_config_runtime: 'electron',
  npm_config_target: '20.1.4',
  npm_config_build_from_source: true,
  npm_config_devdir: '/tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp'
}
gyp info it worked if it ends with ok
gyp info using node-gyp@8.4.1
gyp info using node@18.10.0 | linux | x64
gyp info find Python using Python version 3.10.7 found at "/usr/lib/python-exec/python3.10/python3"
gyp http GET https://electronjs.org/headers/v20.1.4/node-v20.1.4-headers.tar.gz
gyp http 200 https://artifacts.electronjs.org/headers/dist/v20.1.4/node-v20.1.4-headers.tar.gz
gyp http GET https://electronjs.org/headers/v20.1.4/SHASUMS256.txt
gyp http 200 https://artifacts.electronjs.org/headers/dist/v20.1.4/SHASUMS256.txt
gyp info spawn /usr/lib/python-exec/python3.10/python3
gyp info spawn args [
gyp info spawn args   '/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp/20.1.4/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp/20.1.4',
gyp info spawn args   '-Dnode_gyp_dir=/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp/20.1.4/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp: name 'openssl_fips' is not defined while evaluating condition 'openssl_fips != ""' in binding.gyp while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/node-gyp/lib/configure.js:259:16)
gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Linux 5.19.8-JJ
gyp ERR! command "/usr/bin/node" "/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd /tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build
gyp ERR! node -v v18.10.0
gyp ERR! node-gyp -v v8.4.1
gyp ERR! not ok

The text was updated successfully, but these errors were encountered:

TheJJ · 2022-10-19T14:29:05Z

More minimal example:
stripped down common.gypi from node-headers:

{
  'variables': {
    'openssl_fips%': '',
    'openssl_no_asm%': 0,

    'conditions': [
      ['openssl_fips != ""', {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }, {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }],
    ],
  },
}

startfile.gypi

{}

Run

python3 node-gyp/gyp/gyp_main.py -Icommon.gypi startfile.gypi --depth=. --no-parallel --generator-output build -Goutput_dir=.

Then one gets openssl_fips is undefined.

ProcessVariablesAndConditionsInDict is called two times: once for the "root" dict this directly recurses into "variables". the default variable (for %) setting in LoadVariablesFromVariablesDict is tried in the second call, but since the_dict.get("variables") returns None since there's no more nested variables dict, default values are not set, and hence the condition variable is not set yet.

I'm still not sure what the proper fix is - it seems the condition evaluation happens before the variable default was set.

cclauss · 2022-10-19T14:33:14Z

Related https://github.com/nodejs/node-gyp/issues?q=label%3AOpenSSL

TheJJ · 2022-10-19T14:52:40Z

My idea for a possible fix is to adjust ProcessVariablesAndConditionsInDict and move the first call to

LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key)

to happen before the recursion step into the variables-sub-dict, i.e. move the call from directly after the if "variables" in the_dict: block to before it. This would make sense since the documentation says that variables are inherited into nested scopes?
With this patch, the default variables are set before we descend into nested scopes, and so the variable is set before the conditions are evaluated.

--- /tmp/node-gyp/gyp/pylib/gyp/input.py	2022-10-19 12:15:28.240012241 +0200
+++ /tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/node_modules/node-gyp/gyp/pylib/gyp/input.py	2022-10-19 16:50:19.555358975 +0200
@@ -1304,6 +1334,8 @@
     variables = variables_in.copy()
     LoadAutomaticVariablesFromDict(variables, the_dict)
 
+    LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key)
+
     if "variables" in the_dict:
         # Make sure all the local variables are added to the variables
         # list before we process them so that you can reference one
@@ -1321,8 +1353,6 @@
             the_dict["variables"], phase, variables, build_file, "variables"
         )
 
-    LoadVariablesFromVariablesDict(variables, the_dict, the_dict_key)
-
     for key, value in the_dict.items():
         # Skip "variables", which was already processed if present.
         if key != "variables" and type(value) is str:

richardlau · 2022-10-19T14:58:15Z

I'm still not sure what the proper fix is - it seems the condition evaluation happens before the variable default was set.

From https://gyp.gsrc.io/docs/InputFormatReference.md#user_defined-variables:

A user-defined variable may be defined in terms of other variables, but not other variables that have definitions provided in the same scope.

I think this is because .gyp/.gypi files are Python dictionaries with no implicit ordering of keys.

cclauss · 2022-10-19T15:02:40Z

Python dicts are guaranteed to be insert order preserving in Python >= 3.7.

https://stackoverflow.com/questions/39980323/are-dictionaries-ordered-in-python-3-6

richardlau · 2022-10-19T15:18:07Z

There's some old discussion in https://groups.google.com/g/gyp-developer/c/1EWXAXe-qWs but maybe things could be different now in gyp-next with Python 3.7+ 🤷.

TheJJ · 2022-10-19T15:23:01Z

The question is what should happen?
Should this default variable be usable within the conditions block?

If it should, then one needs to evaluate/set the default vars before descending into nested scopes (my proposed patch).
Alternatively, the nested scope somehow has to figure out that there's a not-yet-set variable in an outer scope.

If it should not, then electron has to somehow rewrite their gypi file?

richardlau · 2022-10-19T15:45:05Z

openssl_fips used in /tmp/portage/net-im/element-desktop-1.11.10/homedir/.electron-gyp/20.1.4/include/node/common.gypi, which is fetched from https://electronjs.org/headers/v20.1.4/node-v20.1.4-headers.tar.gz and put in npm_config_devdir, i.e. ~/.electron-gyp.

include/node/common.gypi looks like this:
{
  'variables': {
    ...,
    'openssl_fips%': '',
    'openssl_no_asm%': 0,
    'conditions': [
      ...,
      ['openssl_fips != ""', {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }, {
        'openssl_product': '<(STATIC_LIB_PREFIX)openssl<(STATIC_LIB_SUFFIX)',
      }],
      ...
    ]
}

FWIW I believe Electron 20 contains Node.js 16 (and this matches what is in the current https://github.com/nodejs/node/blob/v16.x/common.gypi). This is a redundant condition as both the branches assign the same value to openssl_product. I'll open a PR in Node.js to fix this in future versions of Node.js 16. Node.js 18 doesn't have this condition as it was written by a later semver-major commit.

TheJJ · 2022-10-19T15:58:54Z

Thanks! That should fix it, but seems like another workaround, possible here since the condition is redundant :)

So in my opinion maybe the root cause should also be clarified/fixed, i.e. how to handle the default variable propagation.

richardlau · 2022-10-19T16:25:02Z

Opened nodejs/node#45076 for this specific case. Discussion around the general case is worth having, although I guess ultimately if changes were to be made they would go upstream in https://github.com/nodejs/gyp-next.

TheJJ · 2022-10-19T16:43:28Z

Great, thanks! Out of curiosity, how could this ever have worked previously so that electron ships the file that way?

richardlau · 2022-10-19T16:56:35Z

Great, thanks! Out of curiosity, how could this ever have worked previously so that electron ships the file that way?

I don't know about Electron but Node.js writes the openssl_fips variable into process.config.
e.g.

$ nvm run 16 -p process.config
Running node v16.17.0 (npm v8.15.0)
{
  target_defaults: {
    cflags: [],
    default_configuration: 'Release',
    defines: [ 'NODE_OPENSSL_CONF_NAME=nodejs_conf' ],
    include_dirs: [],
    libraries: []
  },
  variables: {
...
    openssl_fips: '',
    openssl_is_fips: false,
    openssl_quic: true,
...
  }
}

node-gyp uses that if custom headers are not used

node-gyp/lib/create-config-gypi.js

Lines 18 to 35 in d7687d5

    
           async function getBaseConfigGypi ({ gyp, nodeDir }) { 
        
             // try reading $nodeDir/include/node/config.gypi first when: 
        
             // 1. --dist-url or --nodedir is specified 
        
             // 2. and --force-process-config is not specified 
        
             const useCustomHeaders = gyp.opts.nodedir || gyp.opts.disturl || gyp.opts['dist-url'] 
        
             const shouldReadConfigGypi = useCustomHeaders && !gyp.opts['force-process-config'] 
        
             if (shouldReadConfigGypi && nodeDir) { 
        
               try { 
        
                 const baseConfigGypiPath = path.resolve(nodeDir, 'include/node/config.gypi') 
        
                 const baseConfigGypi = await fs.promises.readFile(baseConfigGypiPath) 
        
                 return parseConfigGypi(baseConfigGypi.toString()) 
        
               } catch (err) { 
        
                 log.warn('read config.gypi', err.message) 
        
               } 
        
             } 
        
             // fallback to process.config if it is invalid 
        
             return JSON.parse(JSON.stringify(process.config))

TheJJ · 2022-10-21T10:24:08Z

I see, thanks - so the error in the element-desktop build stems from:

> nodejs-16 no longer defines openssl_fips as variable in their default config
element-desktop uses electron, which internally uses node-16
since i build with node-18, openssl_fips is no longer defined in the default process.config, so it's undefined in include/node/common.gypi. the default definition in that file has always been ignored since same-scope default variable definitions are not supported by gyp.

is that right?

for my pracical issue - should we just wait until the fix is upstream in node-16 and electron ships the new gyp file, or do you see a better way?

richardlau · 2022-10-21T14:45:13Z

@TheJJ

gyp info spawn args   '-I',
gyp info spawn args   '/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/build/config.gypi',

Could you check if /tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/build/config.gypi contains a

                 'openssl_fips': '',

line? This file is written out by node-gyp in https://github.com/nodejs/node-gyp/blob/main/lib/create-config-gypi.js and if it is not there we should try to figure out why.

richardlau · 2022-10-21T15:06:53Z

Ah. I think this may have already been fixed in node-gyp. At least I've found that node-gyp@9.0.0 works for me with a test addon:

npm_config_disturl=https://electronjs.org/headers npm_config_runtime=electron npm_
config_target=20.1.4 npm_config_devdir=/home/rlau/.electron-gyp npx node-gyp@9.0.0 rebuild

whereas node-gyp@8.4.1 fails:

npm_config_disturl=https://electronjs.org/headers npm_config_runtime=electron npm_config_target=20.1.4 npm_config_devdir=/home/rlau/.electron-gyp npx node-gyp@8.4.1 rebuild
...
gyp: name 'openssl_fips' is not defined while evaluating condition 'openssl_fips != ""' in binding.gyp while trying to load binding.gyp
g

Looking at the changes that went into node-gyp@9.0.0 v8.4.1...v9.0.0 it looks like #2547 is the most likely to have fixed this as npm_config_disturl is involved.

TheJJ · 2022-10-23T11:09:12Z

/tmp/portage/net-im/element-desktop-1.11.10/work/element-desktop-1.11.10/.hak/keytar/x86_64-unknown-linux-gnu/build/build/config.gypi

it does not define openssl_fips, but it does define "openssl_is_fips": "false" and "openssl_quic": "false".

TheJJ · 2022-11-15T20:57:34Z

Any idea how we can properly resolve this issue? Thanks! Wait until electron depends on >=node-gyp-9.0 or no longer includes the openssl_fips-using config file?

TheJJ · 2022-11-16T13:46:50Z

I've now "fixed" this "problem" for my concrete gentoo ebuild with a lot of duct-tape: SFTtech/gentoo-overlay@3eb701f

I couldn't just depend on node-16 because some dependency png-to-ico requires "@types/node": "^17.0.36",. I'm impressed how these projects do ever even build once.

Both paths for the condition being removed result in the same value being assigned to `openssl_product`. This condition was also problematic as it was testing a variable in the same scope which gyp/gyp-next currently does not support. Refs: https://gyp.gsrc.io/docs/InputFormatReference.md#user_defined-variables PR-URL: nodejs#45076 Refs: nodejs/node-gyp#2750 Refs: nodejs#38633 Reviewed-By: Michael Dawson <midawson@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

Both paths for the condition being removed result in the same value being assigned to `openssl_product`. This condition was also problematic as it was testing a variable in the same scope which gyp/gyp-next currently does not support. Refs: https://gyp.gsrc.io/docs/InputFormatReference.md#user_defined-variables PR-URL: nodejs/node#45076 Refs: nodejs/node-gyp#2750 Refs: nodejs/node#38633 Reviewed-By: Michael Dawson <midawson@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

Without this workaround node18 fails to generate prebuilds for node versions < 18 with: gyp: name 'openssl_fips' is not defined while evaluating condition 'openssl_fips != ""' in binding.gyp while trying to load binding.gyp

TheJJ · 2023-06-29T12:05:22Z

I think this has now been fixed in newer electron releases.

cclauss added the OpenSSL Related to https://www.openssl.org label Oct 19, 2022

cclauss added the Linux label Oct 19, 2022

richardlau mentioned this issue Oct 19, 2022

[v16.x] build: remove redundant condition from common.gypi nodejs/node#45076

Merged

TheJJ mentioned this issue Oct 19, 2022

Any reason element-web's dependencies couldn't have nodejs-16 allowed? SFTtech/gentoo-overlay#5

Closed

cswendrowski mentioned this issue Mar 20, 2023

Swap linux-arm build to use linux-arm64-lts Level/classic-level#71

Merged

TheJJ closed this as completed Jun 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

default variable ignored for `openssl_fips` #2750

default variable ignored for `openssl_fips` #2750

TheJJ commented Oct 19, 2022

TheJJ commented Oct 19, 2022

cclauss commented Oct 19, 2022

TheJJ commented Oct 19, 2022

richardlau commented Oct 19, 2022

cclauss commented Oct 19, 2022

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022 •

edited

Loading

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022 •

edited

Loading

richardlau commented Oct 19, 2022 •

edited

Loading

TheJJ commented Oct 21, 2022

richardlau commented Oct 21, 2022

richardlau commented Oct 21, 2022 •

edited

Loading

TheJJ commented Oct 23, 2022

TheJJ commented Nov 15, 2022 •

edited

Loading

TheJJ commented Nov 16, 2022

TheJJ commented Jun 29, 2023

default variable ignored for openssl_fips #2750

default variable ignored for openssl_fips #2750

Comments

TheJJ commented Oct 19, 2022

TheJJ commented Oct 19, 2022

cclauss commented Oct 19, 2022

TheJJ commented Oct 19, 2022

richardlau commented Oct 19, 2022

cclauss commented Oct 19, 2022

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022 • edited Loading

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022

richardlau commented Oct 19, 2022

TheJJ commented Oct 19, 2022 • edited Loading

richardlau commented Oct 19, 2022 • edited Loading

TheJJ commented Oct 21, 2022

richardlau commented Oct 21, 2022

richardlau commented Oct 21, 2022 • edited Loading

TheJJ commented Oct 23, 2022

TheJJ commented Nov 15, 2022 • edited Loading

TheJJ commented Nov 16, 2022

TheJJ commented Jun 29, 2023

default variable ignored for `openssl_fips` #2750

default variable ignored for `openssl_fips` #2750

TheJJ commented Oct 19, 2022 •

edited

Loading

TheJJ commented Oct 19, 2022 •

edited

Loading

richardlau commented Oct 19, 2022 •

edited

Loading

richardlau commented Oct 21, 2022 •

edited

Loading

TheJJ commented Nov 15, 2022 •

edited

Loading