deps: add template for generated headers

[danbev]

OpenSSL 3.0 has a number of header files that are generated, and
currently these headers are copied into the architecture specific
directories. This is done for each asm type, 'asm', 'asm_avx2', and
'no-asm' which has takes up quite a lot of disk space and also becomes
an issue with the headers.tar file which has increased due to this.

This commit adds copies the headers to a common directory for the
architecture, for example with linux-x86_64 there will be a directory
named deps/openssl/config/archs/linux-x86_64/common/include where the headers
will be copied (into subdirectories 'openssl' and 'crypto'. And in the
original locations a header file with the same name will be generated
which points (includes) the common header file.

Fixes: #42081

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43345/

[danbev]

With these changes the size of include/node/openssl/archs is:

$ du -hs node-v18.0.0/include/node/openssl/archs/
37M	node-v18.0.0/include/node/openssl/archs/

I'm not sure if this is acceptable or not but at least it is an improvement current size which is around 61M.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43351/

[mhdawson]

I'm not sure if this is acceptable or not but at least it is an improvement current size which is around 61M.

Definitely a good improvement. I think we should compare to the pre-18.x size to judge whether this is a great first step or the complete fix.

[richardlau]

I'm not sure if this is acceptable or not but at least it is an improvement current size which is around 61M.

Definitely a good improvement. I think we should compare to the pre-18.x size to judge whether this is a great first step or the complete fix.

It's a step in the right direction but still much bigger than when we were using OpenSSL 1.1.1. For comparison the equivalent directories in the headers package for Node.js 16.14.0 is 2.7M (#42081).

[danbev]

For comparison the equivalent directories in the headers package for Node.js 16.14.0 is 2.7M (#42081).

I've added a comment to #42081 about the reason for the larger size in OpenSSL 3.x, compared to 1.1.1. While there might be more options to cut the size I don't think we will get it down to a size close to that of 1.1.1 as there are now more headers that are generated specifically for an architecture in 3.x than there were for 1.1.1. But I'll take another look and see if there is more that could be done.

[danbev]

I've been able to find one set of headers that are not being used anymore, and another set for providers which could be shared per architecture like is done in this PR. I'll make these changes and see what the size of the headers.tar is after that.

[mhdawson]

@danbev as an FYI some of the failures in the CI do look related to compiling openssl

[danbev]

@danbev as an FYI some of the failures in the CI do look related to compiling openssl

Thanks, I'll complete the changes I've got and then run through CI again on Monday.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43427/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43431/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43435/

[danbev]

With the latest changes the sizes of the sub directories in archs in the produced headers tar file looks like this:

$ ls | xargs du -sch                                                            
1.5M    aix64-gcc-as                                                            
1.5M    aix-gcc                                                                 
1.5M    BSD-x86                                                                 
1.5M    BSD-x86_64                                                              
1.5M    darwin64-arm64-cc                                                       
1.5M    darwin64-x86_64-cc                                                      
1.5M    darwin-i386-cc                                                             
1.5M    linux32-s390x                                                              
1.5M    linux64-mips64                                                             
1.1M    linux64-riscv64                                                            
1.5M    linux64-s390x                                                              
1.5M    linux-aarch64                                                           
1.5M    linux-armv4                                                             
1.5M    linux-elf                                                               
1.5M    linux-ppc                                                               
1.5M    linux-ppc64                                                             
1.5M    linux-ppc64le                                                           
1.5M    linux-x86_64                                                            
1.5M    solaris64-x86_64-gcc                                                    
1.5M    solaris-x86-gcc                                                         
2.8M    VC-WIN32                                                                
2.8M    VC-WIN64A                                                               
936K    VC-WIN64-ARM  
                                                          
35M     total                                                                   

The windows variants are larger but they are also excluded from the headers handling that is part of this PR. We could look into them and see what can be done but if we assume that we can do something similar to what we have done to the others that would probably only get the total size down to something like 34M which I'm not really sure makes that much of a difference.

[richardlau]

With the latest changes the sizes of the sub directories in archs in the produced headers tar file looks like this:

$ ls | xargs du -sch                                                            
1.5M    aix64-gcc-as                                                            
1.5M    aix-gcc                                                                 
1.5M    BSD-x86                                                                 
1.5M    BSD-x86_64                                                              
1.5M    darwin64-arm64-cc                                                       
1.5M    darwin64-x86_64-cc                                                      
1.5M    darwin-i386-cc                                                             
1.5M    linux32-s390x                                                              
1.5M    linux64-mips64                                                             
1.1M    linux64-riscv64                                                            
1.5M    linux64-s390x                                                              
1.5M    linux-aarch64                                                           
1.5M    linux-armv4                                                             
1.5M    linux-elf                                                               
1.5M    linux-ppc                                                               
1.5M    linux-ppc64                                                             
1.5M    linux-ppc64le                                                           
1.5M    linux-x86_64                                                            
1.5M    solaris64-x86_64-gcc                                                    
1.5M    solaris-x86-gcc                                                         
2.8M    VC-WIN32                                                                
2.8M    VC-WIN64A                                                               
936K    VC-WIN64-ARM  
                                                          
35M     total                                                                   

The windows variants are larger but they are also excluded from the headers handling that is part of this PR. We could look into them and see what can be done but if we assume that we can do something similar to what we have done to the others that would probably only get the total size down to something like 34M which I'm not really sure makes that much of a difference.

I'm wondering if we can get rid of some of these. e.g. aix-gcc, darwin-i386-cc, linux32-s390x, linux-ppc, linux-ppc64, solaris-x86-gcc.

[richardlau]

I'm also not sure what linux-armv4 is.

[danbev]

I'm wondering if we can get rid of some of these. e.g. aix-gcc, darwin-i386-cc, linux32-s390x, linux-ppc, linux-ppc64, solaris-x86-gcc.

I don't have the answer to that question, but I'd be all for removing them if they are no longer needed 👍

[mhdawson]

@miladfarca can you check with Vascili on which of we need?

1.5M aix64-gcc-as
1.5M aix-gcc

[mhdawson]

I think we could get rid of linux-ppc and linux-ppc64 as we only support linux-ppc64le as long as that is what is used for linux-ppc64 builds.

[miladfarca]

@miladfarca can you check with Vascili on which of we need?

1.5M aix64-gcc-as 1.5M aix-gcc

I've forwarded the link to Vasili to confirm.
Assuming -maix64 is used during build with gcc (like how its done on the V8 side) then 64 bit headers might need to be used i.e aix64-gcc-as.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/43688/

[danbev]

Landed in 33bc537, 4f1f14e, d8365ac, d68c4e9, 3f0b2a2, and 4ecfe3f

[yanovich]

commit 7fae2c9 breaks C++ addons which use any openssl header:

$ yarn build 
yarn run v1.22.18
$ node-gyp configure --silent && node-gyp build --silent
make: Entering directory '/home/user/src/addon/build'
  CXX(target) Release/obj.target/addon/addon.o
In file included from /home/s/.cache/node-gyp/18.1.0/include/node/openssl/opensslconf.h:9,
                 from /home/s/.cache/node-gyp/18.1.0/include/node/openssl/macros.h:14,
                 from /home/s/.cache/node-gyp/18.1.0/include/node/openssl/evp.h:14,
                 from ../addon.cc:5:
/home/s/.cache/node-gyp/18.1.0/include/node/openssl/./opensslconf_asm.h:97:11: fatal error: ./archs/linux-x86_64/asm/include/openssl/opensslconf.h: No such file or directory
   97 | # include "./archs/linux-x86_64/asm/include/openssl/opensslconf.h"
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [addon.target.mk:120: Release/obj.target/addon/addon.o] Error 1
$ head -n 5 addon.cc
#include <stdlib.h>
#include <string.h>
#include <iostream>

#include <openssl/evp.h>

[juanarbol]

This is a dep of #42356, so, this can not be landed in v16.x