Compilation on Windows fails

[rhtlk891]

Hi,
I wasn't able to compile semla on a windows machine. I tried to fix some issues I encountered with the provided building instructions, however I didn't manage to get it compiled in the end. My setup is the following:

• Windows 11
• MSVC 2019 Developer Shell

Version strings of Perl, cmake and nasm:

• This is perl 5, version 22, subversion 0 (v5.22.0) built for MSWin32-x64-multi-thread
• cmake version 3.30.1
• NASM version 2.16.03 compiled on Apr 17 2024
  Don't know if it matters, but I adapted the PATH variable for the location of NASM only for the Powershell I used for the build process.

Furthermore:

• I checked out the master branch of the commit 3cc48d3e0a9b16010d570c518c98998873a8d6aa
• I create a build directory with a subdirectory openssl_keys
• Within openssl_keys I created the keys as described in the CMakeLists.txt in the src directory:

openssl genrsa -out "private_key_tool.pem" 4096
openssl genrsa -out "private_key_lve.pem" 4096
openssl rsa -pubout -in ".\private_key_tool.pem" -out ".\public_key_tool.pem"

I manually created the file public_key_tools.txt which contains only a single line of text:
public_key_tool.pem
The directory build\openssl_keys contains:

private_key_lve.pem
private_key_tool.pem
public_key_tool.pem
public_key_tools.txt

So far, so good. I cd to the build directory and tried to follow the build instructions provided in src\README, which appear to be incomplete. I had to add the key file path definitions to the cmake command to get it running without error:

cmake "C:\myprogs\repos2\semla\src" `
-DCMAKE_INSTALL_PREFIX="C:\myprogs\repos2\semla\build" `
-DDECRYPTOR="C:\myprogs\repos2\semla\src\decryptors\default" `
-DLICENSE_MANAGER="C:\myprogs\repos2\semla\src\license_managers\testingdummy" `
-DOBFUSCATOR="C:\myprogs\repos2\semla\src\obfuscators\dummy" `
-DTOOL_PRIVATE_KEY_DIRECTORY="C:\myprogs\repos2\semla\build\openssl_keys" `
-DLVE_KEYS_DIRECTORY="C:\myprogs\repos2\semla\build\openssl_keys" `
-DTOOLS_PUBLIC_KEYS_DIRECTORY="C:\myprogs\repos2\semla\build\openssl_keys" `
-G"NMake Makefiles"

You can infer my directory structure from the paths provided above. I cloned the repo to the folder C:\myprogs\repos2\semla s.t. C:\myprogs\repos2\semla\src is the semla source directory. You can ignore the backtics `, these are just needed for linebreaks in powershell.
Here's the output of the cmake command:
cmake_log.txt
It seems to work fine. There's some file path weirdness as slashes / and backslashes \ occur both. However, this doesn't seem to faze cmake as no error is thrown. I'm a bit puzzled why cmake assumes a 32bit target.
In the next step I tried to build the code via
cmake --build . --config Release --target install
and everything works fine until the compilation fails with

*** Installing runtime programs
Copying: apps//openssl.exe to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/bin/openssl.exe
Copying: apps//openssl.pdb to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/bin/openssl.pdb
Copying: tools//c_rehash.pl to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/bin/c_rehash.pl
created directory `C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/certs'
created directory `C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/private'
created directory `C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/misc'
Copying: ./apps/openssl.cnf to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/openssl.cnf.dist
Copying: ./apps/openssl.cnf to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/openssl.cnf
Copying: apps//CA.pl to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/misc/CA.pl
Copying: apps//tsget.pl to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/misc/tsget.pl
Copying: ./apps/ct_log_list.cnf to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/ct_log_list.cnf.dist
Copying: ./apps/ct_log_list.cnf to C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/ct_log_list.cnf
[ 11%] Completed 'openssl'
[ 11%] Built target openssl
[ 12%] Building C object obfuscators/obfuscator/CMakeFiles/obfuscator_module.dir/obfuscator.c.obj
obfuscator.c
[ 13%] Linking C static library obfuscator_module.lib
[ 13%] Built target obfuscator_module
[ 15%] Building C object CMakeFiles/obfuscate_utils.dir/embedfile/obfuscate_utils.c.obj
obfuscate_utils.c
[ 16%] Linking C static library obfuscate_utils.lib
[ 16%] Built target obfuscate_utils
[ 18%] Building C object CMakeFiles/obfuscate.dir/embedfile/obfuscate.c.obj
obfuscate.c
C:\myprogs\repos2\semla\src\embedfile\obfuscate.c(133): warning C4477: "fprintf": Die Formatzeichenfolge "%s" erfordert ein Argument vom Typ "char *", das variadic-Argument "1" weist aber den Typ "int" auf.
NMAKE : fatal error U1073: ""openssl\Program Files\OpenSSL\lib\libssl.lib"" konnte nicht erstellt werden
Stop.
NMAKE : fatal error U1077: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\nmake.exe"": R³ckgabe-Code "0x2"
Stop.
NMAKE : fatal error U1077: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\nmake.exe"": R³ckgabe-Code "0x2"
Stop.

So, openssl is successfully built. The file paths openssl is copied to (e.g. C:/myprogs/repos2/semla/build/openssl/myprogs/repos2/semla/build/openssl/bin/openssl.exe) are a bit weird to me, but I don't know whether that matters. Sth's wrong with openssl\Program Files\OpenSSL\lib\libssl.lib which is indeed not the file path the compiled openssl is copied to?
I can also post the whole stdout/stderr streams during compilation if that helps?
At the obfuscator module, the build fails. I tried to only build the obfuscator module but I got the same error:

PS C:\myprogrs\repos2\semla\build> cmake --build . --config Release --target obfuscate
[ 57%] Built target openssl
[ 71%] Built target obfuscator_module
[ 85%] Built target obfuscate_utils
NMAKE : fatal error U1073: ""openssl\Program Files\OpenSSL\lib\libssl.lib"" konnte nicht erstellt werden
Stop.
NMAKE : fatal error U1077: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\nmake.exe"": Rückgabe-Code "0x2"
Stop.
NMAKE : fatal error U1077: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\nmake.exe"": Rückgabe-Code "0x2"
Stop.
NMAKE : fatal error U1077: ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\nmake.exe"": Rückgabe-Code "0x2"
Stop.

So, to sum up:

• The provided cmake commands in /src/README appear to be incomplete for the windows build. Would be great if you could take a look at that.
• I tried to follow the build instructions to my best knowledge, but couldn't get it done. I am by no means a cmake expert and it could be due to a very silly mistake. Nevertheless, help would be greatly appreciated!

[axelmartenssonmodelon]

Hi,
Thank you for the detailed bug report 👍. I can reproduce the error. Let me get back to you once I have found a fix.

[rhtlk891]

I could narrow down the error a bit:

• 64bit compilation works fine.
• For some reason I don't understand, my windows 11 opens the developer shell default-wise in 32bit mode s.t. cmake follows the 32bit branch for building. I guess it's related to some recent update or other windows weirdness. I'm quite sure that this wasn't the case before.
• You can check whether your developer shell is 32 or 64bit with the command [IntPtr]::size which returns the size of a int pointer (8 for 64bit, 4 for 32bit)
• How to start a devloper-shell with a defined bitness is described here.
• I found a workaround for the 32bit compilation:
  Assume the build directory is C:\myprogs\repos2\semla\build

1. Build openssl with cmake --build . --config Release --target openssl
2. The compiled openssl files will be located at some weird mixed up file path: C:\myprogs\repos2\semla\build\openssl\\myprogs\repos2\semla\build\openssl. Copy the content of this folder into a new folder C:\myprogs\repos2\semla\build\openssl\Program Files\OpenSSL (this is the location where it's copied to for the 64bit build).
3. Build the rest of the project with cmake --build . --config Release --target install (btw: I needed an administrator Powershell for this command to run smoothly)

So I think it boils down to a file path issue. Openssl builds but is copied to the wrong location. My guess would be that the 32bit branch in BuildSemlaOpenSSL.cmake is somehow flawed. I'm not sure, but maybe the values for --openssldir=. --prefix=${openssl_dir} are not correct?

include(ExternalProject)

set(openssl_dir ${CMAKE_CURRENT_BINARY_DIR}/openssl)
# - OS
if(WIN32)
    # Values for Windows
    if(BITS64)
        if (MSVC)
            set(openssl_conf_cmd    perl Configure VC-WIN64A)
            set(openssl_conf_extra  perl configdata.pm --dump)
        else()
            set(openssl_conf_cmd    perl Configure mingw64)
            set(openssl_conf_extra  make depend)
        endif()
    else()
        if (MSVC)
           set(openssl_conf_cmd    perl Configure no-shared no-idea no-mdc2 no-rc5 --openssldir=. --prefix=${openssl_dir} VC-WIN32 )
            set(openssl_conf_extra  perl configdata.pm --dump)
        else()
            set(openssl_conf_cmd    perl Configure mingw)
            set(openssl_conf_extra  make depend)
        endif()
    endif()

[... remainder omitted ...]

[axelmartenssonmodelon]

Yes, I also think that the problem is probably located in the 32-bit branch in BuildSemlaOpenSSL.cmake. Thank you for looking into it.

Does using the 64-bit version work for you, or do you need the 32-bit version? We are considering removing the 32-bit version.

If the 64-bit version works for you, then I will close this issue.