Cross compiling with a native dependency from a subproject #10947

yotamN · 2022-10-23T19:42:05Z

Describe the bug
I'm not sure of this is the same as #8043 because it says something about CMake there. But, I can't seem to compile a native dependency that is fetched using the wrap system when cross compiling to a different system. It will build the subproject but say nothing was found. Here is the full log:

log

Build started at 2022-10-23T22:39:00.864711
Main binary: /usr/bin/python3
Build Options: '--cross-file x86_64-w64-mingw32.txt'
Python system: Linux
The Meson build system
Version: 0.62.2
Source dir: /home/yotam/Projects/Frida/test-meson-cc
Build dir: /home/yotam/Projects/Frida/test-meson-cc/build-mingw
Build type: cross build
Project name: tutorial
Project version: undefined
Sanity testing C compiler: x86_64-w64-mingw32-gcc
Is cross compiler: True.
Sanity check compiler command line: x86_64-w64-mingw32-gcc sanitycheckc.c -o sanitycheckc_cross.exe -D_FILE_OFFSET_BITS=64 -c
Sanity check compile stdout:

Sanity check compile stderr:

C compiler for the host machine: x86_64-w64-mingw32-gcc (gcc 11.2.1 "x86_64-w64-mingw32-gcc (GCC) 11.2.1 20211019 (Fedora MinGW 11.2.1-5.fc36)")

C linker for the host machine: x86_64-w64-mingw32-gcc ld.bfd 2.37-4

Compiler for language c for the build machine not found.

Build machine cpu family: x86_64

Build machine cpu: x86_64

Host machine cpu family: x86_64

Host machine cpu: x86_64

Target machine cpu family: x86_64

Target machine cpu: x86_64

Pkg-config binary for 0 is not cached.

Pkg-config binary missing from cross or native file, or env var undefined.

Trying a default Pkg-config fallback at pkg-config

Found pkg-config: /usr/bin/pkg-config (1.8.0)

Determining dependency 'quickjs' with pkg-config executable '/usr/bin/pkg-config'

env[PKG_CONFIG_PATH]:

Called /usr/bin/pkg-config --modversion quickjs -> 1
CMake binary for 0 is not cached

CMake binary missing from cross or native file, or env var undefined.

Trying a default CMake fallback at cmake

Did not find CMake 'cmake'

Found CMake: NO

CMake binary for machine 0 not found. Giving up.

Build-time dependency quickjs found: NO (tried pkgconfig and cmake)

Looking for a fallback subproject for the dependency quickjs
Executing subproject quickjs
Project name: quickjs

Project version: 2021-03-27-frida

C compiler for the host machine: x86_64-w64-mingw32-gcc (gcc 11.2.1 "x86_64-w64-mingw32-gcc (GCC) 11.2.1 20211019 (Fedora MinGW 11.2.1-5.fc36)")

C linker for the host machine: x86_64-w64-mingw32-gcc ld.bfd 2.37-4

Compiler for language c for the build machine not found.

Running compile:

Working directory:  /tmp/tmp1kaiic84

Command line:  x86_64-w64-mingw32-gcc /tmp/tmp1kaiic84/testfile.c -E -P -D_FILE_OFFSET_BITS=64 -P -O0
Code:
    #ifndef EMSCRIPTEN
    # define EMSCRIPTEN
    #endif
    "MESON_GET_DEFINE_DELIMITER"

EMSCRIPTEN

Compiler stdout:

"MESON_GET_DEFINE_DELIMITER"
Compiler stderr:
Fetching value of define "EMSCRIPTEN" :

Run-time dependency threads found: YES

Running compile:

Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpker8zrfv

Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpker8zrfv/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpker8zrfv/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -ldl -Wl,--end-group -Wl,--allow-shlib-undefined
Code:

int main(void) { return 0; }
Compiler stdout:
Compiler stderr:

/usr/lib/gcc/x86_64-w64-mingw32/11.2.1/../../../../x86_64-w64-mingw32/bin/ld: cannot find -ldl

collect2: error: ld returned 1 exit status
Library dl found: NO

Running compile:

Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpldfytmqy

Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpldfytmqy/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpldfytmqy/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lm -Wl,--end-group -Wl,--allow-shlib-undefined
Code:

int main(void) { return 0; }
Compiler stdout:
Compiler stderr:
Library m found: YES

Running compile:

Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpvmhuuf0s

Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpvmhuuf0s/testfile.c -E -P -D_FILE_OFFSET_BITS=64 -P -O0
Code:
    #ifdef __has_include
     #if !__has_include("unistd.h")
      #error "Header 'unistd.h' could not be found"
     #endif
    #else
     #include <unistd.h>
    #endif

Compiler stdout:
Compiler stderr:
Has header "unistd.h" : YES

Running compile:

Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp6zy6bzan

Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp6zy6bzan/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp6zy6bzan/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
Code:
    #define malloc_usable_size meson_disable_define_of_malloc_usable_size
    
    #include <limits.h>
    #undef malloc_usable_size
    
    #ifdef __cplusplus
    extern "C"
    #endif
    char malloc_usable_size (void);
    
    #if defined __stub_malloc_usable_size || defined __stub___malloc_usable_size
    fail fail fail this function is not going to work
    #endif
    
    int main(void) {
      return malloc_usable_size ();
    }

Compiler stdout:
Compiler stderr:

/usr/lib/gcc/x86_64-w64-mingw32/11.2.1/../../../../x86_64-w64-mingw32/bin/ld: /tmp/ccAw8AeD.o:testfile.c:(.text+0xe): undefined reference to `malloc_usable_size'

collect2: error: ld returned 1 exit status
Running compile:

Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp87ap1jgm

Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp87ap1jgm/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp87ap1jgm/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
Code:
    int main(void) {

    /* With some toolchains (MSYS2/mingw for example) the compiler
     * provides various builtins which are not really implemented and
     * fall back to the stdlib where they aren't provided and fail at
     * build/link time. In case the user provides a header, including
     * the header didn't lead to the function being defined, and the
     * function we are checking isn't a builtin itself we assume the
     * builtin is not functional and we just error out. */
    #if !1 && !defined(malloc_usable_size) && !0
        #error "No definition for __builtin_malloc_usable_size found in the prefix"
    #endif

    #ifdef __has_builtin
        #if !__has_builtin(__builtin_malloc_usable_size)
            #error "__builtin_malloc_usable_size not found"
        #endif
    #elif ! defined(malloc_usable_size)
        __builtin_malloc_usable_size;
    #endif
    return 0;
    }

Compiler stdout:
Compiler stderr:

/home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp87ap1jgm/testfile.c: In function 'main':

/home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp87ap1jgm/testfile.c:17:18: error: #error "__builtin_malloc_usable_size not found"

17 |                 #error "__builtin_malloc_usable_size not found"

|                  ^~~~~
Checking for function "malloc_usable_size" : NO

Configuring config.h using configuration

Build targets in project: 1

Subproject quickjs finished.
WARNING: Subproject 'quickjs' did not override 'quickjs' dependency and no variable name specified

Dependency quickjs from subproject subprojects/quickjs found: NO
meson.build:2:0: ERROR: Dependency 'quickjs' is required but not found.

To Reproduce
A minimal meson.build for example:

project('example', 'c')
native_dep = dependency('quickjs', native: true)
executable('example', 'main.c', dependencies: native_dep)

with a subproject containing quickjs from Frida.
Just need to cross compile it.

Expected behavior
The expected behavior of course would be that it would work just like without cross compilation.

system parameters

This is a cross build
Using Fedora 36
Using Python 3.10.7
Using Meson 0.62.2
Using Ninja 1.10.2

The text was updated successfully, but these errors were encountered:

eli-schwartz · 2022-10-23T20:35:47Z

The quickjs subproject overrides a cross version of the dependency. But you tried to look up a native version of it.

yotamN · 2022-10-24T04:16:57Z

You are right, I forgot the native flag for the executable but even with it it doesn't work? This is the updated meson example:

project('tutorial', 'c')
native_dep = dependency('quickjs', native: true)
executable('demo', 'main.c', native: true, dependencies: native_dep)

and the logs:

Build started at 2022-10-24T07:15:09.143767
Main binary: /usr/bin/python3
Build Options: '--cross-file x86_64-w64-mingw32.txt'
Python system: Linux
The Meson build system
Version: 0.62.2
Source dir: /home/yotam/Projects/Frida/test-meson-cc
Build dir: /home/yotam/Projects/Frida/test-meson-cc/build-mingw
Build type: cross build
Project name: tutorial
Project version: undefined
Sanity testing C compiler: x86_64-w64-mingw32-gcc
Is cross compiler: True.
Sanity check compiler command line: x86_64-w64-mingw32-gcc sanitycheckc.c -o sanitycheckc_cross.exe -D_FILE_OFFSET_BITS=64 -c
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
C compiler for the host machine: x86_64-w64-mingw32-gcc (gcc 11.2.1 "x86_64-w64-mingw32-gcc (GCC) 11.2.1 20211019 (Fedora MinGW 11.2.1-5.fc36)")
C linker for the host machine: x86_64-w64-mingw32-gcc ld.bfd 2.37-4
Sanity testing C compiler: cc
Is cross compiler: False.
Sanity check compiler command line: cc sanitycheckc.c -o sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/sanitycheckc.exe
C compiler for the build machine: cc (gcc 12.2.1 "cc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
C linker for the build machine: cc ld.bfd 2.37-36
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
Pkg-config binary for 0 is not cached.
Pkg-config binary missing from cross or native file, or env var undefined.
Trying a default Pkg-config fallback at pkg-config
Found pkg-config: /usr/bin/pkg-config (1.8.0)
Determining dependency 'quickjs' with pkg-config executable '/usr/bin/pkg-config'
env[PKG_CONFIG_PATH]: 
Called `/usr/bin/pkg-config --modversion quickjs` -> 1

CMake binary for 0 is not cached
CMake binary missing from cross or native file, or env var undefined.
Trying a default CMake fallback at cmake
Did not find CMake 'cmake'
Found CMake: NO
CMake binary for machine 0 not found. Giving up.
Build-time dependency quickjs found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency quickjs

Executing subproject quickjs 

Project name: quickjs
Project version: 2021-03-27-frida
C compiler for the host machine: x86_64-w64-mingw32-gcc (gcc 11.2.1 "x86_64-w64-mingw32-gcc (GCC) 11.2.1 20211019 (Fedora MinGW 11.2.1-5.fc36)")
C linker for the host machine: x86_64-w64-mingw32-gcc ld.bfd 2.37-4
C compiler for the build machine: cc (gcc 12.2.1 "cc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
C linker for the build machine: cc ld.bfd 2.37-36
Running compile:
Working directory:  /tmp/tmpk_2opchy
Command line:  x86_64-w64-mingw32-gcc /tmp/tmpk_2opchy/testfile.c -E -P -D_FILE_OFFSET_BITS=64 -P -O0 

Code:
 
        
        #ifndef EMSCRIPTEN
        # define EMSCRIPTEN
        #endif
        "MESON_GET_DEFINE_DELIMITER"
EMSCRIPTEN
Compiler stdout:
         "MESON_GET_DEFINE_DELIMITER"


Compiler stderr:
 
Fetching value of define "EMSCRIPTEN" :  
Run-time dependency threads found: YES
Running compile:
Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpkj7t_b6u
Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpkj7t_b6u/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpkj7t_b6u/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -ldl -Wl,--end-group -Wl,--allow-shlib-undefined 

Code:
 int main(void) { return 0; }

Compiler stdout:
 
Compiler stderr:
 /usr/lib/gcc/x86_64-w64-mingw32/11.2.1/../../../../x86_64-w64-mingw32/bin/ld: cannot find -ldl
collect2: error: ld returned 1 exit status

Library dl found: NO
Running compile:
Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp9mjyouyl
Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp9mjyouyl/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmp9mjyouyl/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lm -Wl,--end-group -Wl,--allow-shlib-undefined 

Code:
 int main(void) { return 0; }

Compiler stdout:
 
Compiler stderr:
 
Library m found: YES
Running compile:
Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmprmkf56jh
Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmprmkf56jh/testfile.c -E -P -D_FILE_OFFSET_BITS=64 -P -O0 

Code:
 
        #ifdef __has_include
         #if !__has_include("unistd.h")
          #error "Header 'unistd.h' could not be found"
         #endif
        #else
         #include <unistd.h>
        #endif
Compiler stdout:
 
Compiler stderr:
 
Has header "unistd.h" : YES 
Running compile:
Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpzlybe2y9
Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpzlybe2y9/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmpzlybe2y9/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group 

Code:
 
        #define malloc_usable_size meson_disable_define_of_malloc_usable_size
        
        #include <limits.h>
        #undef malloc_usable_size
        
        #ifdef __cplusplus
        extern "C"
        #endif
        char malloc_usable_size (void);
        
        #if defined __stub_malloc_usable_size || defined __stub___malloc_usable_size
        fail fail fail this function is not going to work
        #endif
        
        int main(void) {
          return malloc_usable_size ();
        }
Compiler stdout:
 
Compiler stderr:
 /usr/lib/gcc/x86_64-w64-mingw32/11.2.1/../../../../x86_64-w64-mingw32/bin/ld: /tmp/ccOHMGGy.o:testfile.c:(.text+0xe): undefined reference to `malloc_usable_size'
collect2: error: ld returned 1 exit status

Running compile:
Working directory:  /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmppv4hk3us
Command line:  x86_64-w64-mingw32-gcc /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmppv4hk3us/testfile.c -o /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmppv4hk3us/output.exe -D_FILE_OFFSET_BITS=64 -O0 -Wl,--start-group -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group 

Code:
 
        int main(void) {

        /* With some toolchains (MSYS2/mingw for example) the compiler
         * provides various builtins which are not really implemented and
         * fall back to the stdlib where they aren't provided and fail at
         * build/link time. In case the user provides a header, including
         * the header didn't lead to the function being defined, and the
         * function we are checking isn't a builtin itself we assume the
         * builtin is not functional and we just error out. */
        #if !1 && !defined(malloc_usable_size) && !0
            #error "No definition for __builtin_malloc_usable_size found in the prefix"
        #endif

        #ifdef __has_builtin
            #if !__has_builtin(__builtin_malloc_usable_size)
                #error "__builtin_malloc_usable_size not found"
            #endif
        #elif ! defined(malloc_usable_size)
            __builtin_malloc_usable_size;
        #endif
        return 0;
        }
Compiler stdout:
 
Compiler stderr:
 /home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmppv4hk3us/testfile.c: In function 'main':
/home/yotam/Projects/Frida/test-meson-cc/build-mingw/meson-private/tmppv4hk3us/testfile.c:17:18: error: #error "__builtin_malloc_usable_size not found"
   17 |                 #error "__builtin_malloc_usable_size not found"
      |                  ^~~~~

Checking for function "malloc_usable_size" : NO 
Configuring config.h using configuration
Build targets in project: 1
Subproject quickjs finished.

WARNING: Subproject 'quickjs' did not override 'quickjs' dependency and no variable name specified
Dependency quickjs from subproject subprojects/quickjs found: NO

meson.build:2:0: ERROR: Dependency 'quickjs' is required but not found.

xclaesse · 2022-10-25T17:13:35Z

The native kwarg is not respected when doing a fallback, so it cross compiled quickjs and you can't use that dependency to build a native executable. That's a known limitation of Meson that I really would like to fix too.

yotamN · 2022-11-08T19:58:11Z

I hope I'm not in way over my head, but could you point me to what prevent from using the same cross compilation configuration for the subprojects? I would like to take a look at it if you think it's possible for someone without experience in the Meson codebase

v1993 · 2023-11-18T09:51:18Z

On a related note: whenever this is implemented, I'd appreciate if native subprojects would not install as well, since the only (I think?) reason to have something native is to use it exclusively during build.

Meson currently doesn't support subprojects to be native and non-native at the same time. See: mesonbuild/meson#10947 Unfortunately, sdb depends on rz_util which in turn depends on PCRE2. Excluding PCRE2 from the native build makes linking of rz_util not possible anymore. Adding it, will make Meson complain that the dependencies cannot be mixed. Hence, we compile a copy of PCRE2 for the native build if required.

* Replace OpenBSD regex library with PCRE2. PCRE2 has way better performance than the OpenBSD library (something around 20 times faster). The following flags are enabled for every pattern: - PCRE2_UTF - PCRE2_MATCH_INVALID_UTF - PCRE2_NO_UTF_CHECK All the others are optional. Changes made: - Adds PCRE2 as subproject. - Changes the API away from POSIX to PCRE2. - Edits many regex patterns because: - ' ' is skipped in patterns, if the EXTENDED flag is set for matching. '\s' must be set now. - '.' doesn't match newlines by default. - Changes the API so matches and their groups are bundled into PVectors. - Moves the regex component to rz_util. * Fix cross build - add copy of PCRE2 dependecy Meson currently doesn't support subprojects to be native and non-native at the same time. See: mesonbuild/meson#10947 Unfortunately, sdb depends on rz_util which in turn depends on PCRE2. Excluding PCRE2 from the native build makes linking of rz_util not possible anymore. Adding it, will make Meson complain that the dependencies cannot be mixed. Hence, we compile a copy of PCRE2 for the native build if required.

Where we cannot use the same subproject state for both machines. We may not even need to use a subproject for both machines, for example if the build machine has a dependency installed, and we only need to fall back to a subproject for the host machine. Fixes: mesonbuild#10947

oleavr · 2024-02-06T23:07:27Z

I'm currently working on solving this. As of this evening it works as long as a subproject is only needed for either MachineChoice.BUILD or MachineChoice.HOST, but not both at the same time. (One can be using the subproject while the other uses an external dependency, though.) But I'm hopeful that I'm close to having it working properly, and will open a PR once I get to that point 🤞

Where we cannot use the same subproject state for both machines. We may not even need to use a subproject for both machines, for example if the build machine has a dependency installed, and we only need to fall back to a subproject for the host machine. Fixes: mesonbuild#10947

Nomura-RH mentioned this issue Aug 10, 2023

Start adding native : 'both' #12022

Open

Nomura-RH mentioned this issue Sep 16, 2023

"native: true" not passed to dependency fallback #12245

Open

oleavr mentioned this issue Feb 7, 2024

Fix subprojects logic in cross build scenarios #12822

Closed

Rot127 mentioned this issue Feb 18, 2024

Wrong default name of wrap directory? #7330

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross compiling with a native dependency from a subproject #10947

Cross compiling with a native dependency from a subproject #10947

yotamN commented Oct 23, 2022

eli-schwartz commented Oct 23, 2022

yotamN commented Oct 24, 2022

xclaesse commented Oct 25, 2022

yotamN commented Nov 8, 2022

v1993 commented Nov 18, 2023

oleavr commented Feb 6, 2024 •

edited

Loading

Cross compiling with a native dependency from a subproject #10947

Cross compiling with a native dependency from a subproject #10947

Comments

yotamN commented Oct 23, 2022

eli-schwartz commented Oct 23, 2022

yotamN commented Oct 24, 2022

xclaesse commented Oct 25, 2022

yotamN commented Nov 8, 2022

v1993 commented Nov 18, 2023

oleavr commented Feb 6, 2024 • edited Loading

oleavr commented Feb 6, 2024 •

edited

Loading