Meson - a new shining star in scientific python community

[alexlib]

scipy, scikit-image and apparently numpy are moving to meson build system

Shall we try as well ? @timdewhirst @ErichZimmer

https://labs.quansight.org/blog/2021/07/moving-scipy-to-meson

[timdewhirst]

might be interesting - cmake is full of magic, and the integration with distutils and setup.py is pretty odd, so might be worth taking a look.

[ErichZimmer]

I tried meson earlier this year for an unrelated python project with a c++ backend, but I had troubles on Windows (the only platform to fail CI testing). Additionally, I don't know how to use vcpkg with it, unless adding the third party packages through GitHub Submodules and python scripts. However, I really do like it a lot, especially with GCC on Windows. It is very flexible and a possible cmake replacement. Perhaps, I'll give it another shot.

[eli-schwartz]

What kind of troubles?

[ErichZimmer]

@eli-schwartz I do not remember what exactly happened, but it was something about how the python source files were installed. This could be, and likely is, attributed to user error, but I do not know where as I copied how scipy used meson in my project. I'll try again this Sunday with some dummy projects.

[eli-schwartz]

Cool, feel free to ping me if you have any issues.

Also a few things have become easier to do based on discussions SciPy had with the Meson core team while implementing their build, so it may be worth checking out pymod.find_installation(pure: false) and py.install_sources(..., preserve_path: true) as both relate to installing source files.

[ErichZimmer]

@eli-schwartz
I played around with the meson build system for a short while, but I am having trouble exporting symbols on Windows due to c++ name mangling. On cmake, windows_export_all_symbols is used and the dynamic libraries are linked using the .lib file. However, I do not know how to generate this .lib file for MSVC or .a file for GCC for linking purposes. I noticed a few GH issues have been raised, such as GH issue #2132, GH issue 4298, and GH discussion #10822, but I have not yet employed any techniques discussed in these subforums and won't be able to do so for some time. With your vast knowledge of the meson build system, is there any tips you can share for the generation of the link file for the dynamic libraries? If necessary, I won't mind adding prepcocessor directives to the openpiv-c-qt project for meson compatibility.

[eli-schwartz]

One way is to add __declspec(dllexport) to the front of public functions, usually by a macro such as

#if defined _WIN32 || defined __CYGWIN__
    #if defined BUILDING_DLL
        #define DLL_PUBLIC __declspec(dllexport)
    #else
        #define DLL_PUBLIC __declspec(dllimport)
    #endif
#else 
    // unix code can go here
#endif

In your case, all functions are are considered public, right? Since unix compiler toolchains default to "EXPORT_ALL_SYMBOLS", this means you don't need any unix code.

---

Another way is to add a file called openpiv_cxx.def that contains the following:

EXPORTS
	name_of_func
	name_of_other_func

and list every single function extracted by running dumpbin /symbols *.o on all object files for example using something like this perl script: https://github.com/anarazel/postgres/blob/a2f0e678474234d4c6d7f88e1d533ee4d0d8b1de/src/tools/msvc/gendef.pl#L32

You'd check the openpiv_cxx.def file into git and use it in meson as:

library('mylib', 'foo.c', 'bar.c', vs_module_def: 'openpiv_cxx.def')

However due to CXX name mangling the precise symbol names might periodically change for reasons less straightforward than just adding/removing entire functions. The ideal thing to do here is to teach meson how to internally inject a build rule after building all the objects, to run dumpbin itself and generate this .def file on the fly before performing the final link, thus ensuring that everything is in sync.

[ErichZimmer]

So, if I tailor the build script towards MinGW on Windows (which is recommended due to the use of vector extensions for simd), then I don't have to worry about exporting name-mangled symbols since GCC and CLANG does this by default? If so, then I will delve more into MinGW for Windows compilation since I dislike MSVC anyways.

[eli-schwartz]

https://sourceware.org/binutils/docs/ld/WIN32.html#index-exporting-DLL-symbols

The cygwin/mingw ld has several ways to export symbols for dll’s.

By default ld exports symbols with the auto-export functionality, which is controlled by the following command-line options:

• –export-all-symbols [This is the default]

If ‘--export-all-symbols’ is not given explicitly on the command line, then the default auto-export behavior will be disabled if either of the following are true:

• A DEF file is used.
• Any symbol in any object file was marked with the __declspec(dllexport) attribute.

[eli-schwartz]

The mingw functionality does seem sort of sane 😜 because the alternative is to do like MSVC and create a shared DLL that has zero symbols and is, effectively, totally useless.

[ErichZimmer]

Thanks for your insight on using the meson build system. After 15 minutes of free time and playing around with meson, I successfully compiled libopenpiv and produced an import lib. Meson is truly a miracle, especially for those who have limited experience with programming and build systems.

P.S., The dependency system is awesome! After setting up wrap files for google-benchmark, Catch2, cxxopts, fmt, and pybind11, things ran quite smoothly.

[ErichZimmer]

@eli-schwartz
Is there a way to bundle the dynamic libraries from dependencies to a specified location in the build directory? Basically, I want to place all dependency dynamic libraries, perhaps even test and example executables, in a single folder due to relative path shenanigans on Windows. On CMake, everything is placed in an out folder in the build directory which makes things simple. Perhaps I missed something in the filesystem module or something related in the meson reference manuals? Meson github issues and discussions gave me some ideas, but they did not help much in my particular case.

@timdewhirst
After compiling openpiv-c--qt with GCC 10.3 on Windows, the correlation now takes only 6.80 μs per 32x32 interrogation windows (mean of 10 runs with window step of 2 pixels). If I remember right, that was just about, if not faster, than a FFTW3 implementation I made earlier this year.
On a side note, I would hopefully have the meson build system fully setup for OpenPIV-c--qt done soon along with the other enhancements (if university and work doesn't get too rough).

CPU: Intel i5-1235U 1.30 GHz

compiler	average time per 32x32 px IW (μs )	improvement
MSVC (release)	9.66	0.0%
GCC (debug-optimize)	8.17	15.4%
GCC SSE2 (debug-optimize)	6.84	29.2%
GCC AVX (debug-optimize)	6.73	30.3%
GCC AVX2 (debug-optimize)	6.76	30.0%
GCC AVX2 (release)	5.85	39.4%

I have no idea why my computer caps at SSE2 intrinsics.