cmake: Switch to libsecp256k1 upstream build system #192

hebasto · 2024-05-07T14:23:08Z

To test this PR, one can compare the ouputs of the following commands:

Autotools:

$ make -C src/secp256k1 libsecp256k1.la V=1

CMake:

$ cmake --build build -t secp256k1 --verbose

Also sets of the libsecp256k1-specific tests can be compared between make check and ctest.

src/CMakeLists.txt

hebasto · 2024-05-07T14:35:57Z

For CMake-based projects, it is not common to print the summary at the end of the configuration stage.

Printing summaries by subprojects--libsecp256k1 in this PR and minisketch in the future--makes them less useful for the user due to their verbosity.

hebasto · 2024-05-07T16:15:13Z

cc @real-or-random @theuni @laanwj @fanquake

fanquake · 2024-05-08T08:09:38Z

For CMake-based projects, it is not common to print the summary at the end of the configuration stage.

I'd say that atleast half of the projects we are (going to) build in depends with CMake print some sort of configuration? So I don't think it's uncommon.

Printing summaries by subprojects--libsecp256k1 in this PR and minisketch in the future--makes them less useful for the user due to their verbosity.

I'm not exactly sure what you're trying to say here. It's not less useful if you actually want to see how things have been configured / are going to be compiled, which, if the plan is to use the CMake build systems of our subtrees, we definitely do. This is useful for developers, in CI etc etc.

theuni · 2024-05-08T17:20:06Z

src/CMakeLists.txt

+option(SECP256K1_BUILD_TESTS "" OFF)
+option(SECP256K1_BUILD_EXHAUSTIVE_TESTS "" OFF)
+option(SECP256K1_BUILD_CTIME_TESTS "" OFF)
+add_subdirectory(secp256k1)


Does this need EXCLUDE_FROM_ALL to keep it from installing?

Nope :)

Thanks to:

bitcoin/src/secp256k1/CMakeLists.txt

Line 52 in 43a66c5

option(SECP256K1_INSTALL "Enable installation." ${PROJECT_IS_TOP_LEVEL})

I think it still makes sense to go ahead and set that option though, since this is what it's meant for. We could remove the SECP256K1_BUILD_* options and not have to worry about any new binaries being added in the future.

Thanks! Reworked.

See also bitcoin-core/secp256k1#1545 (comment).

real-or-random

Concept ACK

The libsecp256k1 config looks good to me.

hebasto · 2024-05-25T12:51:23Z

Rebased on top of the #93.

hebasto · 2024-06-17T08:06:41Z

Friendly ping @TheCharlatan @real-or-random @theuni @fanquake :)

hebasto · 2024-06-18T13:38:40Z

Rebased.

real-or-random · 2024-06-20T07:57:06Z

src/CMakeLists.txt

+set(SECP256K1_ECMULT_GEN_KB 86 CACHE BOOL "" FORCE)
+include(GetTargetInterface)
+get_target_interface(core_c_flags "" core_base_interface COMPILE_OPTIONS)
+set(SECP256K1_LATE_CFLAGS ${core_c_flags} CACHE STRING "" FORCE)


What are the flags added on various platforms?

The core_c_flags variable holds the COMPILE_OPTIONS property of the core_base_interface library, which contains:

for MinGW:

bitcoin/CMakeLists.txt

Lines 285 to 287 in d7e2989

# Avoid the use of aligned vector instructions when building for Windows.

# See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412.

try_append_cxx_flags("-Wa,-muse-unaligned-vector-move" TARGET core_base_interface SKIP_LINK)

There are no compile options that are specific to other platforms.

Also it includes sanitizer flags:

bitcoin/CMakeLists.txt

Lines 324 to 327 in d7e2989

try_append_cxx_flags("-fsanitize=${SANITIZERS}" TARGET core_base_interface

RESULT_VAR cxx_supports_sanitizers

SKIP_LINK

)

The core_c_flags variable holds the COMPILE_OPTIONS property of the core_base_interface library, which contains:
There are no compile options that are specific to other platforms.

Why are we adding that flag specifically, into cflags, for mingw, and no other similar flags on different platforms?

re offline discussion: why do we have a core_c_flags variable, if we aren't otherwise loading or using a c compiler (and how are things ending up in it).

And these are cxx_flags, so they shouldn't apply to libsecp256k1? Or is it just that try_append_cxx_flags is a bit of a misleading name (in the end it's just COMPILE_OPTIONS?) I'm not sure after looking at https://github.com/hebasto/bitcoin/blob/cmake-staging/cmake/module/TryAppendCXXFlags.cmake .

Anyway, I wonder if this flag should be part of the core_base_interface. Just because it's useful for core, doesn't mean we should pass it also when building the dependencies.

re offline discussion: why do we have a core_c_flags variable, if we aren't otherwise loading or using a c compiler (and how are things ending up in it).

Not sure if this answers your question, but the variable is defined only in the get_target_interface(core_c_flags ... It's just a local variable in this file and unset further below.

Or is it just that try_append_cxx_flags is a bit of a misleading name (in the end it's just COMPILE_OPTIONS?) I'm not sure after looking at cmake-staging/cmake/module/TryAppendCXXFlags.cmake .

I still think that this is a bit confusing. Okay, the sanitizer flags are typically the same for the C++ compiler and the C compiler. If we make this assumption, would it be a good idea to state it explicitly? Something like this:

# -fsanitize and related flags apply to both C++ and C, so we can pass them down to libsecp256k1 as CFLAGS get_target_interface(core_sanitizier_cxx_flags "" sanitize_interface COMPILE_OPTIONS) set(SECP256K1_LATE_CFLAGS ${core_sanitizer_cxx_flags} CACHE STRING "" FORCE)

Thanks! Reworked.

hebasto · 2024-06-26T21:42:06Z

Considering the recent discussion, which was nicely summarized in #192 (comment), it seems reasonable to proceed in this PR with enabling the secp256k1 tests as it is done in the master branch.

Two new commits did this job.

hebasto · 2024-06-28T11:24:18Z

Addressed @real-or-random's #192 (comment).

This change mirrors behavior in the master branch.

This change ensures that both summaries—secp256k1's and Bitcoin Core's— are displayed without interleaving of the `find_package(Doxygen)` output from the `doc` subdirectory.

hebasto · 2024-06-29T09:22:36Z

Rebased to resolve a conflict.

real-or-random

utACK 458d54c as far as the secp256k1 config is concerned

hebasto · 2024-06-29T10:34:38Z

I've updated the PR descriptions with possible ways to test this PR.

dc32026 cmake: Fix `clang-tidy` CI task (Hennadii Stepanov) Pull request description: #192 introduced a regression in the "clang-tidy" CI task. For instance, see https://cirrus-ci.com/task/6601636905222144 from bitcoin#29790. The reason is that disabling `EXPORT_COMPILE_COMMANDS` for `secp256k1` target only is not enough because the subtree's build system has other intermediate build targets internally. ACKs for top commit: TheCharlatan: ACK dc32026 Tree-SHA512: 134e7106c18abd7f07c15742edd73a16847f09452646ed4408d8cd35f7bca9f8501028f94b718720720c7ecc330a91dfd94680c091d010be88952ce4f4ef4c4e

fanquake · 2024-07-01T10:23:39Z

CMakeLists.txt

@@ -599,6 +580,9 @@ if(DEFINED ENV{LDFLAGS})
  deduplicate_flags(CMAKE_EXE_LINKER_FLAGS)
 endif()

+if(BUILD_TESTS)


Why is this needed for secp256k1 specifically?

It is not secp256k1-specific change. As a part of the "[FIXUP] cmake: Enable tests before adding subdirectories" commit this change adjusts the moment when enable_testing() being called. Previously, it was done implicitly via include(CTest) command after processing the src subdirectory. Now it done explicitly and before processing the src subdirectory. Which activates any add_test commands within the src subdirectory including the src/secp256k1 subdirectory as well.

fanquake · 2024-07-01T10:26:10Z

src/CMakeLists.txt

+set(SECP256K1_BUILD_CTIME_TESTS OFF CACHE BOOL "" FORCE)
+set(SECP256K1_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
+include(GetTargetInterface)
+# -fsanitize and related flags apply to both C++ and C,


Shouldn't this also be passing down LDFLAGS? Or is that being done somwhere else?

Shouldn't this also be passing down LDFLAGS?

Why? libsecp256k1 is a static library, and no linker is used to produce it, right?

See bitcoin#30791 (comment).

fanquake · 2024-07-01T10:28:45Z

src/CMakeLists.txt

+get_target_interface(core_sanitizer_cxx_flags "" sanitize_interface COMPILE_OPTIONS)
+set(SECP256K1_LATE_CFLAGS ${core_sanitizer_cxx_flags} CACHE STRING "" FORCE)
+unset(core_sanitizer_cxx_flags)
+# We want to build libsecp256k1 with the most tested RelWithDebInfo configuration.


We want to build libsecp256k1 with the most tested RelWithDebInfo configuration.

Not sure this comment adds much value, and the CMake config is not really the most relevant thing? I'd say what matters more is the secp256k1 configuration, which currently, diverges from upstream, and isn't extensively tested.

Can you suggest an improved comment or other approach?

fanquake · 2024-07-01T10:32:02Z

src/CMakeLists.txt

+set(SECP256K1_BUILD_BENCHMARK OFF CACHE BOOL "" FORCE)
+set(SECP256K1_BUILD_TESTS ${BUILD_TESTS} CACHE BOOL "" FORCE)
+set(SECP256K1_BUILD_EXHAUSTIVE_TESTS ${BUILD_TESTS} CACHE BOOL "" FORCE)
+set(SECP256K1_BUILD_CTIME_TESTS OFF CACHE BOOL "" FORCE)


I thought we landed on that running the ctime tests would actually be valuable? If not, or if it's waiting on something else, it'd be good to have a comment here explaining why these tests are being skipped.

I'm sorry if I misunderstood the previous discussion, but I thought that the idea was to mirror the master branch behaviour during migration to CMake.

I thought that the idea was to mirror the master branch behaviour during migration to CMake.

Ok, but the ctime tests are currently compiled, and runnable there.

Thanks! Fixed in #253.

d2ab006 cmake: Do not disable libsecp's constant time tests (Hennadii Stepanov) Pull request description: This PR addresses #192 (comment): > Ok, but the ctime tests are currently compiled, and runnable there. and mirrors the behavior of the master branch. Assuming the Valgrind is installed, after building the user can tun: ``` $ valgrind ./build/src/secp256k1/src/ctime_tests ``` ACKs for top commit: TheCharlatan: ACK d2ab006 Tree-SHA512: c4dd4967473fecb7297e1b60b02001a08bc4f302266512fcb76e8742e12305b2f3a4aa29a7fb72489fee68548199c9c87e5750ee4184b5eb64f8788083efeef3

… summaries 2995f0f cmake: Move `FUZZ_BINARY_LINKS_WITHOUT_MAIN_FUNCTION` check (Hennadii Stepanov) 2296161 cmake: Use `TYPE` property to tell Qt libraries type (Hennadii Stepanov) 69afb91 cmake: Move Qt checks (Hennadii Stepanov) 9afe0c8 cmake: Fix `translate` target when `ENABLE_WALLET=OFF` (Hennadii Stepanov) Pull request description: This PR addresses #192 (comment). Additionally, fixed a bug in the `translate` target for the `-DBUILD_GUI=ON -DENABLE_WALLET=OFF` configuration. ACKs for top commit: m3dwards: ACK 2995f0f Tree-SHA512: 03aad0a81077112ee26e04b6faa78abe0bde854faa706b22352f5dd8b0a1be65be5942fad985ec0277eca93227ef005e3a02712059858e6dd6e09ccc5c664e49

77876ca cmake: Move `CMAKE_C_FLAGS` deduplication after enabling C language (Hennadii Stepanov) 56d3878 cmake: Use C++ rather than C in `test-security-check` target (Hennadii Stepanov) 64dd87c contrib: c++ify test stubs after switching to c++ compilers (Cory Fields) 2ea2f53 contrib: rename cc to cxx in binary checking scripts (Cory Fields) 19307f5 contrib: use c++ rather than c for binary tests (Cory Fields) 439ceb0 guix: use glibc 2.31 (fanquake) Pull request description: #192 introduced a regression in the Guix build. For instance, `env HOSTS=x86_64-w64-mingw32 ./contrib/guix/guix-build` for 1eb1250. The C language must be enabled _before_ any reference to `CMAKE_C_*` variables and _in the scope_ of those references (in case of non-cache variables). However, this is not the case on the staging branch at present for the following references: https://github.com/hebasto/bitcoin/blob/1eb125095fc5578d8faf7501f2cf023237b14d59/CMakeLists.txt#L573-L575 and https://github.com/hebasto/bitcoin/blob/1eb125095fc5578d8faf7501f2cf023237b14d59/cmake/module/Maintenance.cmake#L39-L46 This PR fixes this issue by moving the `enable_language(C)` to the top of the root listfile. Top commit has no ACKs. Tree-SHA512: 246745f84973ee38a88fec5dc99887f4b8065bc837b7eb8fd0db44aaded7ea24ece3a2b54baf89df5f5a42a9206ca93690d54733946c9b01e345e4485103493e

hebasto commented May 7, 2024

View reviewed changes

src/CMakeLists.txt Outdated Show resolved Hide resolved

hebasto commented May 7, 2024

View reviewed changes

src/CMakeLists.txt Outdated Show resolved Hide resolved

hebasto force-pushed the 240507-cmake-secp branch from a02f802 to 586d32b Compare May 7, 2024 15:30

theuni reviewed May 8, 2024

View reviewed changes

hebasto force-pushed the 240507-cmake-secp branch from 586d32b to e5e8304 Compare May 8, 2024 18:30

hebasto mentioned this pull request May 8, 2024

[RFC] cmake: Build option naming #194

Closed

real-or-random reviewed May 9, 2024

View reviewed changes

hebasto force-pushed the cmake-staging branch from 1d76056 to e8e0e69 Compare May 21, 2024 11:17

hebasto force-pushed the 240507-cmake-secp branch from e5e8304 to b44d5fb Compare May 25, 2024 12:24

hebasto mentioned this pull request May 25, 2024

cmake: Fix hardening flags #211

Merged

hebasto added the enhancement New feature or request label May 25, 2024

hebasto force-pushed the 240507-cmake-secp branch 2 times, most recently from 37ab6d2 to 9f4bf00 Compare May 26, 2024 09:18

hebasto force-pushed the 240507-cmake-secp branch from 9f4bf00 to e6d1796 Compare June 4, 2024 08:35

hebasto force-pushed the cmake-staging branch from 0e6def0 to 597da83 Compare June 5, 2024 14:35

hebasto force-pushed the 240507-cmake-secp branch 2 times, most recently from 035c96d to 44a172c Compare June 8, 2024 14:48

hebasto marked this pull request as ready for review June 8, 2024 14:49

hebasto force-pushed the 240507-cmake-secp branch from 44a172c to dd47ca9 Compare June 17, 2024 08:20

hebasto added this to the Ready for master milestone Jun 18, 2024

hebasto force-pushed the 240507-cmake-secp branch from dd47ca9 to d7e2989 Compare June 18, 2024 13:38

hebasto mentioned this pull request Jun 18, 2024

cmake: Do not modify build types when integrating by downstream project bitcoin-core/secp256k1#1543

Merged

real-or-random reviewed Jun 20, 2024

View reviewed changes

hebasto force-pushed the 240507-cmake-secp branch from d7e2989 to b3766a3 Compare June 20, 2024 11:26

hebasto force-pushed the 240507-cmake-secp branch from 6fb9b31 to 669b611 Compare June 28, 2024 11:23

hebasto added 8 commits June 29, 2024 10:19

cmake: Switch to libsecp256k1 upstream build system

f0aec78

cmake: Let libsecp256k1 manage config-specific C flags

c3134b0

cmake: Remove C stuff from summary

469e1ad

cmake: Pass to secp256k1 build system only sanitizer flags

65f15d1

This change mirrors behavior in the master branch.

cmake: Override build configurations for libsecp256k1

cdd843a

cmake: Reorder subdirectories

70ec2d9

This change ensures that both summaries—secp256k1's and Bitcoin Core's— are displayed without interleaving of the `find_package(Doxygen)` output from the `doc` subdirectory.

[FIXUP] cmake: Enable tests before adding subdirectories

bfe7c4d

cmake: Run secp256k1 tests

458d54c

hebasto force-pushed the 240507-cmake-secp branch from 669b611 to 458d54c Compare June 29, 2024 09:21

real-or-random approved these changes Jun 29, 2024

View reviewed changes

hebasto merged commit a7cc7d8 into cmake-staging Jun 30, 2024
38 checks passed

hebasto mentioned this pull request Jun 30, 2024

cmake: Fix clang-tidy CI task #249

Merged

fanquake reviewed Jul 1, 2024

View reviewed changes

This was referenced Jul 1, 2024

cmake: Enable C language in the root CMakeLists.txt #252

Merged

cmake: Do not disable libsecp's constant time tests #253

Merged

hebasto mentioned this pull request Jul 2, 2024

cmake: Remove log lines from between libsecp's and Core's summaries #254

Merged

hebasto mentioned this pull request Aug 26, 2024

fuzz coverage fails with clang? #341

Closed

fanquake mentioned this pull request Sep 2, 2024

build: Use correct variable name bitcoin/bitcoin#30791

Merged

	# Avoid the use of aligned vector instructions when building for Windows.
	# See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412.
	try_append_cxx_flags("-Wa,-muse-unaligned-vector-move" TARGET core_base_interface SKIP_LINK)

	try_append_cxx_flags("-fsanitize=${SANITIZERS}" TARGET core_base_interface
	RESULT_VAR cxx_supports_sanitizers
	SKIP_LINK
	)

cmake: Switch to libsecp256k1 upstream build system #192

cmake: Switch to libsecp256k1 upstream build system #192

Conversation

hebasto commented May 7, 2024 • edited Loading

hebasto commented May 7, 2024

hebasto commented May 7, 2024

fanquake commented May 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

real-or-random left a comment

Choose a reason for hiding this comment

hebasto commented May 25, 2024

hebasto commented Jun 17, 2024

hebasto commented Jun 18, 2024

Choose a reason for hiding this comment

hebasto Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

real-or-random Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto commented Jun 26, 2024

hebasto commented Jun 28, 2024

hebasto commented Jun 29, 2024

real-or-random left a comment

Choose a reason for hiding this comment

hebasto commented Jun 29, 2024

Choose a reason for hiding this comment

hebasto Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hebasto commented May 7, 2024 •

edited

Loading

hebasto Jun 20, 2024 •

edited

Loading

real-or-random Jun 20, 2024 •

edited

Loading

hebasto Jul 1, 2024 •

edited

Loading