Merge #174: Upstream PRs 1064, 1049, 899, 1068, 1072, 1069, 1074, 102…

…6, 1033, 748, 1079, 1088, 1090, 731, 1089, 995, 1094, 1093

645d9c5 examples: let musig use random.h instead of /dev/urandom (Jonas Nick)
eccba5b examples: relicense musig example to CC0 public domain (Jonas Nick)
7c5af74 ci: fix missing EXPERIMENTAL flags (Jonas Nick)
03bea1e configure: add -zkp modules to dev-mode and remove redundant code (Jonas Nick)
2adb741 examples: rename example_musig to musig_example for consistency (Jonas Nick)
37d3692 tests: Add tests for _read_be32 and _write_be32 (Tim Ruffing)
616b43d util: Remove endianness detection (Tim Ruffing)
8d89b9e hash: Make code agnostic of endianness (Tim Ruffing)
55512d3 doc: clean up module help text in configure.ac (Elliott Jin)
d9d94a9 doc: mention optional modules in README (Elliott Jin)
7f09d0f README: mention that ARM assembly is experimental (Jonas Nick)
80cf4ee build: stop treating schnorrsig, extrakeys modules as experimental (Jonas Nick)
b8f8b99 docs: Fix return value for functions that don't have invalid inputs (Tim Ruffing)
f813bb0 schnorrsig: Adapt example to new API (Tim Ruffing)
99e6568 schnorrsig: Rename schnorrsig_sign to schnorsig_sign32 and deprecate (Tim Ruffing)
fc94a2d Use SECP256K1_DEPRECATED for existing deprecated API functions (Tim Ruffing)
3db0560 Add SECP256K1_DEPRECATED attribute for marking API parts as deprecated (Tim Ruffing)
f8d9174 Add SHA256 bit counter tests (Tim Ruffing)
9b514ce Add test vector for very long SHA256 messages (Tim Ruffing)
8e3dde1 Simplify struct initializer for SHA256 padding (Tim Ruffing)
eb28464 Change SHA256 byte counter from size_t to uint64_t (Tim Ruffing)
21b2eba configure: Remove redundant pkg-config code (Tim Ruffing)
0d253d5 configure: Use modern way to set AR (Tim Ruffing)
e0838d6 configure: Add hidden --enable-dev-mode to enable all the stuff (Tim Ruffing)
fabd579 configure: Remove redundant code that sets _enable variables (Tim Ruffing)
0d4226c configure: Use canonical variable prefix _enable consistently (Tim Ruffing)
7c9502c Add a copy of the CC0 license to the examples (Elichai Turkel)
42e0343 Add usage examples to the readme (Elichai Turkel)
517644e Optionally compile the examples in autotools, compile+run in travis (Elichai Turkel)
422a7cc Add a ecdh shared secret example (Elichai Turkel)
b0cfbcc Add a Schnorr signing and verifying example (Elichai Turkel)
fee7d4b Add an ECDSA signing and verifying example (Elichai Turkel)
e848c37 Update sage files for new formulae (Peter Dettman)
d64bb5d Add fe_half tests for worst-case inputs (Peter Dettman)
4eb8b93 Further improve doubling formula using fe_half (Peter Dettman)
557b31f Doubling formula using fe_half (Pieter Wuille)
2cbb4b1 Run more iterations of run_field_misc (Pieter Wuille)
9cc5c25 Add test for secp256k1_fe_half (Pieter Wuille)
925f78d Add _fe_half and use in _gej_add_ge (Peter Dettman)
3531a43 ecdh: Make generator_basepoint test depend on global iteration count (Tim Ruffing)
c881dd4 ecdh: Add test computing shared_secret=basepoint with random inputs (Tim Ruffing)
e51ad3b ci: Retry `brew update` a few times to avoid random failures (Tim Ruffing)
b1cb969 ci: Revert "Attempt to make macOS builds more reliable" (Tim Ruffing)
e0db3f8 build: Replace use of deprecated autoconf macro AC_PROG_CC_C89 (laanwj)
d9396a5 ci: Attempt to make macOS builds more reliable (Tim Ruffing)
ebb1bee sage: Ensure that constraints are always fastfracs (Tim Ruffing)
d8d5485 ci: Run sage prover on CI (Tim Ruffing)
77cfa98 sage: Normalize sign of polynomial factors in prover (Tim Ruffing)
eae7586 sage: Exit with non-zero status in case of failures (Tim Ruffing)
b54d843 sage: Fix printing of errors (Tim Ruffing)
e108d00 sage: Fix incompatibility with sage 9.4 (Tim Ruffing)
b797a50 Create a SECP256K1_ECMULT_TABLE_VERIFY macro. (Russell O'Connor)
a731200 Replace ECMULT_TABLE_GET_GE_STORAGE macro with a function. (Russell O'Connor)
fe34d9f Eliminate input_pos state field from ecmult_strauss_wnaf. (Russell O'Connor)
0397d00 Eliminate na_1 and na_lam state fields from ecmult_strauss_wnaf. (Russell O'Connor)
7ba3ffc Remove the unused pre_a_lam allocations. (Russell O'Connor)
b3b57ad Eliminate the pre_a_lam array from ecmult_strauss_wnaf. (Russell O'Connor)
ae7ba0f Remove the unused prej allocations. (Russell O'Connor)
e5c1889 Eliminate the prej array from ecmult_strauss_wnaf. (Russell O'Connor)
c9da1ba Move secp256k1_fe_one to field.h (Russell O'Connor)
070e772 Faster fixed-input ecmult tests (Pieter Wuille)
45f37b6 Modulo-reduce msg32 inside RFC6979 nonce fn to match spec. Fixes #1063. (Paul Miller)

Pull request description:

  [bitcoin-core/secp256k1#1064]: Modulo-reduce msg32 inside RFC6979 nonce fn to match spec. Fixes #1063
  [bitcoin-core/secp256k1#1049]: Faster fixed-input ecmult tests
  [bitcoin-core/secp256k1#899]: Reduce stratch space needed by ecmult_strauss_wnaf.
  [bitcoin-core/secp256k1#1068]: sage: Fix incompatibility with sage 9.4
  [bitcoin-core/secp256k1#1072]: ci: Attempt to make macOS builds more reliable
  [bitcoin-core/secp256k1#1069]: build: Replace use of deprecated autoconf macro AC_PROG_CC_C89
  [bitcoin-core/secp256k1#1074]: ci: Retry brew update a few times to avoid random failures
  [bitcoin-core/secp256k1#1026]: ecdh: Add test computing shared_secret=basepoint with random inputs
  [bitcoin-core/secp256k1#1033]: Add _fe_half and use in _gej_add_ge and _gej_double
  [bitcoin-core/secp256k1#748]: Add usage examples
  [bitcoin-core/secp256k1#1079]: configure: Add hidden --enable-dev-mode to enable all the stuff
  [bitcoin-core/secp256k1#1088]: configure: Use modern way to set AR
  [bitcoin-core/secp256k1#1090]: configure: Remove redundant pkg-config code
  [bitcoin-core/secp256k1#731]: Change SHA256 byte counter from size_t to uint64_t
  [bitcoin-core/secp256k1#1089]: Schnorrsig API improvements
  [bitcoin-core/secp256k1#995]: build: stop treating schnorrsig, extrakeys modules as experimental
  [bitcoin-core/secp256k1#1094]: doc: Clarify configure flags for optional modules
  [bitcoin-core/secp256k1#1093]: hash: Make code agnostic of endianness

  This PR can be recreated  with `./sync-upstream.sh range 8746600`.

ACKs for top commit:
  real-or-random:
    ACK 645d9c5 I rederived the tree, and tested it with MSVC, including the musig example

Tree-SHA512: 3b771630806ed8481053958c21820dce6e869371833cd18a5c430a2768bda8064ad2bb247afbe38e3fa37320a8b1dbbe65ad68c8963efb995d96aa29ae574884

.cirrus.yml

@@ -28,6 +28,8 @@ env:
   BENCH: yes
   SECP256K1_BENCH_ITERS: 2
   CTIMETEST: yes
+  # Compile and run the tests
+  EXAMPLES: yes
 
 cat_logs_snippet: &CAT_LOGS
   always:
@@ -70,12 +72,12 @@ task:
   << : *LINUX_CONTAINER
   matrix: &ENV_MATRIX
     - env: {WIDEMUL:  int64,  RECOVERY: yes}
-    - env: {WIDEMUL:  int64,                 ECDH: yes, EXPERIMENTAL: yes, SCHNORRSIG: yes, ECDSA_S2C: yes,  RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
+    - env: {WIDEMUL:  int64,                 ECDH: yes, SCHNORRSIG: yes, EXPERIMENTAL: yes, ECDSA_S2C: yes,  RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
     - env: {WIDEMUL: int128}
-    - env: {WIDEMUL: int128,  RECOVERY: yes,            EXPERIMENTAL: yes, SCHNORRSIG: yes}
-    - env: {WIDEMUL: int128,                 ECDH: yes, EXPERIMENTAL: yes, SCHNORRSIG: yes, ECDSA_S2C: yes, RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
+    - env: {WIDEMUL: int128,  RECOVERY: yes,            SCHNORRSIG: yes}
+    - env: {WIDEMUL: int128,                 ECDH: yes, SCHNORRSIG: yes, EXPERIMENTAL: yes, ECDSA_S2C: yes, RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
     - env: {WIDEMUL: int128,  ASM: x86_64}
-    - env: {                  RECOVERY: yes,            EXPERIMENTAL: yes, SCHNORRSIG: yes, ECDSA_S2C: yes, RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
+    - env: {                  RECOVERY: yes,            SCHNORRSIG: yes, EXPERIMENTAL: yes, ECDSA_S2C: yes, RANGEPROOF: yes, WHITELIST: yes, GENERATOR: yes, MUSIG: yes, ECDSAADAPTOR: yes}
     - env: {BUILD: distcheck, WITH_VALGRIND: no, CTIMETEST: no, BENCH: no}
     - env: {CPPFLAGS: -DDETERMINISTIC}
     - env: {CFLAGS: -O0, CTIMETEST: no}
@@ -98,8 +100,8 @@ task:
     HOST: i686-linux-gnu
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
+    EXPERIMENTAL: yes
     ECDSA_S2C: yes
     RANGEPROOF: yes
     WHITELIST: yes
@@ -148,7 +150,8 @@ task:
   ##   - rm /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
   ##
   brew_valgrind_pre_script:
-    - brew update
+    # Retry a few times because this tends to fail randomly.
+    - for i in {1..5}; do brew update && break || sleep 15; done
     - brew config
     - brew tap LouisBrunner/valgrind
     # Fetch valgrind source but don't build it yet.
@@ -188,8 +191,8 @@ task:
     WITH_VALGRIND: no
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
+    EXPERIMENTAL: yes
     ECDSA_S2C: yes
     RANGEPROOF: yes
     WHITELIST: yes
@@ -214,12 +217,11 @@ task:
     WITH_VALGRIND: no
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
     CTIMETEST: no
   matrix:
     - env: {}
-    - env: {ASM: arm}
+    - env: {EXPERIMENTAL: yes, ASM: arm}
   << : *MERGE_BASE
   test_script:
     - ./ci/cirrus.sh
@@ -235,7 +237,6 @@ task:
     WITH_VALGRIND: no
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
     CTIMETEST: no
   << : *MERGE_BASE
@@ -253,7 +254,6 @@ task:
     WITH_VALGRIND: no
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
     CTIMETEST: no
   << : *MERGE_BASE
@@ -271,7 +271,6 @@ task:
     WITH_VALGRIND: no
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
     CTIMETEST: no
   << : *MERGE_BASE
@@ -286,8 +285,8 @@ task:
   env:
     ECDH: yes
     RECOVERY: yes
-    EXPERIMENTAL: yes
     SCHNORRSIG: yes
+    EXPERIMENTAL: yes
     ECDSA_S2C: yes
     RANGEPROOF: yes
     WHITELIST: yes
@@ -340,11 +339,17 @@ task:
     CC: gcc
     MAKEFLAGS: -j4 CC=g++ CFLAGS=-fpermissive\ -g
     WERROR_CFLAGS:
-    EXPERIMENTAL: yes
     ECDH: yes
     RECOVERY: yes
     SCHNORRSIG: yes
   << : *MERGE_BASE
   test_script:
     - ./ci/cirrus.sh
   << : *CAT_LOGS
+
+task:
+  name: "sage prover"
+  << : *LINUX_CONTAINER
+  test_script:
+    - cd sage
+    - sage prove_group_implementations.sage

.gitignore

@@ -8,11 +8,16 @@ exhaustive_tests
 precompute_ecmult_gen
 precompute_ecmult
 valgrind_ctime_test
+ecdh_example
+ecdsa_example
+schnorr_example
 *.exe
 *.so
 *.a
 *.csv
 !.gitignore
+*.log
+*.trs
 
 Makefile
 configure
@@ -43,6 +48,7 @@ coverage.*.html
 
 src/libsecp256k1-config.h
 src/libsecp256k1-config.h.in
+build-aux/ar-lib
 build-aux/config.guess
 build-aux/config.sub
 build-aux/depcomp
@@ -60,4 +66,4 @@ src/stamp-h1
 libsecp256k1.pc
 contrib/gh-pr-create.sh
 
-example_musig
+musig_example

Makefile.am

@@ -65,6 +65,7 @@ noinst_HEADERS += contrib/lax_der_parsing.h
 noinst_HEADERS += contrib/lax_der_parsing.c
 noinst_HEADERS += contrib/lax_der_privatekey_parsing.h
 noinst_HEADERS += contrib/lax_der_privatekey_parsing.c
+noinst_HEADERS += examples/random.h
 
 PRECOMPUTED_LIB = libsecp256k1_precomputed.la
 noinst_LTLIBRARIES = $(PRECOMPUTED_LIB)
@@ -141,13 +142,49 @@ exhaustive_tests_LDFLAGS = -static
 TESTS += exhaustive_tests
 endif
 
+if USE_EXAMPLES
+noinst_PROGRAMS += ecdsa_example
+ecdsa_example_SOURCES = examples/ecdsa.c
+ecdsa_example_CPPFLAGS = -I$(top_srcdir)/include
+ecdsa_example_LDADD = libsecp256k1.la
+ecdsa_example_LDFLAGS = -static
+if BUILD_WINDOWS
+ecdsa_example_LDFLAGS += -lbcrypt
+endif
+TESTS += ecdsa_example
+if ENABLE_MODULE_ECDH
+noinst_PROGRAMS += ecdh_example
+ecdh_example_SOURCES = examples/ecdh.c
+ecdh_example_CPPFLAGS = -I$(top_srcdir)/include
+ecdh_example_LDADD = libsecp256k1.la
+ecdh_example_LDFLAGS = -static
+if BUILD_WINDOWS
+ecdh_example_LDFLAGS += -lbcrypt
+endif
+TESTS += ecdh_example
+endif
+if ENABLE_MODULE_SCHNORRSIG
+noinst_PROGRAMS += schnorr_example
+schnorr_example_SOURCES = examples/schnorr.c
+schnorr_example_CPPFLAGS = -I$(top_srcdir)/include
+schnorr_example_LDADD = libsecp256k1.la
+schnorr_example_LDFLAGS = -static
+if BUILD_WINDOWS
+schnorr_example_LDFLAGS += -lbcrypt
+endif
+TESTS += schnorr_example
+endif
 if ENABLE_MODULE_MUSIG
-noinst_PROGRAMS += example_musig
-example_musig_SOURCES = examples/musig.c
-example_musig_CPPFLAGS = -I$(top_srcdir)/include
-example_musig_LDADD = libsecp256k1.la
-example_musig_LDFLAGS = -static
-TESTS += example_musig
+noinst_PROGRAMS += musig_example
+musig_example_SOURCES = examples/musig.c
+musig_example_CPPFLAGS = -I$(top_srcdir)/include
+musig_example_LDADD = libsecp256k1.la
+musig_example_LDFLAGS = -static
+if BUILD_WINDOWS
+musig_example_LDFLAGS += -lbcrypt
+endif
+TESTS += musig_example
+endif
 endif
 
 ### Precomputed tables

README.md

@@ -17,7 +17,7 @@ Features:
 * Suitable for embedded systems.
 * Optional module for public key recovery.
 * Optional module for ECDH key exchange.
-* Optional module for Schnorr signatures according to [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki) (experimental).
+* Optional module for Schnorr signatures according to [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
 * Optional module for ECDSA adaptor signatures (experimental).
 
 Experimental features have not received enough scrutiny to satisfy the standard of quality of this library but are made available for testing and review by the community. The APIs of these features should not be considered stable.
@@ -36,6 +36,7 @@ Implementation details
   * Optimized implementation of arithmetic modulo the curve's field size (2^256 - 0x1000003D1).
     * Using 5 52-bit limbs (including hand-optimized assembly for x86_64, by Diederik Huys).
     * Using 10 26-bit limbs (including hand-optimized assembly for 32-bit ARM, by Wladimir J. van der Laan).
+      * This is an experimental feature that has not received enough scrutiny to satisfy the standard of quality of this library but is made available for testing and review by the community.
 * Scalar operations
   * Optimized implementation without data-dependent branches of arithmetic modulo the curve's order.
     * Using 4 64-bit limbs (relying on __int128 support in the compiler).
@@ -70,6 +71,16 @@ libsecp256k1 is built using autotools:
     $ make check  # run the test suite
     $ sudo make install  # optional
 
+To compile optional modules (such as Schnorr signatures), you need to run `./configure` with additional flags (such as `--enable-module-schnorrsig`). Run `./configure --help` to see the full list of available flags.
+
+Usage examples
+-----------
+  Usage examples can be found in the [examples](examples) directory. To compile them you need to configure with `--enable-examples`.
+  * [ECDSA example](examples/ecdsa.c)
+  * [Schnorr signatures example](examples/schnorr.c)
+  * [Deriving a shared secret (ECDH) example](examples/ecdh.c)
+  To compile the Schnorr signature and ECDH examples, you also need to configure with `--enable-module-schnorrsig` and `--enable-module-ecdh`.
+
 Test coverage
 -----------
 

build-aux/m4/bitcoin_secp.m4

@@ -38,3 +38,16 @@ AC_DEFUN([SECP_TRY_APPEND_CFLAGS], [
   unset flag_works
   AC_SUBST($2)
 ])
+
+dnl SECP_SET_DEFAULT(VAR, default, default-dev-mode)
+dnl Set VAR to default or default-dev-mode, depending on whether dev mode is enabled
+AC_DEFUN([SECP_SET_DEFAULT], [
+  if test "${enable_dev_mode+set}" != set; then
+    AC_MSG_ERROR([[Set enable_dev_mode before calling SECP_SET_DEFAULT]])
+  fi
+  if test x"$enable_dev_mode" = x"yes"; then
+    $1="$3"
+  else
+    $1="$2"
+  fi
+])

ci/cirrus.sh

@@ -21,6 +21,8 @@ valgrind --version || true
     --enable-module-ecdsa-s2c="$ECDSA_S2C" \
     --enable-module-rangeproof="$RANGEPROOF" --enable-module-whitelist="$WHITELIST" --enable-module-generator="$GENERATOR" \
     --enable-module-schnorrsig="$SCHNORRSIG"  --enable-module-musig="$MUSIG" --enable-module-ecdsa-adaptor="$ECDSAADAPTOR" \
+    --enable-module-schnorrsig="$SCHNORRSIG" \
+    --enable-examples="$EXAMPLES" \
     --with-valgrind="$WITH_VALGRIND" \
     --host="$HOST" $EXTRAFLAGS
 

ci/linux-debian.Dockerfile

@@ -19,7 +19,8 @@ RUN apt-get install --no-install-recommends --no-upgrade -y \
         gcc-arm-linux-gnueabihf libc6-dev-armhf-cross libc6-dbg:armhf \
         gcc-aarch64-linux-gnu libc6-dev-arm64-cross libc6-dbg:arm64 \
         gcc-powerpc64le-linux-gnu libc6-dev-ppc64el-cross libc6-dbg:ppc64el \
-        wine gcc-mingw-w64-x86-64
+        wine gcc-mingw-w64-x86-64 \
+        sagemath
 
 # Run a dummy command in wine to make it set up configuration
 RUN wine64-stable xcopy || true