Add macOS to the CI

[elichai]

This adds macOS to the travis so we can test all the same configurations under macOS, both with Apple's clang and proper gcc on macOS.
It also runs the ctime test, and valgrind on the tests just like on linux.

The current travis script is pretty messy because we added more and more configuration over time each one required more complex ifs making it harder to read, and because of the somewhat complex logic it failed[1] on macOS(having an old bash version) so I decided to just throw it into a standalone sh script instead, that way it can be formatted nicely and with -e it will fail on every error without needing to put && everywhere.

Some changed in the script:

• Replaced the usage of libtool with the locally generated libtool (Add valgrind constant-time test to make check #723 (comment))

• Added --error-exitcode=42 to the ctime tests because they currently silently fail on -O0 (https://travis-ci.org/github/bitcoin-core/secp256k1/jobs/681571334#L454) and disabled the ctime tests on -O0.

• Moved the valgrind tests to the matrix so that they'll run on both gcc and clang and on macOS.
  (also, now that Eliminate harmless non-constant time operations on secret data. #710 is merged we always pass -DVALGRIND when the valgrind headers exist but I left the explicit CFLAGS in those tests anyway, there's no harm in explicitly doing that)

• Removed the use of EXTRAFLAGS for setting CFLAGS, it's enough to just set CFLAGS directly and it can cause troubles in sh (the whole EXTRAFLAGS="--disable-openssl-tests CPPFLAGS=-DVALGRIND")

• We have to explicitly set the gcc version on macOS+gcc because macOS ship with a fake gcc which is basically just an alias to their clang compiler, and installing proper gcc from brew adds a gcc-* binary and doesn't replace the gcc binary, so we have to explicitly set CC=gcc-9 under that scenario, so I also explicitly install gcc@9 so it shouldn't break when macOS gets gcc-10.

• Bumped ubuntu to bionic because of Add usage examples #748 (comment) (the end of End of Standard Support is in a year anyway) it's in a separate commit so that if anyone have concerns I'll just drop that commit.

1. https://travis-ci.org/github/elichai/secp256k1/jobs/681663742#L336

[real-or-random]

That's very neat. I'll have a closer look later. Let's try to get this merged before I continue to work on #723 because this here solves a bunch of the issues I encountered there.

[real-or-random]

Fwiw, I think travis_wait does not work for us currently and just hides the timeout. It does not work with forking processes.

Proof: https://travis-ci.org/github/bitcoin-core/secp256k1/jobs/682613807#L1112

See

• Unable to use travis_wait with a complex command travis-ci/travis-ci#7431
• 10 minute timeout failure when using travis_wait 30 travis-ci/travis-ci#8526
• cannot build long running tasks. travis-ci/travis-ci#5281

I had noticed this already when working on #723 but never wrote it down. If you want, you can give it a try. The issues above should provide some pointers, in particular PRs from other projects. But we can do this also in another PR. After this, was broken before this PR already.

[real-or-random]

Any idea if that dsymutil thing here is specific to valgrind on MacOS?
https://travis-ci.org/github/bitcoin-core/secp256k1/jobs/682613807#L1057

[elichai]

Any idea if that dsymutil thing here is specific to valgrind on MacOS?
https://travis-ci.org/github/bitcoin-core/secp256k1/jobs/682613807#L1057

Sounds like it:
https://llvm.org/docs/CommandGuide/dsymutil.html

And here: https://valgrind.org/docs/manual/manual-core.html

This option is only relevant when running Valgrind on Mac OS X.

[elichai]

Much better now :)
2-3 minutes per macOS job instead of ~11 minutes. now I hope the valgrind test won't time out

[real-or-random]

Great, so I think this is ready (from my side at least ^^). One last question: Have you verified that a ct-test failure indeed makes the build fail? I guess this should be tested because this can be broken in subtle ways as I learned in my own PR...

[elichai]

Great, so I think this is ready (from my side at least ^^). One last question: Have you verified that a ct-test failure indeed makes the build fail? I guess this should be tested because this can be broken in subtle ways as I learned in my own PR...

I applied the following diff:

diff --git a/src/field_5x52_impl.h b/src/field_5x52_impl.h
index 71aa550..348cca0 100644
--- a/src/field_5x52_impl.h
+++ b/src/field_5x52_impl.h
@@ -465,13 +465,12 @@ static SECP256K1_INLINE void secp256k1_fe_cmov(secp256k1_fe *r, const secp256k1_
 }
 
 static SECP256K1_INLINE void secp256k1_fe_storage_cmov(secp256k1_fe_storage *r, const secp256k1_fe_storage *a, int flag) {
-    uint64_t mask0, mask1;
-    mask0 = flag + ~((uint64_t)0);
-    mask1 = ~mask0;
-    r->n[0] = (r->n[0] & mask0) | (a->n[0] & mask1);
-    r->n[1] = (r->n[1] & mask0) | (a->n[1] & mask1);
-    r->n[2] = (r->n[2] & mask0) | (a->n[2] & mask1);
-    r->n[3] = (r->n[3] & mask0) | (a->n[3] & mask1);
+    if (flag) {
+        r->n[0] = a->n[0];
+        r->n[1] = a->n[1];
+        r->n[2] = a->n[2];
+        r->n[3] = a->n[3];
+    }
 }

And as you can see here https://travis-ci.org/github/elichai/secp256k1/builds/684392980 everything failed except the jobs with FIELD=32bit

[jonasnick]

Thank you for cleaning up. ACK mod nits.

[real-or-random]

@elichai Is this ready for review?

[elichai]

Yes

[real-or-random]

ACK 71757da I inspected the diff

[jonasnick]

ACK 71757da