Enable static builds of binaries

[rainbreak]

Following on from ethereum/webthree-helpers#157, enable static linking of some more binaries by switching add_executable to eth_simple_add_executable.

This is a master issue. Target binaries:

• eth
  • wrap add_exe link eth binary with custom add_executable webthree#163
• solc Use add exe wrapper for solc, soltest, lllc solidity#528
• soltest ^
• lllc ^
  • fix macro arguments Fix macro arguments. webthree-helpers#164
• ethminer
  • wrap add_exe Use add_exe wrapper for ethminer libethereum#248
  • macro changed. dynamic working. static still has -lcurl ordering issue
• Fix accidental REQUIRED introduced onto curl for dynamic builds (Only looking for ssh2, openssl and lib when static linkage is enabled, to avoid spammy messages webthree-helpers#165)
• Usage of STATIC_LINKING across platforms
  • Test static linkage on Alpine Linux
  • Test static linkage on Debian (working, you said?) (no, curl link problems)
  • Test static linkage on Ubuntu
  • Test static linkage on OS X (partial static-linkage working).
    • Suppress some linker warnings
  • Test static linkage on Windows

[rainbreak]

Just running a local build now to sanity check the above build with static linking.

[rainbreak]

Everything works except ethminer which fails on linking with a curl problem:

[100%] Linking CXX executable ethminer
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::HttpClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
httpclient.cpp:(.text+0x14d): undefined reference to `curl_easy_init'
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::~HttpClient()':
httpclient.cpp:(.text+0x1e4): undefined reference to `curl_easy_cleanup'
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::SendRPCMessage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)':
httpclient.cpp:(.text+0x2b5): undefined reference to `curl_easy_setopt'
...

[bobsummerwill]

Link order issue? A la http://stackoverflow.com/questions/28165518/use-libcurl-undefined-reference-to-curl-easy-init?

[rainbreak]

That's puzzling. Not sure why it isn't an issue when building eth, which also uses jsonrpccpp.

[bobsummerwill]

Well, if it just a command-line order issue, it would all just be down to the order in which targets were executed. So maybe just some fluky alphabetical thing, where -lcurl is ending up early in this case?

[rainbreak]

This is the actual link command (formatted):

/src/webthree-umbrella/build # cat libethereum/ethminer/CMakeFiles/ethminer.dir/link.txt 

/usr/bin/c++   -std=c++11 -Wall -Wno-unknown-pragmas -Wextra -Werror -pedantic -DSHAREDLIB -fPIC
-std=c++11 -Wall -Wno-unknown-pragmas -Wextra -Werror -pedantic -DSHAREDLIB -fPIC
-Wno-error -fstack-protector-strong -Wstack-protector -O3 -DNDEBUG -DETH_RELEASE  -static

CMakeFiles/ethminer.dir/main.cpp.o  -o ethminer -rdynamic 
/src/built/lib/libjsoncpp.a 

-lssh2 -lssl -lcrypto -lz -lcurl                     <------------- curl link
/src/built/lib/libjsonrpccpp-common.a 
/src/built/lib/libjsonrpccpp-client.a

/src/built/lib/libjsoncpp.a
/src/built/lib/libleveldb.a 
-lboost_thread-mt -lboost_random -lboost_filesystem -lboost_system -lpthread 
../../libweb3core/libdevcore/libdevcore.a 
../../webthree-helpers/utils/libscrypt/libscrypt.a /src/built/lib/libcryptlib.a 
../../webthree-helpers/utils/secp256k1/libsecp256k1.a

../../libweb3core/libdevcrypto/libdevcrypto.a 
../libethcore/libethcore.a 
../libevmcore/libevmcore.a
../libevm/libevm.a 
/src/built/lib/libminiupnpc.a 
../../libweb3core/libp2p/libp2p.a 
-lboost_regex

../libethereum/libethereum.a 
/src/built/lib/libcryptlib.a 
../libethash/libethash.a
../libethashseal/libethashseal.a 
../libethereum/libethereum.a 
../libevm/libevm.a 
../libethcore/libethcore.a
../libevmcore/libevmcore.a 
../../libweb3core/libp2p/libp2p.a 
../../libweb3core/libdevcrypto/libdevcrypto.a
../../libweb3core/libdevcore/libdevcore.a 
/src/built/lib/libjsoncpp.a 
/src/built/lib/libleveldb.a 
-lboost_thread-mt -lboost_random -lboost_filesystem -lboost_system -lpthread
../../webthree-helpers/utils/libscrypt/libscrypt.a 
../../webthree-helpers/utils/secp256k1/libsecp256k1.a 
-lgmp /src/built/lib/libminiupnpc.a -lboost_regex 
../libethash/libethash.a /src/built/lib/libcryptlib.a 

I notice that ethminer isn't linking the json rpc server, only the client, so maybe there is some problem from there.

[rainbreak]

Manually appending -lcurl -lssh2 -lssl -lcrypto -lssl -lcrypto -lz fixes it (that's the output of pkg-config --libs --static libcurl). Now just need to coerce cmake into adding it.

[bobsummerwill]

'I notice that ethminer isn't linking the json rpc server, only the client, so maybe there is some problem from there."

^ Yeah. I think that this whole stack of "JSON-RPC stuff" which then drags in the curl chain is all a very thin thread of dependencies, but which is a bit messy. I did some minimization before which removed the unnecessary JsonRpc::Server dependency from ethminer. There is probably plenty more which can be trimmed, if we better understood it.

Good luck!

[bobsummerwill]

So it looks like part of my CURL REQUIRED thing was due to only having a partially synced workspace, but I've made one clean up which is building now, and then that's good and everything is healthy for the dynamic builds and you just have that curl link order thing for ethminer left.

I'm going to try doing static builds for OS X and Windows now, for eth at least, and hopefully more if I have time.

[bobsummerwill]

See ethereum/webthree-helpers#167.

[rainbreak]

I've got a Dockerfile for building (unsuccessfully) on Debian here:

https://github.com/rainbeam/eth-static/blob/debian/Dockerfile

Really minimal curl can be configured with:

./configure --prefix=/opt/curl --enable-static --disable-shared \
--disable-ldap --disable-ldaps --without-libidn --disable-rtsp --without-librtmp \
--disable-manual --disable-tls-srip --without-gnutls \
--without-ssl --without-libssh2 --without-zlib

However, even if we get past the curl linking problems, there is a bigger problem in that glibc is not designed to be statically linked. The STATIC_LINKING that I've implemented tries to static link everything. This works fine on a musl based platform (i.e. Alpine), but isn't going to work on Debian / Ubuntu. I don't know what the story is on Windows.

The binary this produces is completely portable across Linux based distros though - the limitation is only in the build environment.

As such, I don't regard the Debian / Ubuntu problems as part of this issue. The ethminer problem is still relevant though.

As for other platforms (other than x86_64)... I've not looked into it much but I saw that there are musl cross compilers provided here:

http://musl.codu.org/

Might be useful in your mobile adventures 😄 .

[bobsummerwill]

Thanks for that extra info, which will indeed be useful. Flipping to a statically linked musl cross-compiled runtime could be very useful.

I found that OS X as does not want to be completely statically linked (see https://developer.apple.com/library/mac/qa/qa1118/_index.html), so added checks for that to give an error message rather than just failing. I will try to "weaken that off", so that it statically links everything except the C++ runtime library.

Windows can be used with either DLLs or static libraries for the runtime. That will because unlike Linux, there is a binary compatibility guarantee for Windows, which is why old Windows apps work for years and years. Linux, I understand, has never had that, hence the monolithic kernel including drivers for everything under the sun. They have to be in-tree because the ABI can change arbitrarily between kernel and drivers.

Seems the ABI guarantee is a bit weak on OS X too.

As for curl, yeah, would be great to do a minimal compile like that, and I will certainly refer back to your notes here as-and-when I get around to pulling that in as a sub-module.

Did you try building curl like that for your Alpine Linux build? So with such settings, if looks like OpenSSL, ssh2 and zlib would NOT be needed?

[bobsummerwill]

"Manually appending -lcurl -lssh2 -lssl -lcrypto -lssl -lcrypto -lz fixes it (that's the output of pkg-config --libs --static libcurl). Now just need to coerce cmake into adding it."

^ Did you manage to work out the magic spell for this, btw? :-)

[rainbreak]

No. I messed around a bit with moving the eth_use(CURL) around and with linking against a minimal libcurl, but no luck. Stopped short of doing a full debug of how CMake constructs its linker arguments. It doesn't have a verbose logging mode, so I think you have to debug by adding lots of message calls.

Also tried a bit more with Debian. I reckon it is possible to build fully static by using musl-gcc from musl-tools, but still ran into problems with linking curl.

https://github.com/rainbeam/eth-static/tree/ethminer

https://github.com/rainbeam/eth-static/tree/debian

Did you have a look at Windows? (sorry missed your post above)

[rainbreak]

Just did an Alpine build with minimal curl - ssh2, openssl and zlib appear not to be needed.

[bobsummerwill]

Thanks for the update.

I've been doing a partially static build for OS X, with some success, and will try Windows as well when I get a chance.

"Just did an Alpine build with minimal curl - ssh2, openssl and zlib appear not to be needed." - That is good to know, and makes sense.

OS X static build is pulling in Snappy as an indirect dependency of leveldb, which is a bit weird, but I suspect it is just an artifact of the fact that leveldb is still a dylib dependency at the partial stage I got to:

[100%] Linking CXX executable eth
[100%] Built target eth
Bobs-MacBook-Air:build bob$ otool -L webthree/eth/eth
webthree/eth/eth:
    /usr/local/opt/jsoncpp/lib/libjsoncpp.1.dylib (compatibility version 1.0.0, current version 1.6.5)
    /usr/local/opt/leveldb/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/miniupnpc/lib/libminiupnpc.15.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-common.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-client.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-server.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 14.0.0, current version 14.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

So, getting to a static binary isn't possible on OS X, but statically linking all the libs should be possible. The binary I have right now runs just fine, but the following libs are still using .dylib, not .a:

• jsoncpp
• json-rpc-cpp
• leveldb
• miniupnp
• gmp

To get those ones statically linking, I'll need to build them from source. libz and curl ship as part of the platform, but again I could build them from source and statically link them.

[bobsummerwill]

PS. OS X partially statically linked executable is 12MB.

[bobsummerwill]

OK - So testing this for Windows is a horror-show, because we'll need LIB files for all the pre-built external dependencies. I'm going to defer that for now.

So @rainbeam ... are you pretty much done here now?

And you still didn't work out a magic-mantra to get the libcurl ordering thing resolved, right? That was on Debian? Or Alpine?

[rainbreak]

Yeah I think I'm done here.

Couldn't resolve the libcurl problem, except by manually modifying the link command after cmake was configured. This only affects ethminer. The problem exists on both Alpine and Debian.

Debian has the added problem of glibc, which isn't going to statically link. A similar solution to OSX (i.e. dynamic linking the system libs) might be needed if people want to build on that platform (rather than just use a static binary compiled on a musl system). Alternatively, Debian does have musl-tools etc. which may allow actual building of a static binary from Debian. I couldn't quite get this working.

Thanks for your help on this Bob :)

[bobsummerwill]

FYI ... My first Boost post made it through their filters ...

http://lists.boost.org/Archives/boost/2016/05/229385.php

... the one about thread setname/getname.

And got a first reply:

http://lists.boost.org/Archives/boost/2016/05/229388.php

[bobsummerwill]

See boostorg/thread#84, where there is now an issue tracking the Boost TODO, which I have offered to put on my backlog.

[bobsummerwill]

BTW - @Genoil - fyi - all this stuff :-) ^^^^

[nerdralph]

@bobsummerwill I've been talking to JW all along.
https://github.com/Genoil/cpp-ethereum/issues/87

[nerdralph]

I get a page not found error for the unity link.
http://buffered.io/posts/screencast-setting-up-unity-builds/

On Aug 8, 2016 19:59, "Bob Summerwill" notifications@github.com wrote:

@nerdralph https://github.com/nerdralph Yep - will test, of course.
Doing so now in https://github.com/bobsummerwill/cpp-ethereum/
commits/merge_repos.

@rainbeam https://github.com/rainbeam Nice sizes. Yeah - no doubt we
can get those down further!

On the LTO theme, another related thing which I was to do is
BulkBuild/UnityBuilds (#415
#415). Have either
of you done those before? They really are magic, but they work for the same
reason as LTO does. Just stuff all the code for a library together into a
single compilation unit and:

1. It builds like 10x faster, because the pre-processor isn't
   processing all the headers N times (one per compilation unit)
2. And the optimizer can do a better job, because even within each
   compilation unit, it has "whole program" visibility.

BulkBuild delivers stunning build speed-ups and improves the generated
code quality. It is likely kind of redundant now in terms of optimizations
if you have LTCG/LTO in your toolset, but the speed-up is still unique.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#495 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFTc-gjN8mFSXN1JsschOURIXHN37Fpiks5qd7TEgaJpZM4ITgPB
.

[bobsummerwill]

Some more unity-build links, @nerdralph:

• https://www.google.ca/?ion=1&espv=2#q=setting+up+unity+builds+C%2B%2B
• https://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/

RE: Talking to Genoil. Great!

[nerdralph]

@bobsummerwill The info you provided on unity/blob builds shows it to be a hack, and it would even seem to break some things (like static in C).
Avoiding nested includes is one thing I recommend for faster compilation and cleaner code. It's one of the Chromium coding guidelines:
https://www.chromium.org/developers/coding-style/cpp-dos-and-donts

[nerdralph]

Just did some testing and found out that lto breaks the build with g++-5.3 and 6.1 :-(

[bobsummerwill]

Oh yeah, @nerdralph.

It absolutely is a hack, but a hack with beautiful upside in terms of developer productivity.

It is very easy to counter the downsides, by maintaining both "loose" and 'BulkBuild" modes within your automated build process, but just using "BulkBuild" by default for your local builds.

You do sometimes need to make minor changes to avoid symbol clashes when force-combining compilation units in that way, but they are surprisingly easy to resolve.

The upside is that your builds are like 10x faster, which is a productivity benefit which it is very hard to argue with.

RE: Nested includes ... I don't see that in your referenced Chromium guide.

I have seen old advice saying that you should never include headers within headers, which is god-awful advice.

The bible for a lot of this physical dependencies stuff is 10 years old now, but most C++ developers have never even heard of it :-)

https://www.amazon.ca/Large-Scale-Software-Design-John-Lakos/dp/0201633620

[bobsummerwill]

What do those LTO breaks look like, @nerdralph?

[nerdralph]

With -flto I was getting lots of undefined refernce errors with gcc-5.3 and gcc-6.1, but not with 4.8.
I remembered having a similar problem before with lto and gcc-4.9, and found where I had read about it:
http://hubicka.blogspot.ca/2014/04/linktime-optimization-in-gcc-2-firefox.html

As a quick hack, I tried renaming ar to ar-binutils and making a symlink to gcc-ar, only to find that it crashed my machine when building!

[nerdralph]

@bobsummerwill Re build speedups, I assume you are talking about on Windoze/VC++. Real developers use Unix ;-)
Regarding the nested includes in the Chromium guide, see the heading "Don't include unneeded headers"
I've never heard of Lakos, and I've been doing C++ for over 25 years (since the cfront days), and have work on really large projects like telecom software at Nortel. The only bibles I know of in C/C++ are by K&R and Stroustrup.

[nerdralph]

Solved the lto issue with by setting CMAKE_AR and CMAKE_RANLIB to gcc-ar and gcc-ranlib.
I was expecting additional size reduction compared to 4.8.4, but with 6.1 the binary is only about 1% smaller.

[bobsummerwill]

Build speedups are for all compilers, because all C++ toolsets are subject to the redundant work per compilation unit for the pre-processor. Well, maybe pre-compiled headers can help a bit, but they really don't.

"Bible" in terms of physical dependencies in C++, which is something that most people don't even think about. Do get yourself a copy if you haven't seen it. I think that you will really enjoy it, though some of it looks a little dated.

1% smaller? Yeah - not so hot!

[bobsummerwill]

I was just dicking about with BulkBuild on my MacBook Air.
Got ethminer build down to 3 mins.

[nerdralph]

@bobsummerwill If there was a way for you to lend me a copy of it, I'd read it, but I can't think of any books I consider worth $54! My favorite book, "Blink" by Malcolm Gladwell costs less than $20 in paperback.

[nerdralph]

I just timed the build with gcc-6.1 (lto enabled), G1840 dual-core celeron, 2G RAM, 400G 2.5" SATA HD.
[100%] Built target ethminer
real 2m29.466s
user 2m21.188s
sys 0m5.648s

[bobsummerwill]

Second hand copies on Amazon from $2.50 + ~$4 for shipping.

Nice. Linux certainly pisses on Mac for build times. Your build times will be crazy fast with BulkBuild added too.

I'm seeing about 30mins -> 10 mins for the whole of cpp-ethereum, with some more work to do to unhack it into a state where I could commit.

[bobsummerwill]

Fat finger!

[nerdralph]

@bobsummerwill That was just a single-threaded make. make -j2 is even faster:
[100%] Built target ethminer
real 1m32.557s
user 2m30.144s
sys 0m6.228s

Normal development builds where you change a file or two and rebuild is in the range of 30s.

[nerdralph]

@bobsummerwill I found some of his book online (he published several articles in the C++ Report that went into the book). Seems pretty mundane, and nothing I didn't already know so far.
The example of LakosianFooBars in the interface of the Bank class seems to be exactly what I was talking about when it comes to avoiding nested includes.

p.s. And the more I read the less I like it. His Stack class uses an int for the index (d_sp) and size, while a smart developer would use unsigned int.

[bobsummerwill]

Well - no doubt plenty of his examples look dated for syntax, yes.
C++ syntax has advanced a lot since 1996.

His writing is primarily on physical dependencies and architecture - and really on the consequences of header files, compilation units, etc.

[nerdralph]

C++ has always had unsigned, and the pitfalls of using int for things that
should only have positive values was known even before C++ was invented.

On Aug 9, 2016 00:45, "Bob Summerwill" notifications@github.com wrote:

Well - no doubt plenty of his examples look dated for syntax, yes.
C++ syntax has advanced a lot since 1996.

His writing is primarily on physical dependencies and architecture - and
really on the consequences of header files, compilation units, etc.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#495 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFTc-qE-d-segQIsqQBEtOYgqnxNDa88ks5qd_fYgaJpZM4ITgPB
.

[bobsummerwill]

Great - so my EOD time for cpp-ethereum building (all apps) is now 7 mins on my machine, where that process currently takes around 30-46 mins in Travis, depending on the OS X version.

I like those numbers. Will clean it up tomorrow, to make a PR which could be submitted soon after the repo-move back to ethereum/cpp-ethereum. I'll do a PR-on-a-PR-on-a-PR:

merge_repos -> cmake_fixes -> bulkbuild_support.

Good night!