Design a system to provide binaries of C++ libraries built for different ABIs

[lucaswoj]

@jfirebaugh explained the fundamental issue:

... there is an issue with using prebuilt ​_C++_​ libraries with mason that so far we don't have a solution for: C++ libraries that use C++ standard library objects in their APIs then have an ​_ABI_​ dependency on the C++ standard library they were build with. They can't be linked into a program that uses a different C++ standard library. But precompiled mason libraries are ​_not namespaced by C++ stdlib_​.

In other words: ​_mason is not currently safe to use with C++ libraries_​.

Unless you force it to build from source.

the majority of libraries we use mason for are ​_C_​ libraries, not ​_C++_​ libraries

This ticket is for brainstorming and designing solutions to this problem. Should Mason to provide a set of binaries for C++ libraries built different stdlib versions? How should we go about building such a system?

cc @jfirebaugh @mikemorris @springmeyer @ljbade

ref mapbox/mapbox-gl-native#2348

[ljbade]

Specific to Android only:
Currently the Android NDK packages are built with only libc++. So we know on Android all binaries have been built with the same C++ library.

(I use libc++ since Android's stdlibc++ is not compatible with mapbox-gl-native)

[danpat]

Example of this in another setting:

http://boost.teeks99.com/

In some ways, it's no different to having MacOS binaries that can't operate on Linux. Any ABI incompatibility should probably just become part of the platform string, and we'll need to be more explicit about dependencies (i.e. you can't just depend on the linux binary, you have to depend on the linux-libstdc++5).

When we publish any binaries, we should include any ABI-sensitive parts in the platform component.

i.e. MASON_PLATFORM needs to be a bit more verbose for C++ projects.

[lucaswoj]

Any ABI incompatibility should probably just become part of the platform string

Sounds great @danpat! Will this be as simple as

• creating a way to mark / detect packages that require a binary per C++ ABI
• setting up Travis to recognize marked packages and build a binary per C++ ABI

[mikemorris]

Been digging and can't find a simple way to print the C++ stdlib version a package was compiled with on OS X. ldd looks like it might work for grabbing this on Linux though.

From reading http://www.trilithium.com/johan/2005/06/static-libstdc/, could building C++ packages with gcc and -static-libstdc++ be an option?

The new -static-libstdc++ option directs g++ to link the C++ library statically, even if the default would normally be to link it dynamically.
https://gcc.gnu.org/gcc-4.5/changes.html

[danpat]

For OS/X, use otool:

$ otool -L ~/mapbox/osrm-backend-clean/build/osrm-extract | grep c\+\+
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

On my system, I seem to have both libc++ and libstdc++:

$ ls -l /usr/lib | grep c\+\+
-rwxr-xr-x   1 root  wheel  1414480 Sep  9  2014 libc++.1.dylib
lrwxr-xr-x   1 root  wheel       14 Apr 14 13:06 libc++.dylib -> libc++.1.dylib
-rwxr-xr-x   1 root  wheel   441136 Sep  9  2014 libc++abi.dylib
-rw-r--r--   1 root  wheel    10260 Jul 29 01:56 libkmodc++.a
-rwxr-xr-x   1 root  wheel  1459408 Sep  9  2014 libstdc++.6.0.9.dylib
lrwxr-xr-x   1 root  wheel       21 Apr 14 13:06 libstdc++.6.dylib -> libstdc++.6.0.9.dylib
lrwxr-xr-x   1 root  wheel       17 Apr 14 13:06 libstdc++.dylib -> libstdc++.6.dylib

Static linking would only fix things if the symbols can be hidden, otherwise they'll possibly conflict when your library is linked upstream and another lib(std)c++ is linked in with similar symbol names. I'm not sure if you can hide the symbols for other libraries that you statically link in, it might only apply to stuff you compile yourself (see https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/CppRuntimeEnv/Articles/SymbolVisibility.html)

[kkaefer]

could building C++ packages with gcc and -static-libstdc++ be an option?

That would likely work, but cause larger binaries since parts of the stdlib need to be included in the executable rather than loading the shared lib.

[jfirebaugh]

I'm not seeing how linking C++ stdlib statically would help. The issue is in the following scenario:

• Library foo build by mason uses C++ stdlib A in the public interface (e.g., methods accept or return std::string).
• Downstream project depends on foo, uses C++ stdlib B.
• A and B have incompatible ABIs.

Whether foo links A dynamically or statically doesn't matter -- in either case C++ stdlib A's ABI is part of the interface, and will conflict with downstream projects that use B.

[danpat]

Partly for my own education, and partly to help this discussion, I ran a little experiment to show what ABI compatibility really means:

test.hpp

#include <string>
struct Foo {
  void Bar(int a);
  void Baz(const std::string &x);
};

test.cpp

#include <string>
#include <iostream>
#include "test.hpp"

void Foo::Bar(int a) {
    std::cout << a << std::endl;
}
void Foo::Baz(const std::string &x) {
    std::cout << x << std::endl;
}

Compiling this on Linux against both libc++ and libstdc++ gives:

root@ip-172-31-15-63:~# clang++ --stdlib=libc++ -c test.cpp
root@ip-172-31-15-63:~# objdump -t test.o | grep Baz
0000000000000000 l    d  .text._ZN3Foo3BazERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE   0000000000000000 .text._ZN3Foo3BazERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE
0000000000000000  w    F .text._ZN3Foo3BazERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE   0000000000000045 _ZN3Foo3BazERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE

root@ip-172-31-15-63:~# g++ -c test.cpp
root@ip-172-31-15-63:~# objdump -t test.o | grep Baz
0000000000000000 l    d  .text._ZN3Foo3BazERKSs 0000000000000000 .text._ZN3Foo3BazERKSs
0000000000000000  w    F .text._ZN3Foo3BazERKSs 0000000000000030 _ZN3Foo3BazERKSs

So, what this means is that when someone has test.o and test.hpp, they need to be able to turn the function signatures in test.hpp into ones that match the symbols in test.o for linking to be successful. It's sort of a one-way hash: the linker can't look for possibly matching symbols, the compiler has to be able to generate matching symbols from the beginning. The symbols in test.o depend on what I compiled it with, and if the consumer doesn't know or have that combo, they're going to have a hard time linking.

https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

Putting all of these ideas together results in the C++ Standard Library ABI, which is the compilation of a given library API by a given compiler ABI. In a nutshell:
“ library API + compiler ABI = library ABI ”

Now, what is important to understand is that C types have a pretty well defined symbol generation rules that all compilers seem to agree on:

root@ip-172-31-15-63:~# g++ -c test.cpp
root@ip-172-31-15-63:~# objdump -t test.o | grep Bar
0000000000000000 l    d  .text._ZN3Foo3BarEi    0000000000000000 .text._ZN3Foo3BarEi
0000000000000000  w    F .text._ZN3Foo3BarEi    000000000000002d _ZN3Foo3BarEi
root@ip-172-31-15-63:~# clang++ --stdlib=libc++ -c test.cpp
root@ip-172-31-15-63:~# objdump -t test.o | grep Bar
0000000000000000 l    d  .text._ZN3Foo3BarEi    0000000000000000 .text._ZN3Foo3BarEi
0000000000000000  w    F .text._ZN3Foo3BarEi    0000000000000043 _ZN3Foo3BarEi

Here, because the class function only uses void and int, both compilers are generating the same function signatures. Thus, the Foo::Bar function will likely be findable by people using different compilers trying to link to this library, but Foo::Baz will need the same C++ library.

So, next steps:

1. We should avoid exposing stdlib types in our public API (http://programmers.stackexchange.com/a/183981).
2. Where we have to, we will need to publish platform specific binaries, where "platform" includes both OS and stdlib, e.g. "linux-stdc++4.8".
3. We should be aware that it might get worse than that, libc++/libstdc++ are not the only things like this (if we expose/consume boost types, we would also likely need to do boost-specific versions, like linux-stdc++4.8-boost-1.55).

[mikemorris]

@danpat otool -L wasn't working for me on OS X since I'm trying to inspect static libraries? Likewise, nm mason_packages/linux-x86_64/geojsonvt/1.1.0/lib/libgeojsonvt.a | grep c\+\+ doesn't give me anything...

[danpat]

@mikemorris Ah, right, static. Best you can do is extract the symbol table from that (i.e. what functions exist in the module) using otool -v -S libgeojsonvt.a, which gives things like:

$ otool -v -S libgeojsonvt.a | grep convertFeatures
geojsonvt.o      __ZN6mapbox4util9geojsonvt9GeoJSONVT15convertFeaturesERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEhdb

On Linux, use objdump -t or nm:

# objdump -t libgeojsonvt.a | grep convertFeatures
0000000000000000 g     F .text  00000000000004ec _ZN6mapbox4util9geojsonvt9GeoJSONVT15convertFeaturesERKSshdb

This looks like it corresponds to:

namespace mapbox {
namespace util {
namespace geojsonvt {

class GeoJSONVT {
public:
    static std::vector<ProjectedFeature> convertFeatures(const std::string& data,
                                                         uint8_t baseZoom = 14,
                                                         double tolerance = 3,
                                                         bool debug = false);
}

These symbol names are generated by the compiler/api combination they're built with (see my previous comment), and when you want to link against libgeojsonvt.a, you need to have a compiler/lib combination that generates matching symbol names.

AFAIK, there will be no other info in the .a file about how they were built. Dynamic libs (.so/.dylib) at least give you a filename to work with. This is why I'm suggesting we make our <platform> a bit more verbose.

[mikemorris]

I'm a bit stuck here, I found -dumpversion which prints a simple version number for both clang and gcc, and it looks like I need major.minor from the GCC ABI Policy, but I can't figure out how to determine what compiler is being used on Linux builds.

clang is explicitly set on OS X and Android builds, and I assume it would be safe to hardcode on iOS, but I'm not seeing anything comparable on Linux, and while CXX is set for Travis builds I don't think there is any guarantee for it to be set locally?

Is there another way to check what compiler/stdlib is available for a consumer trying to fetch a binary?

[mikemorris]

The path I'm trying to implement is to add MASON_CXX_VERSION in mason.sh and only append it to the upload/download path if a flag like MASON_CXX: true is set in the package script.

[ljbade]

Are there any special compiler #define that provides compiler version? Perhaps have a stub C++ app that simply prints that value of that to the console, then read the output. It should be compiled with the mason environment.

[mikemorris]

That may be possible with builtin macros for clang and gcc (or __GLIBCXX__) @ljbade. This might work but I wonder if it's too much overhead to compile this code to grab the version before building each library?

[mikemorris]

Tried setting _GLIBCXX_USE_CXX11_ABI downstream to no avail.

[springmeyer]

I think we should pause on trying to support multiple standard library versions in Mason. And therefore put on ice #115.

In the future this may be useful and therefore #115 will be great, but at this point I'm not seeing any concrete need to support both libstdc++ and libc++. This is because the originating ticket that seemed to require stdlib versioning I think is not solved by stdlib versioning: mapbox/mapbox-gl-native#2237 (comment).

Instead I think we should continue for now with the assumption that Mason packages (and only works with):

• libc++ on OS X
• libc++ on Android
• libstdc++ on Linux

And therefore the next action is not versioning on stdlib but rather narrowing down what the actual cause of linking errors are when using geojson-vt compiled with clang++-3.5 against mgbl compiled with g++-4.9. Because the problem that prompted this effort was specific to mgbl let's track things over at mapbox/mapbox-gl-native#2502.

[jfirebaugh]

Putting this on ice per above.