FLiT shared library and C++ standard library incompatibility

[IanBriggs]

I've run into errors trying to use newer versions of gcc and clang, as well as errors when compiling flit with one version of a compiler and using another version for the tests.
Problem 1:
The first is caused by different c++ standard libraries. As of version 5.0 gcc uses two abi incompatible versions of libstdc++, they call this 'dual abi'.

See https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html.

To force gcc to use the same abi it used before 5.0 the command line argument -D_GLIBCXX_USE_CXX11_ABI=0 needs to be used.

Along the same vein llvm has introduced their own c++ standard library. This is documented at https://libcxx.llvm.org/. The fix for this is -stdlib=libstdc++.

Problem 2:
When compiling flit itself with a different version of gcc then used in the tests I have run into errors along the lines of "version GLIBC_2.2.5 not defined in file libc.so.6". The quick fix is to recompile flit with the version used for the tests, but this is a special case of a more general problem. When a compiler uses it's own libraries and leaves resolution to link time, but another compiler is used to perform linking then expected symbols or even whole libraries can be missing. This is a problem seen when using the xlc compiler, as it will want to link against libibmc++ and libxlopt.

Solutions:
For the first problem, the flags make everything link to a common abi compatible stdlibc++.
The solution to the second problem is not so clear. Since GLIBC is backwards compatible we could use the libc.so.6 that has the most up to date versions, but this leaves the general case of compiler libraries.

[ganeshutah]

Thanks for the brave debugging ventures, Ian. Like Mike said, do not stress excessively. We are trying a complex experiment for which such breakages of software are impending. Mike also likes to involve in writing more, and not submit a version that we do not get time to polish. Given all this, let's still plug away, but not over-stress. Let's do a good job. I'm going to continue to write. If some magic happens and you get good #s and working libraries by evening, one push to finish is worthwhile. Else we cool off and seek a new venue. Not to deflate anyone, but telling everyone what's reality and what's not :-) I'm sitting in my backyard and writing (too many students looking for me as an assignment is due) Watching bees and writing is fun

…

On Fri, Sep 7, 2018 at 1:20 PM Ian Briggs ***@***.***> wrote: I've run into errors trying to use newer versions of gcc and clang, as well as errors when compiling flit with one version of a compiler and using another version for the tests. Problem 1: The first is caused by different c++ standard libraries. As of version 5.0 gcc uses two abi incompatible versions of libstdc++, they call this 'dual abi'. This website documents their change: https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html <http://url>. To force gcc to use the same abi it used before 5.0 the command line argument -D_GLIBCXX_USE_CXX11_ABI=0 needs to be used. Along the same vein llvm has introduced their own c++ standard library documented at this website https://libcxx.llvm.org/ <http://url>. The fix for this is -stdlib=libstdc++. Problem 2: When compiling flit itself with a different version of gcc then used in the tests I have run into errors along the lines of "version GLIBC_2.2.5 not defined in file libc.so.6". The quick fix is to recompile flit with the version used for the tests, but this is a special case of a more general problem. When a compiler uses it's own libraries and leaves resolution to link time, but another compiler is used to perform linking then expected symbols or even whole libraries can be missing. This is a problem seen when using the xlc compiler, as it will want to link against libibmc++ and libxlopt. Solutions: For the first problem, the flags make everything link to a common abi compatible stdlibc++. The solution to the second problem is not so clear. Since GLIBC is backwards compatible we could use the libc.so.6 that has the most up to date versions, but this leaves the general case of compiler libraries. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#210>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEyiCmFfSSBvS5hEQZpdSfAjxhGDJfWSks5uYsbdgaJpZM4WfWJS> .

[ganeshutah]

Ian, The problems you describe are of a general nature that patchwork solutions won't get us anywhere. If we realize FLiT, we should have these checks performed a priori before bisect etc are launched. That means there are more paper opportunities hidden here. Some of our papers could be just the precise question of how to aggressively mix compilers. We can also investigate the ABI issues. The work is at a good point, but if we cannot be completely happy with our build and deploy methods, we can't claim a lot in our paper. So let's try to solve all this today but working sanely. I'll meanwhile write more, and also be ready with other deadlines we can aim for. In particular, PLDI is a good venue. It is highly prestigious. We can develop a TON of good material. It is in November. Gives enough time. Just a thought Ganesh On Fri, Sep 7, 2018 at 1:27 PM Ganesh Gopalakrishnan <ganesh@cs.utah.edu> wrote:

…

Thanks for the brave debugging ventures, Ian. Like Mike said, do not stress excessively. We are trying a complex experiment for which such breakages of software are impending. Mike also likes to involve in writing more, and not submit a version that we do not get time to polish. Given all this, let's still plug away, but not over-stress. Let's do a good job. I'm going to continue to write. If some magic happens and you get good #s and working libraries by evening, one push to finish is worthwhile. Else we cool off and seek a new venue. Not to deflate anyone, but telling everyone what's reality and what's not :-) I'm sitting in my backyard and writing (too many students looking for me as an assignment is due) Watching bees and writing is fun On Fri, Sep 7, 2018 at 1:20 PM Ian Briggs ***@***.***> wrote: > I've run into errors trying to use newer versions of gcc and clang, as > well as errors when compiling flit with one version of a compiler and using > another version for the tests. > Problem 1: > The first is caused by different c++ standard libraries. As of version > 5.0 gcc uses two abi incompatible versions of libstdc++, they call this > 'dual abi'. This website documents their change: > https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html > <http://url>. To force gcc to use the same abi it used before 5.0 the > command line argument -D_GLIBCXX_USE_CXX11_ABI=0 needs to be used. > > Along the same vein llvm has introduced their own c++ standard library > documented at this website https://libcxx.llvm.org/ <http://url>. The > fix for this is -stdlib=libstdc++. > > Problem 2: > When compiling flit itself with a different version of gcc then used in > the tests I have run into errors along the lines of "version GLIBC_2.2.5 > not defined in file libc.so.6". The quick fix is to recompile flit with the > version used for the tests, but this is a special case of a more general > problem. When a compiler uses it's own libraries and leaves resolution to > link time, but another compiler is used to perform linking then expected > symbols or even whole libraries can be missing. This is a problem seen when > using the xlc compiler, as it will want to link against libibmc++ and > libxlopt. > > Solutions: > For the first problem, the flags make everything link to a common abi > compatible stdlibc++. > The solution to the second problem is not so clear. Since GLIBC is > backwards compatible we could use the libc.so.6 that has the most up to > date versions, but this leaves the general case of compiler libraries. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#210>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AEyiCmFfSSBvS5hEQZpdSfAjxhGDJfWSks5uYsbdgaJpZM4WfWJS> > . >

[mikebentley15]

I found documentation from Intel specifying how to specifically choose which gcc is used in the backend. You would want to use the -gxx-name flag. We may also want to consider using -ffat-lto-objects when linking if we are linking using the Intel compiler.

See https://software.intel.com/en-us/cpp-compiler-18.0-developer-guide-and-reference-gcc-compatibility-and-interoperability

[mikebentley15]

Currently the -gxx-name flag is not being used. The framework should be using the --gcc-toolchain flag, but as I found out, at least for clang version 3.8 and older, we may need to create our own directory structure with the gcc path symbolically linked in (see https://stackoverflow.com/a/48940188). This is something to investigate. Hopefully, it should be easy to discover with the -v and -print-search-dirs flags to clang.

I like Ian's suggestion to add to the flit-config.toml, an entry of which gcc to use for the standard library. For both clang and intel types of compilers, the required field of gcc_toolkit could be added. Or we could use the gcc type compiler specified in flit-config.toml (if there is one) and assume we are to use that with each compiler. The latter is the current approach for Clang, but is not used for Intel.

As for me, I'd prefer to do the latter approach of using the gcc toolkit provided by the flit-config.toml script. However, that poses a problem if one is not specified, and we are using more than one compiler. We could do one of two things in that case:

1. Use the gcc that is in the path, causing an error if there is none
2. Require the gcc_toolchain argument in flit-config.toml only if a gcc type compiler is not present

I would prefer number 1. If we choose to do it this way, then it would need to be documented what we are doing and why.

[mikebentley15]

It seems we have decided to move away from this direction of automagically inserting the correct --gcc-toolchain flag for Clang and --gxx-name flag for Intel. Instead, we give users the ability to specify their own flags for each compiler to induce the same behavior. However, we have added documentation describing how to go about doing this sort of thing using the features we have implemented.

I propose that we add further documentation on how to achieve these results for Clang to use a particular GCC from a standard installation (e.g., g++-6 installed in /usr/bin right next to g++ which is version 4.8).

We were thinking of adding another feature to the flit-config.toml where each compiler would have a field specifying which standard c++ library they want used. We could go that route, but I think it better to document and teach users to do it themselves and give them the power to enact it inside of the configuration file.

[mikebentley15]

I now keep this issue around to document the following problem:

Problem

When compiling the FLiT shared library, one has to decide which standard library implementation they wish to use. Only that one standard library implementation will be available for their use when they are trying to use FLiT.

I find it extremely limiting to allow only one standard library implementation to be explored, especially in situations where the user may want to explore differences when using various standard library implementations.

Proposed Solution

For each specified compiler, either within flit-config.toml or on the command-line to flit bisect, we compile either

• a separate libflit.so shared library, or
• a separate libflit.a static library

We would not need to do this for every compilation, but rather just for each compiler, since the compiler should be static about which standard library is being used.

Note: This assumption may be wrong if the user is specifying different standard libraries in their search space of flags. In that case, we would want to recompile the flit library for each compilation.

I propose that we create a separate libflit.a static library for each compiler used, and to create these "on the fly" within the FLiT directory. That would have a few repercussions

1. Compile-time would increase. This would be significant for small projects, and insignificant for large projects. For instance, this would likely significantly increase the time to run the automated tests.
2. Disk usage would increase. This is minor since libflit is small
3. FLiT itself would not need to be compiled, instead the source code for libflit would be installed and built in each flit test directory
4. The autogenerated Makefile and build.ninja files would become more complicated as they now need to be able to compile multiple versions of libflit.a for each compiler.

Alternative Approaches

• Turn FLiT itself into a header-only library so that it gets recompiled every time
• Allow for multiple compilations of FLiT to be done and installed. This could use the Module system to reload the correct flit shared library depending on which compiler is used.
• Allow for caching of compiled libflit libraries, perhaps in /tmp or somewhere else. When the same one is requested, it uses the cached one instead.
• Something else I've not thought of...

Discussion

I am open for discussion on this topic. It is a fairly large rework of FLiT's architecture. We can talk in person if desired, but I think it would be good to capture here any relevant points from discussions.

[mikebentley15]

One other option is to compile the FLiT source code into each compilation as a separate set of source files (so that they are not included in bisect search).

With this option, we can do what's called a "Unity Build" where all of the FLiT code will be compiled into a single object file. This significantly reduces the built time for FLiT. In order to do this, you make a single source file that does #include for all the other source files. This new source file is the only one that needs to be compiled.

[mikebentley15]

Unity build resources:

• The Magic of Unity Builds
• Make VC++ Compiles Fast Through Parallel Compilation
• Cotire: An obsolete CMake module that added Unity Build functionality
• CMake Unity Build Documentation (since version 3.16)

[mikebentley15]

I did a timing experiment to determine which approach is best:

Header-Only Timing Experiment

I now want to estimate the effect of using header-only FLiT to compile FLiT
tests. Here I test the timing it takes to compile these projects sequentially
on my laptop (i.e., make -j1).

Compiler Versions

compiler	version
g++	9.2.0

Empty Test

Running only on the project containing a single empty test.

source version	compilation	time
original	g++ -O0	4.46s
header-only	g++ -O0	5.74s
original	g++ -O3 -funsafe-math-optimizations	9.70s
header-only	g++ -O3 -funsafe-math-optimizations	14.53s

For g++ -O0, it is a 28.6% increase in compile time. For g++ -O3 -funsafe-math-optimizations, it is a 49.8% increase in compile time.

Granted, this is the worst-case situation where there is absolutely no user
code and there are two object files to be compiled that are both utilizing the
FLiT header-only library.

Litmus Tests

source version	compilation	time
original	g++ -O0	57.89s
header-only	g++ -O0	72.46s
original	g++ -O3 -funsafe-math-optimizations	120.16s
header-only	g++ -O3 -funsafe-math-optimizations	147.58s

For g++ -O0, it is a 25.2% increase in compile time. For g++ -O3 -funsafe-math-optimizations, it is a 22.8% increase in compile time.

This is also nearly a worst-case scenario since there are many tests that are
miniscule in size, especially compared to the FLiT library.

MFEM Tests

source version	compilation	time
original	g++ -O2	472.1s
header-only	g++ -O2	529.6s
original	g++ -O3 -funsafe-math-optimizations ...	560.5s
header-only	g++ -O3 -funsafe-math-optimizations ...	621.0s

For g++ -O2, we see a 12.2% increase in compile time. For g++ -O3 ..., we
see a 10.8% increase in compile time.

This is more like a real-world situation. I'm actually surprised that the
increase in compile-time is so much in this case. There are quite a few tests,
but there are a large number of other source files not related to FLiT. The
amount of source code from the MFEM side is significantly more than the FLiT
side.

It's just that for the tests, each test is essentially compiling its own
version of the flit library code.

FLiT source compiled in timing experiments

Here, we want to see the impact of compiling the FLiT source code with each
compilation instead of the header-only compilation. Here we will compare with
both the original compilation-time (of which it must be an increase) and the
header-only solution.

Empty Test

Running only on the project containing a single empty test.

source version	compilation	time	% overhead
original	g++ -O0	4.46s	-
header-only	g++ -O0	5.74s	28.6
src-in	g++ -O0	11.87s	166.1
src-all-in-one	g++ -O0	7.41s	66.1
original	g++ -O3 -funsafe-math-optimizations	9.70s	-
header-only	g++ -O3 -funsafe-math-optimizations	14.53s	49.8
src-in	g++ -O3 -funsafe-math-optimizations	22.92s	136.3
src-all-in-one	g++ -O3 -funsafe-math-optimizations	17.46s	80.0

Litmus Tests

source version	compilation	time	% overhead
original	g++ -O0	57.89s	-
header-only	g++ -O0	72.46s	25.2
src-in	g++ -O0	64.17s	10.8
src-all-in-one	g++ -O0	62.01s	7.1
original	g++ -O3 -funsafe-math-optimizations	120.16s	-
header-only	g++ -O3 -funsafe-math-optimizations	147.58s	22.8
src-in	g++ -O3 -funsafe-math-optimizations	131.44s	9.4
src-all-in-one	g++ -O3 -funsafe-math-optimizations	126.14s	5.0

MFEM Tests

source version	compilation	time	% overhead
original	g++ -O2	472.1s	-
header-only	g++ -O2	529.6s	12.2
src-in	g++ -O2	482.7s	2.2
src-all-in-one	g++ -O2	479.8s	1.6
original	g++ -O3 -funsafe-math-optimizations ...	560.5s	-
header-only	g++ -O3 -funsafe-math-optimizations ...	621.0s	10.8
src-in	g++ -O3 -funsafe-math-optimizations ...	573.3s	2.3
src-all-in-one	g++ -O3 -funsafe-math-optimizations ...	567.3s	1.2