Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943

cosmicexplorer · 2018-06-08T23:15:34Z

Problem

The native backend subsystems tests introduced in #5490 are still skipped, complaining about crti.o on linux, which is part of libc. See #5662 -- clang's driver will find the directory containing that file on travis, but gcc won't. We should make a way to find this file (which is necessary for creating executables) so we can unskip the native backend testing.

Solution

Fix a mistake in Allow alternate binaries download urls generation and convert GoDistribution and LLVM subsystems to use it #5780 -- we did not check the correct directory with os.path.isdir(), so we never found the LLVM BinaryTool when downloading it from the LLVM download page.
Find libc using the new LibcDev subsystem. This uses the option --libc-dir, if provided, or finds an installation of libc with crti.o by invoking --host-compiler on the host once to get its search directories (this is necessary on travis, due to ubuntu's nonstandard installation location).
Expand the definition of executables, compilers, and linkers in src/python/pants/backend/native/config/environment.py to cover everything needed to actually compile, and give them the ability to generate an environment suitable for passing into subprocess.Popen().
Introduce GCCCCompiler, GCCCppCompiler, LLVMCCompiler, and LLVMCppCompiler to differentiate between the two different compilers we have available for each language.
Expose the libc lib directory to the compilers we create in native_toolchain.py.
Unskip tests in test_native_toolchain.py from combine Clang, GCC, Binutils, and XCodeCLITools to form the NativeToolchain subsystem and consume it in BuildLocalPythonDistributions for native code #5490.
Make the default CCompiler and CppCompiler into clang, for no particular reason (it will pass CI with gcc as well).

The different compilers we can use will likely be denoted using variants in the future, but this solution allows us to separate the resources generated from each subsystem (GCC, LLVM, Binutils, XCodeCLITools) from a fully-functioning compiler that can be invoked (because actually invoking either clang or gcc requires some resources from the other, e.g. headers and libraries). Now, each binary tool only does the job of wrapping the resources it contains, while native_toolchain.py does the job of creating either a clang or a gcc compiler that we can invoke on the current host (with all necessary headers, libs, etc).

Result

The native toolchain tests from #5490 can be unskipped, and we can invoke our toolchain on almost any linux installation without further setup. The tests in test_native_toolchain.py now pass in CI, and as we iterate on the native backend, we will continue to have end-to-end testing for both of our compilers, on osx as well, regardless of whichever one we choose to use for python_dist() compilation.

cosmicexplorer · 2018-06-08T23:27:29Z

This would completely resolve one of the FIXMEs currently left in #5815, which includes /usr/bin in the PATH when compiling native code in python_dist() sources (not through ctypes), if it works. As mentioned in pantsbuild/binaries#70, I can't figure out to repro the issue happening on Travis, even when I try to invoke pants from the docker images in this repo. If there are some canonical instructions for how to do that, it would be helpful in testing this PR.

benjyw · 2018-06-10T00:36:40Z

src/python/pants/backend/native/subsystems/native_toolchain.py

+
+  @memoized_method
+  def glibc_dir(self):
+    # FIXME: this can be removed once #5815 lands (GCC should be made to depend on GLibc on Linux).


It's not clear what the "this" is that can be removed.

Made it a lot more specific in c3aa525, thanks!

illicitonion

Works for me :) Thanks!

cosmicexplorer · 2018-06-14T03:31:54Z

Had a good discussion with @kwlzn on the slack today, and I now believe this should be replaced with a method of searching for crti.o and other files that are necessarily host-specific, very much like what we're doing with XCodeCLITools. I will refactor this PR and comment again when it is finished.

cosmicexplorer · 2018-06-14T17:49:56Z

It looks like we'll have to invoke the system gcc -print-search-dirs, then parse the output of that to find candidates for dirs containing crti.o (I already have code to do this, this interface is very stable). Since we're not using the system gcc to compile anything (so that we can e.g. support newer c++ features), we have to do this bootstrap step in order to create executables and to compile python_dist()s, because the location of the host-specific crti.o is partially baked into the gcc installation (as in, contains stuff that the distro gcc package maintainer added). We'll also need to add the libc6-dev package to the trusty images we use in CI (otherwise, they don't have crti.o). The native backend subsystem tests will ensure we can use clang to compile c/c++ as well using this -- the only reason we need the system gcc is to get the location of crti.o.

The above is all only for Linux hosts. We'll have a good error message like XCodeCLITools if crti.o could not be found. Implementing this now.

cosmicexplorer · 2018-06-15T04:08:37Z

This PR should be ready for review again. Instead of providing glibc, we are finding the host's libc installation by parsing the output of gcc -print-search-dirs from the host gcc (this is done in a configurable way, so not necessarily with the host's gcc using --host-compiler, or at all if the libc location is known in advance, by providing --libc-dir). We don't just need libc, we need what some distros label libc-dev or libc6-dev, which provides crti.o, which is necessary to e.g. create executables or compile python_dist()s. This lets us pass the now-unskipped native backend subsystem tests.

cosmicexplorer · 2018-06-15T06:12:49Z

According to the CI failures on the unit tests, we need to expose our provided gcc's libstdc++.so.6 to clang through LIBRARY_PATH for it to work if we set LIBRARY_PATH to use the host libc installation. I'll work on doing that now.

stuhood · 2018-06-15T18:37:58Z

will wait to review.

cosmicexplorer · 2018-06-28T17:07:55Z

I have updated the OP and CI is green. Please take another look -- this was extremely difficult and time-intensive to get working (mostly because it's very difficult to reproduce the travis environment, and yes I really did try), but after this I am confident we will be able to iterate on the native backend (and the way we invoke compilers, linkers, etc) without fear, which seems like a useful result. The tools we have developed here are not specific to travis whatsoever and are generic enough to make a reliable basis for further extension.

stuhood

Thanks!

stuhood · 2018-06-28T20:09:28Z

src/python/pants/backend/native/subsystems/libc_dev.py

+  def register_options(cls, register):
+    super(LibcDev, cls).register_options(register)
+
+    register('--libc-dir', type=dir_option, default='/usr/lib', advanced=True,


Should both of these options be fingerprinted?

Yes!! Will fix and keep that in mind in the future.

Done in 5a1665b!

stuhood · 2018-06-28T20:16:22Z

src/python/pants/backend/native/subsystems/xcode_cli_tools.py

@@ -26,75 +24,160 @@ class XCodeCLITools(Subsystem):

  options_scope = 'xcode-cli-tools'

-  _REQUIRED_TOOLS = frozenset(['cc', 'c++', 'clang', 'clang++', 'ld', 'lipo'])
+  REQUIRED_FILES = {


Are these intentionally public? Would be good to encapsulate instead if possible, to avoid needing to break APIs to add more data.

Yeah, there's no reason for these to be public and the reason you mentioned is a fantastic reason to keep it private. Will fix.

stuhood · 2018-06-28T20:17:10Z

src/python/pants/backend/native/subsystems/xcode_cli_tools.py

-    register('--xcode-cli-install-location', type=dir_option, default='/usr/bin', advanced=True,
-             help='Installation location for the XCode command-line developer tools.')
+    register('--install-prefixes', type=list, default=cls.INSTALL_PREFIXES_DEFAULT, advanced=True,
+             help='Locations to search for resources from the XCode CLI tools, including a '


ditto fingerprint.

Done in 5a1665b!

stuhood · 2018-06-28T20:23:17Z

src/python/pants/engine/scheduler.py

+      self._scheduler.add_root_selection(native_execution_request, subject, product)
+    return ExecutionRequest(request_specs, native_execution_request)
+
+  # FIXME: note that this does a cross prod!


You updated the comment here... can remove.

Done in 5a1665b!

stuhood · 2018-06-28T20:23:59Z

src/python/pants/engine/scheduler.py

-    as "list of product".
+    An ExecutionRequest for an Address represents exactly one product output, as does
+    SingleAddress. But we differentiate between them here in order to normalize the output for all
+    Spec objects as "list of product".


I don't think this is true any longer... can nuke this paragraph probably.

Done in 5a1665b!

stuhood · 2018-06-28T20:25:07Z

src/python/pants/engine/scheduler.py

@@ -406,15 +406,25 @@ def visualize_graph_to_file(self, filename):
  def visualize_rule_graph_to_file(self, filename):
    self._scheduler.visualize_rule_graph_to_file(filename)

+  def execution_request_literal(self, request_specs):


I think that rather than adding a new method, we'd want to just update execute_request... it only has two callers outside of unit tests. But that would increase the scope here... if you're able to swap out the methods as a quick followup (not here, as this is already a bit large), that would be appreciated.

Also, the SchedulerSession.product_request and product_requests family is easier to use... so adding another method that is shaped like "list of tuples" would be ideal (and maybe even removing/deprecating ExecutionRequest as a public API).

That all sounds like a viable scope for a followup -- I'll make an issue.

Made into #6044!

…m tests" This reverts commit 5a28312.

…is doomed enterprise

…in.py, again

… osx

…ose yet

- add fingerprint=True to new subsystem options - make _REQUIRED_FILES private - add a useful comment - remove a now-false paragraph from the execution_request() docstring

stuhood · 2018-07-02T18:34:19Z

Feel free to merge when ready.

cosmicexplorer · 2018-07-02T18:44:54Z

Travis isn't showing the passing build in this thread for some reason, but 08ba46d has indeed passed CI, so merging now.

cosmicexplorer mentioned this pull request Jun 8, 2018

Add glibc binary pantsbuild/binaries#70

Closed

cosmicexplorer requested review from stuhood, benjyw, illicitonion and kwlzn June 10, 2018 00:27

benjyw reviewed Jun 10, 2018

View reviewed changes

benjyw approved these changes Jun 10, 2018

View reviewed changes

illicitonion approved these changes Jun 11, 2018

View reviewed changes

cosmicexplorer added the native label Jun 12, 2018

cosmicexplorer force-pushed the introduce-glibc-for-crti.o branch from bce6634 to 14b8a7d Compare June 13, 2018 18:20

cosmicexplorer changed the title ~~Introduce glibc BinaryTool on linux to source crti.o and unskip the native backend subsystem tests~~ Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests Jun 15, 2018

cosmicexplorer mentioned this pull request Jun 20, 2018

LLVM 6.0.0 binary not published #5991

Closed

cosmicexplorer force-pushed the introduce-glibc-for-crti.o branch 2 times, most recently from 2c23923 to 1335922 Compare June 27, 2018 09:05

cosmicexplorer mentioned this pull request Jun 27, 2018

Conan (third party) support for ctypes native libraries #5998

Merged

stuhood approved these changes Jun 28, 2018

View reviewed changes

cosmicexplorer mentioned this pull request Jun 28, 2018

expand product_request() and (maybe) deprecate ExecutionRequest #6044

Closed

cosmicexplorer added 6 commits June 29, 2018 11:52

depend on glibc on linux to compile python_dist()s

1de3dcb

fix llvm select() and unskip the native toolchain subsystem tests

7e4eb1f

update default version of glibc as per pantsbuild/binaries#70

673455c

remove glibc, and remove executable step from native subsystem tests

88e6050

set LDSHARED when invoking a setup.py and test making an exe with gcc

d0b9a41

Revert "remove glibc, and remove executable step from native subsyste…

56b5664

…m tests" This reverts commit 5a28312.

cosmicexplorer added 21 commits June 29, 2018 11:53

fix the broken stuff left over from the previous commit

318f5d5

actually add the include dirs to gcc, which was the whole point of th…

548b541

…is doomed enterprise

remove unused products dir

cf0347c

move almost all of the platform-specific nonsense into native_toolcha…

071ea81

…in.py, again

add version testing and fix tuple index

731eab8

use the LLVMCCompiler and LLVMCppCompiler from the NativeToolchain on…

045395d

… osx

refactor setup_py.py to add env vars manually because they don't comp…

fcb52df

…ose yet

add some search dirs back to gcc

4791dad

start some cleanup

b041a7d

remove HostLibcDevInstallation

42e0c60

refactor setup py native environment creation again

1449300

some more mild cleanup

8db17b4

flesh out the last few fixmes

54956c5

fix some mistakes -- hopefully travis agrees

315b17b

don't assign a list to the environment

12987d8

use gcc specifically to link

9dd3b28

rebasing is hard

ed14aef

use clang++ to link c++ and clear up a fixme

691db6f

add hopefully the final workaround

330eb2a

respond to review comments

a4fc486

- add fingerprint=True to new subsystem options - make _REQUIRED_FILES private - add a useful comment - remove a now-false paragraph from the execution_request() docstring

remove fake news

a4c9b3e

cosmicexplorer force-pushed the introduce-glibc-for-crti.o branch from 6520d79 to 08ba46d Compare June 29, 2018 19:01

use is_readable_dir wherever possible

08ba46d

cosmicexplorer merged commit 1d65d4a into pantsbuild:master Jul 2, 2018

cosmicexplorer mentioned this pull request Jul 9, 2018

provide glibc as another BinaryTool #5662

Closed

cosmicexplorer mentioned this pull request Jul 16, 2018

use ParseSearchDirs to find include dirs in the XCodeCLITools and LLVM subsystems #6143

Closed

cosmicexplorer mentioned this pull request Aug 9, 2018

remove env modifications from the setup.py invocation environment #6329

Closed

rtfeldman mentioned this pull request Oct 8, 2020

crti.o, crtn.o, and Scrt1.o should not use absolute paths roc-lang/roc#574

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943

Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943

cosmicexplorer commented Jun 8, 2018 •

edited

Loading

cosmicexplorer commented Jun 8, 2018

benjyw Jun 10, 2018

cosmicexplorer Jun 10, 2018

illicitonion left a comment

cosmicexplorer commented Jun 14, 2018

cosmicexplorer commented Jun 14, 2018

cosmicexplorer commented Jun 15, 2018 •

edited

Loading

cosmicexplorer commented Jun 15, 2018

stuhood commented Jun 15, 2018

cosmicexplorer commented Jun 28, 2018

stuhood left a comment

stuhood Jun 28, 2018

cosmicexplorer Jun 28, 2018

cosmicexplorer Jun 28, 2018

stuhood Jun 28, 2018

cosmicexplorer Jun 28, 2018 •

edited

Loading

stuhood Jun 28, 2018

cosmicexplorer Jun 28, 2018

stuhood Jun 28, 2018

cosmicexplorer Jun 28, 2018

stuhood Jun 28, 2018

cosmicexplorer Jun 28, 2018

stuhood Jun 28, 2018 •

edited

Loading

cosmicexplorer Jun 28, 2018

cosmicexplorer Jun 28, 2018

stuhood commented Jul 2, 2018

cosmicexplorer commented Jul 2, 2018

Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943

Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943

Conversation

cosmicexplorer commented Jun 8, 2018 • edited Loading

Problem

Solution

Result

cosmicexplorer commented Jun 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illicitonion left a comment

Choose a reason for hiding this comment

cosmicexplorer commented Jun 14, 2018

cosmicexplorer commented Jun 14, 2018

cosmicexplorer commented Jun 15, 2018 • edited Loading

cosmicexplorer commented Jun 15, 2018

stuhood commented Jun 15, 2018

cosmicexplorer commented Jun 28, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer Jun 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jun 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood commented Jul 2, 2018

cosmicexplorer commented Jul 2, 2018

cosmicexplorer commented Jun 8, 2018 •

edited

Loading

cosmicexplorer commented Jun 15, 2018 •

edited

Loading

cosmicexplorer Jun 28, 2018 •

edited

Loading

stuhood Jun 28, 2018 •

edited

Loading