-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943
Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests #5943
Conversation
This would completely resolve one of the FIXMEs currently left in #5815, which includes |
|
||
@memoized_method | ||
def glibc_dir(self): | ||
# FIXME: this can be removed once #5815 lands (GCC should be made to depend on GLibc on Linux). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not clear what the "this" is that can be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made it a lot more specific in c3aa525, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works for me :) Thanks!
bce6634
to
14b8a7d
Compare
Had a good discussion with @kwlzn on the slack today, and I now believe this should be replaced with a method of searching for |
It looks like we'll have to invoke the system The above is all only for Linux hosts. We'll have a good error message like XCodeCLITools if |
This PR should be ready for review again. Instead of providing glibc, we are finding the host's libc installation by parsing the output of |
According to the CI failures on the unit tests, we need to expose our provided gcc's |
will wait to review. |
2c23923
to
1335922
Compare
I have updated the OP and CI is green. Please take another look -- this was extremely difficult and time-intensive to get working (mostly because it's very difficult to reproduce the travis environment, and yes I really did try), but after this I am confident we will be able to iterate on the native backend (and the way we invoke compilers, linkers, etc) without fear, which seems like a useful result. The tools we have developed here are not specific to travis whatsoever and are generic enough to make a reliable basis for further extension. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
def register_options(cls, register): | ||
super(LibcDev, cls).register_options(register) | ||
|
||
register('--libc-dir', type=dir_option, default='/usr/lib', advanced=True, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should both of these options be fingerprinted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes!! Will fix and keep that in mind in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 5a1665b!
@@ -26,75 +24,160 @@ class XCodeCLITools(Subsystem): | |||
|
|||
options_scope = 'xcode-cli-tools' | |||
|
|||
_REQUIRED_TOOLS = frozenset(['cc', 'c++', 'clang', 'clang++', 'ld', 'lipo']) | |||
REQUIRED_FILES = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these intentionally public? Would be good to encapsulate instead if possible, to avoid needing to break APIs to add more data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, there's no reason for these to be public and the reason you mentioned is a fantastic reason to keep it private. Will fix.
register('--xcode-cli-install-location', type=dir_option, default='/usr/bin', advanced=True, | ||
help='Installation location for the XCode command-line developer tools.') | ||
register('--install-prefixes', type=list, default=cls.INSTALL_PREFIXES_DEFAULT, advanced=True, | ||
help='Locations to search for resources from the XCode CLI tools, including a ' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto fingerprint.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 5a1665b!
src/python/pants/engine/scheduler.py
Outdated
self._scheduler.add_root_selection(native_execution_request, subject, product) | ||
return ExecutionRequest(request_specs, native_execution_request) | ||
|
||
# FIXME: note that this does a cross prod! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You updated the comment here... can remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 5a1665b!
src/python/pants/engine/scheduler.py
Outdated
as "list of product". | ||
An ExecutionRequest for an Address represents exactly one product output, as does | ||
SingleAddress. But we differentiate between them here in order to normalize the output for all | ||
Spec objects as "list of product". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is true any longer... can nuke this paragraph probably.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 5a1665b!
@@ -406,15 +406,25 @@ def visualize_graph_to_file(self, filename): | |||
def visualize_rule_graph_to_file(self, filename): | |||
self._scheduler.visualize_rule_graph_to_file(filename) | |||
|
|||
def execution_request_literal(self, request_specs): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that rather than adding a new method, we'd want to just update execute_request
... it only has two callers outside of unit tests. But that would increase the scope here... if you're able to swap out the methods as a quick followup (not here, as this is already a bit large), that would be appreciated.
Also, the SchedulerSession.product_request
and product_requests
family is easier to use... so adding another method that is shaped like "list of tuples" would be ideal (and maybe even removing/deprecating ExecutionRequest as a public API).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That all sounds like a viable scope for a followup -- I'll make an issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made into #6044!
…m tests" This reverts commit 5a28312.
…is doomed enterprise
- add fingerprint=True to new subsystem options - make _REQUIRED_FILES private - add a useful comment - remove a now-false paragraph from the execution_request() docstring
6520d79
to
08ba46d
Compare
Feel free to merge when ready. |
Travis isn't showing the passing build in this thread for some reason, but 08ba46d has indeed passed CI, so merging now. |
Problem
The native backend subsystems tests introduced in #5490 are still skipped, complaining about
crti.o
on linux, which is part of libc. See #5662 --clang
's driver will find the directory containing that file on travis, butgcc
won't. We should make a way to find this file (which is necessary for creating executables) so we can unskip the native backend testing.Solution
os.path.isdir()
, so we never found theLLVM
BinaryTool when downloading it from the LLVM download page.LibcDev
subsystem. This uses the option--libc-dir
, if provided, or finds an installation of libc withcrti.o
by invoking--host-compiler
on the host once to get its search directories (this is necessary on travis, due to ubuntu's nonstandard installation location).src/python/pants/backend/native/config/environment.py
to cover everything needed to actually compile, and give them the ability to generate an environment suitable for passing intosubprocess.Popen()
.GCCCCompiler
,GCCCppCompiler
,LLVMCCompiler
, andLLVMCppCompiler
to differentiate between the two different compilers we have available for each language.native_toolchain.py
.test_native_toolchain.py
from combine Clang, GCC, Binutils, and XCodeCLITools to form the NativeToolchain subsystem and consume it in BuildLocalPythonDistributions for native code #5490.CCompiler
andCppCompiler
into clang, for no particular reason (it will pass CI with gcc as well).The different compilers we can use will likely be denoted using variants in the future, but this solution allows us to separate the resources generated from each subsystem (
GCC
,LLVM
,Binutils
,XCodeCLITools
) from a fully-functioning compiler that can be invoked (because actually invoking either clang or gcc requires some resources from the other, e.g. headers and libraries). Now, each binary tool only does the job of wrapping the resources it contains, whilenative_toolchain.py
does the job of creating either a clang or a gcc compiler that we can invoke on the current host (with all necessary headers, libs, etc).Result
The native toolchain tests from #5490 can be unskipped, and we can invoke our toolchain on almost any linux installation without further setup. The tests in
test_native_toolchain.py
now pass in CI, and as we iterate on the native backend, we will continue to have end-to-end testing for both of our compilers, on osx as well, regardless of whichever one we choose to use forpython_dist()
compilation.