Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use liblzma.dylib for xz on osx and add platform-specific testing to the rust osx shard #5936

Merged
merged 16 commits into from
Jun 13, 2018

Conversation

cosmicexplorer
Copy link
Contributor

@cosmicexplorer cosmicexplorer commented Jun 8, 2018

Problem

See #5928. The xz archiver wasn't tested on osx at all, and failed to find liblzma.so on osx (it should have been liblzma.dylib). There were additional errors with library search paths reported in that PR which I was not immediately able to repro. This PR hopefully fixes all of those errors by statically linking xz (from pantsbuild/binaries#74), and ensures they won't happen again with the addition of platform-specific testing (see previous issue at #5920).

Solution

  • Bump the default_version of the XZ BinaryTool to 5.4.2-3 to use a statically-linked executable as per statically link xz binaries#74.
  • Fix the incorrect key 'darwin' in the platform dictionary in the LLVM subsystem (to 'mac').
  • Add the tag platform_specific_behavior to the new python target tests/python/pants_test/backend/python/tasks:python_native_code_testing, which covers the production of python_dist()s with native code.
  • Add the -z argument to build-support/bin/ci.sh to run all tests with the platform_specific_behavior tag. Also clean up old unused options in the getopts call, and convert echo statements to a simpler heredoc.
  • Change the name of the "Rust Tests OSX" shard to "Rust + Platform-specific Tests OSX", and add the -z switch to the ci.sh invocation.

Note: the tests in tests/python/pants_test/backend/native/subsystems are going to be removed in #5815, otherwise they would be tagged similarly.

Result

./pants test tests/python/pants_test/backend/python/tasks:python_native_code_testing now passes on osx, and this fact is now being tested in an osx shard in travis. XZ now works on OSX.

Copy link
Contributor

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together, looks good!

It fixes up the two python problems for me, but I'm still seeing the

tests/python/pants_test/backend/python/tasks/test_build_local_python_distributions.py dyld: Library not loaded: /Users/stuhood/src/pantsbuild-binaries/build-support/bin/xz/mac/10.13/5.2.4/xz-5.2.4-osx/xz-install/lib/liblzma.5.dylib
                       Referenced from: /Users/dwagnerhall/.cache/pants/bin/xz/mac/10.13/5.2.4/xz/bin/xz
                       Reason: Incompatible library version: xz requires version 8.0.0 or later, but liblzma.5.dylib provides version 6.0.0

error on my machine...

@@ -116,15 +130,15 @@ def __init__(self, xz_binary_path, xz_library_path):

if not os.path.isdir(xz_library_path):
raise self.XZArchiverError(
"The path {} does not name an existing directory. A directory containing liblzma.so must "
"The path {} does not name an existing directory. A directory containing liblzma.{so,dylib} must "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's format in self. _liblzma_shared_lib_filename in the error, for clarity (and below)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@@ -17,6 +17,28 @@ python_library(
]
)

python_tests(
name='python_native_code_testing',
sources=[
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be excluded from the tasks target; we don't want to run them twice on linux

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done by adding the python variable python_native_code_test_files to use as sources in the platform-specific test target, and excludes in the generic unit and integration testing targets in that directory. Let me know if there is a better BUILD file pattern I could be using.

@illicitonion
Copy link
Contributor

Also, the new CI shard is failing with:
/build-support/bin/ci.sh: line 235: ./pants.pex: No such file or directory

@cosmicexplorer
Copy link
Contributor Author

The ci.sh failure should be fixed with 0b493d5, and the platform-specific test sources have been excluded from the other test targets in 633a2b6 and a1617a9. Modulo the rest of the ci, I'm going to dig into why you might be seeing that dyld error @illicitonion.

@cosmicexplorer
Copy link
Contributor Author

cosmicexplorer commented Jun 9, 2018

@illicitonion in 34bd769 I set DYLD_FALLBACK_LIBRARY_PATH as per your issue #5928. While I never experienced the failure you did, I did see this on my osx laptop:

> ~/.cache/pants/bin/xz/mac/10.13/5.2.4/xz/bin/xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.3
> DYLD_FALLBACK_LIBRARY_PATH="${HOME}/.cache/pants/bin/xz/mac/10.13/5.2.4/xz/lib" ~/.cache/pants/bin/xz/mac/10.13/5.2.4/xz/bin/xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Note the correct version 5.2.4 of liblzma when setting DYLD_FALLBACK_LIBRARY_PATH. This leads me to believe that this may resolve your issue (which I suspect is that the xz on your PATH has some incompatible version with the one we provide). Please let me know if this commit allows you to successfully run the platform-specific tests.

Copy link
Contributor

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works! Thanks! :)

# Only allow our xz's lib directory to resolve the liblzma.{so,dylib} dependency at runtime.
normalized_os_name = get_normalized_os_name()
if normalized_os_name == 'darwin':
env['DYLD_FALLBACK_LIBRARY_PATH'] = self._xz_library_path
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should probably actually be:
env['DYLD_LIBRARY_PATH'] = '{}:{}'.format(self.._xz_library_path, env.get('DYLD_LIBRARY_PATH', ''))

Two important differences:

  1. Set the main path not the fallback
  2. Preserve existing values; they may be important

(The one below should probably also set the existing path)

But I think it would be better for us to make sure the xz we release to pantsbinaries runs out of the box (either by linking differently, or with a wrapper script), rather than patching this up in pants...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is a very useful method get_joined_path() I added to contextutil.py to do the path operations you describe. However, I believe pantsbuild/binaries#71 lets us remove these lines. I will make the changes here and wait for that PR to be merged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a wrapper script to the xz released to pants-binaries in pantsbuild/binaries#72! Thanks, that was an very useful suggestion.

stuhood pushed a commit to pantsbuild/binaries that referenced this pull request Jun 12, 2018
This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.
Copy link
Contributor

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

# bug? If performance is an issue, investigate further.
# FIXME: --threads=0 is supposed to use "the number of processor cores on the machine", but I
# see no more than 100% cpu used at any point. This seems like it could be a bug? If performance
# is an issue, investigate further.
cmd = [xz_filename, '--decompress', '--stdout', '--keep', '--threads=0', xz_input_file]
env = {
# Isolate the path so we know we're using our provided version of xz.
'PATH': xz_bin_dir,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than setting $PATH here, just using self._xz_binary_path as the first arg of cmd should do the job, right?

Then we can either use the system env, as pants tends to, or an empty env, as I would be very happy about

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! The env bit was only necessary to emulate the kind of wrapping we are now doing in pantsbuild/binaries#72. Were you thinking I might want to explicitly clear the env here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, explicitly clearing the env in general is nice because it means we're more consistent/reproducible.

It probably doesn't matter either way here, but for things on the build path which we cache (possibly even outside of the local machine) things (like invoking gcc), stripping the entire env, and adding whatever we need explicitly, is a nice safe approach to try to ensure reproducibility. It also means that we can think to explicitly add those env vars to any cache keys we may need to (or, with v2 process execution, this happens by default, so it ensures that we actually get cache hits across users, because otherwise things like $HOME will always be different! :))

stuhood pushed a commit to pantsbuild/binaries that referenced this pull request Jun 12, 2018
See #71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.
@stuhood
Copy link
Member

stuhood commented Jun 13, 2018

@cosmicexplorer : I'm not super familiar with the xz situation, but... should the CLI tool be statically linked?

@cosmicexplorer
Copy link
Contributor Author

Yes! I will do that now.

@illicitonion
Copy link
Contributor

Looks great :) Thanks!

stuhood pushed a commit to pantsbuild/binaries that referenced this pull request Jun 13, 2018
See #71, #72, #73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```
@stuhood
Copy link
Member

stuhood commented Jun 13, 2018

Restarted travis after landing #74.

@stuhood stuhood merged commit 27714fe into pantsbuild:master Jun 13, 2018
CMLivingston pushed a commit to CMLivingston/pants that referenced this pull request Jun 22, 2018
Closes pantsbuild#5831
Prep for release 1.8.0dev3 (pantsbuild#5937)

Ban bad `readonly` shell pattern (pantsbuild#5924)

This subverts `set -e` and means that failed commands don't fail the
script, which is responsible for late failures on CI such as
https://travis-ci.org/pantsbuild/pants/jobs/389174049 which failed to
download protoc, but only failed when something tried to use it.
Fixup macosx platform version. (pantsbuild#5938)

This needs to match the Travis CI osx platform we pre-build wheels on.

Work towards pantsbuild#4896.
Allow pants to select targets by file(s) (pantsbuild#5930)

Fixes pantsbuild#5912

Problem
There should be an option to accept literal files, and then select the targets that own those files. Similar to how the --changed-parent triggers a diff and the targets are selected based on the result.
The proposed syntax is something like:

$ ./pants \
  --owner-of=this/is/a/file/name.java \    # flag triggers owner lookup
  --owner-of=this/is/a/file/name/too.py \  # flag triggers owner lookup
  compile                                  # goal
Solution
I've created a global option --owner-of= that takes a list of files as a parameter, and created a OwnerCalculator class to handle the logic similar to how ChangeCalculator works with the --changed-* subsystem.

Result
Now users will be able to run goals on files without needing to know which target owns those files.
Also, the list-owners goal can be deprecated in favor of --owner-of=some/file.py list

It is important to note that multiple target selection methods are not allowed, so it fails when more than one of --changed-*, --owner-of, or target specs are supplied.
e.g. this fails:

$ ./pants \
  --owner-of=this/is/a/file/name.java \    # flag triggers owner lookup
  --owner-of=this/is/a/file/name/too.py \  # flag triggers owner lookup
  compile
  <another target>
Integration test for daemon environment scrubbing (pantsbuild#5893)

This shows that pantsbuild#5898 works, which itself fixed pantsbuild#5854
Add the --owner-of= usage on Target Address documentation (pantsbuild#5931)

Problem
The documentation of the feature proposed in PR pantsbuild#5930

Solution
I decided to put it inside Target Addresses because that is where a user would look if they needed a feature like this, I think.

Result
More docs, and that's always good.
Add script to get a list of failing pants from travis (pantsbuild#5946)

[jvm-compile] template-methodify JvmCompile further; add compiler choices (pantsbuild#5923)

Introduce `JvmPlatform.add_compiler_choice(name)`, which allows plugins to register compilers that can be configured.
This patch pulls out some template methods in JvmCompile to make it easier to extend. It also pushes some of the implementations of those methods down into ZincCompile, where appropriate.

These changes should be covered by existing tests, but it could make sense to add tests around the interfaces of the new template methods. I don't anticipate there being a large number of implementations at this time though, so I didn't think it'd be worth it.

Add the following template methods

* `create_empty_extra_products` Allows subclasses to create extra products that other subclasses might not need, that ought to be constructed even if no compile targets are necessary.

* `register_extra_products_from_contexts` rename of `_register_vts`. This allows subclasses to register their extra products for particular targets.
* `select_runtime_context` Not 100% happy with this, but I'm working on something that needs to have different types of compile contexts. It allows subclasses to specify a context that provides paths for the runtime classpath if the default context isn't quite right for the usages in the base class.
* `create_compile_jobs` Pulled this out into a separate method so that subclasses can create multiple graph jobs per target.

* Pushed down behavior from JvmCompile that should live in zinc via the template methods extracted above. There's probably more that could be done here, but this was the first cut of it.
* Moved the execute definition from BaseZincCompile to ZincCompile so that it's possible to subclass BaseZincCompile with a different compiler name.
release notes for 1.7.0.rc1 (pantsbuild#5942)

Use target not make_target in some tests (pantsbuild#5939)

This pushes parsing of the targets through the engine, rather than
bypassing it.

This is important because I'm about to make these targets require an
EagerFilesetWithSpec as their source/sources arg, rather than being
happy with a list of strings.
Add new remote execution options (pantsbuild#5932)

As described in pantsbuild#5904, a few configuration values that are important to testing of remote execution are currently hardcoded.

Extract existing options to a `ExecutionOptions` collection (which should become a `Subsystem` whenever we add support for consuming `Subsystems` during bootstrap), and add the new options.

Fixes pantsbuild#5904.
Separate the resolution cache and repository cache in Ivy (pantsbuild#5844)

move glob matching into its own file (pantsbuild#5945)

See pantsbuild#5871, where we describe an encapsulation leak created by implementing all of the glob expansion logic in the body of `VFS`.

- Create `glob_matching.rs`, exporting the `GlobMatching` trait, which exports the two methods `canonicalize` and `expand`, which call into methods in a private trait `GlobMatchingImplementation`.

**Note:** `canonicalize` calls `expand`, and vice versa, which is why both methods were moved to `glob_matching.rs`.

Orthogonal glob matching logic is made into a trait that is implemented for all types implementing `VFS`, removing the encapsulation leak. The `VFS` trait is now just four method signature declarations, making the trait much easier to read and understand.
Enable fromfile support for --owner-of and increase test coverage (pantsbuild#5948)

The new `--owner-of` option was broken in the context of `pantsd`, but didn't have test coverage due to the `--changed` and `--owner-of` tests not running under `pantsd`. Additionally, `fromfile` support is useful for this option, but was not enabled.

Mark some integration tests as needing to run under the daemon, and enable `fromfile` support for `--owner-of`.
[pantsd] Robustify client connection logic. (pantsbuild#5952)

Fixes pantsbuild#5812

under full-on-assault stress testing via:

$ watch -n.1 'pkill -f "pantsd \[" pantsd-runner'
this will mostly behave like:

WARN] pantsd was unresponsive on port 55620, retrying (1/3)
WARN] pantsd was unresponsive on port 55620, retrying (2/3)
WARN] pantsd was unresponsive on port 55626, retrying (3/3)
WARN] caught client exception: Fallback(NailgunExecutionError(u'Problem executing command on nailgun server (address: 127.0.0.1:55630): TruncatedHeaderError(u"Failed to read nailgun chunk header (TruncatedRead(u\'Expected 5 bytes before socket shutdown, instead received 0\',)).",)',),), falling back to non-daemon mode

23:30:24 00:00 [main]
               (To run a reporting server: ./pants server)
23:30:38 00:14   [setup]
23:30:39 00:15     [parse]
...
mid-flight terminations (simulated via single-shot pkill calls) also result in a more descriptive error with traceback proxying:

23:40:51 00:04     [zinc]
23:40:51 00:04     [javac]
23:40:51 00:04     [cpp]
23:40:51 00:04     [errorprone]
23:40:51 00:04     [findbugs]CRITICAL]
CRITICAL] lost active connection to pantsd!
Exception caught: (<class 'pants.bin.remote_pants_runner.Terminated'>)
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 73, in <module>
    main()
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 69, in main
    PantsLoader.run()
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 65, in run
    cls.load_and_execute(entrypoint)
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 58, in load_and_execute
    entrypoint_main()
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_runner.py", line 39, in run
    return RemotePantsRunner(self._exiter, self._args, self._env, bootstrap_options).run()
  File "/Users/kwilson/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 162, in run
    self._run_pants_with_retry(port)
  File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 221, in execute
    return self._session.execute(cwd, main_class, *args, **environment)
  File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 94, in execute
    return self._process_session()
  File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 69, in _process_session
    for chunk_type, payload in self.iter_chunks(self._sock, return_bytes=True):
  File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_protocol.py", line 206, in iter_chunks
    chunk_type, payload = cls.read_chunk(sock, return_bytes)
  File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_protocol.py", line 182, in read_chunk
    raise cls.TruncatedHeaderError('Failed to read nailgun chunk header ({!r}).'.format(e))

Exception message: abruptly lost active connection to pantsd runner: NailgunError(u'Problem talking to nailgun server (address: 127.0.0.1:55707, remote_pid: -28972): TruncatedHeaderError(u"Failed to read nailgun chunk header (TruncatedRead(u\'Expected 5 bytes before socket shutdown, instead received 0\',)).",)',)
Re-shade zinc to avoid classpath collisions with annotation processors. (pantsbuild#5953)

Zinc used to be shaded before the `1.x.y` upgrade (pantsbuild#4729), but shading was removed due to an overabundance of optimism. When testing the zinc upgrade internally, we experienced a classpath collision between an annotation processor and zinc (in guava, although zinc has many other dependencies that could cause issues).

Shade zinc, and ensure that our annotation processor uses a very old guava in order to attempt to force collisions in future.
Improve PythonInterpreterCache logging (pantsbuild#5954)

When users have issues building their Python interpreter cache, they are often very confused because does not currently log much about the process to help users debug. Here we add log lines describing what/where Pants looks to build the interpreter cache, and the results of what it found. This should help users better understand/debug the process.
use liblzma.dylib for xz on osx and add platform-specific testing to the rust osx shard (pantsbuild#5936)

See pantsbuild#5928. The `xz` archiver wasn't tested on osx at all, and failed to find `liblzma.so` on osx (it should have been `liblzma.dylib`). There were additional errors with library search paths reported in that PR which I was not immediately able to repro. This PR hopefully fixes all of those errors, and ensures they won't happen again with the addition of platform-specific testing (see previous issue at pantsbuild#5920).

- Switch to a statically linked `xz`.
- Fix the incorrect key `'darwin'` in the platform dictionary in the `LLVM` subsystem (to `'mac'`).
- Add the tag `platform_specific_behavior` to the new python target `tests/python/pants_test/backend/python/tasks:python_native_code_testing`, which covers the production of `python_dist()`s with native code.
- Add the `-z` argument to `build-support/bin/ci.sh` to run all tests with the `platform_specific_behavior` tag. Also clean up old unused options in the getopts call, and convert echo statements to a simpler heredoc.
- Change the name of the "Rust Tests OSX" shard to "Rust + Platform-specific Tests OSX", and add the `-z` switch to the `ci.sh` invocation.

**Note:** the tests in `tests/python/pants_test/backend/native/subsystems` are going to be removed in pantsbuild#5815, otherwise they would be tagged similarly.

`./pants test tests/python/pants_test/backend/python/tasks:python_native_code_testing` now passes on osx, and this fact is now being tested in an osx shard in travis.
Support output directory saving for local process execution. (pantsbuild#5944)

Closes pantsbuild#5860
reimplement a previous PR -- ignore this

This commit is a reimplementation of registering @rules for backends, because this PR began before
that one was split off.

add some simple examples to demonstrate how to use backend rules

...actually make the changes to consume backend rules in register.py

revert accidental change to a test target

remove extraneous log statement

fix lint errors

add native backend to release.sh

isolate native toolchain path and hope for the best

add config subdir to native backend

really just some more attempts

start trying to dispatch based on platform

extend EngineInitializer to add more rules from a backend

refactor Platform to use new methods in osutil.py

refactor the native backend to be a real backend and expose rules

register a rule in the python backend to get a setup.py environment

make python_dist() tests pass

make lint pass

create tasks and targets for c/c++ sources

- refactors the "native toolchain" and introduces the "binaries" subdirectory of subsystems

start by adding a new ctypes testproject

add c/c++ sources

add example BUILD file

add some native targets

add tasks dir

remove gcc

try to start adding tasks

clean some leftover notes in BuildLocalPythonDistributions

update NativeLibrary with headers

move DependencyContext to target.py

add native compile tasks

houston we have compilation

run:

./pants -ldebug --print-exception-stacktrace compile testprojects/src/python/python_distribution/ctypes:

for an idea

use the forbidden product request

change target names to avoid conflict with cpp contrib and flesh out cpp_compile

now we are compiling code

we can link things now

now we know how to infer headers vs sources

fix the test case and fix include dir collection

(un)?suprisingly, everything works, but bdist_wheel doesn't read MANIFEST.in

houston we have c++

bring back gcc so we can compile

halfway done with osx support

now things work on osx????????

ok, now it works on linux again too

first round of review

- NB: refactors the organization of the `results_dir` for python_dist()!
- move ctypes integration testing into python backend tests

revert some unnecessary changes

refactor native_artifact to be a datatype

fix some copyright years

add ctypes integration test

add assert_single_element method in collections.py

decouple the native tools for setup.py from the execution environment

streamline the selection of the native tools for setup.py invocation

make gcc depend on binutils on linux for the 'as' assembler

fix logging visibility by moving it back into the task

make the check for the external llvm url more clear

refactor local dist building a bit

- use SetupPyRunner.DIST_DIR as the source of truth
- add a separate subdir of the python_dist target's results_dir for the
  python_dist sources
- move shraed libs into the new subdir

fix imports

second round of review

- fixed bugs
- expanded error messages and docstrings

make a couple docstring changes

fix dist platform selection ('current' is not a real platform)

lint fixes

fix broken regex which modifies the `.dylib` extension for python_dist()

fix the ctypes integration test

respond to some review comments

clear the error message if we can't find xcode cli tools

refactor the compilation and linking pipeline to use subsystems

- also add `PythonNativeCode` subsystem to bridge the native and python backends

refactor the compilation and linking pipeline to use subsystems

add some notes

fix rebase issues

add link to pantsbuild#5788 -- maybe use variants for args for static libs

move `native_source_extensions` to a new `PythonNativeCode` subsystem

update native toolchain docs and remove bad old tests

move tgt_closure_platforms into the new `PythonNativeCode` subsystem

remove unnecessary logging

remove compile_settings_class in favor of another abstractmethod

refactor `NativeCompile` and add documentation

improve debug logging in NativeCompile

document NativeCompileSettings

refactor and add docstrings

convert provides= to ctypes_dylib= and add many more docstrings

remove or improve TODOs

improve or remove FIXMEs

improve some docstrings, demote a FIXME, and add a TODO

link FIXMEs to a ticket

add notes to the ctypes testproject

update mock object for strict deps -- test passes

fix failing integration test on osx

add hack to let travis pass

fix the system_id key in llvm and add a shameful hack to pass travis

swap the order of alias_types

remove unused EmptyDepContext class

remove -settings suffix from compile subsystem options scopes

add AbstractClass to NativeLibrary

bump implementation_version for python_dist() build

- we have changed the layout of the results_dir in this PR

add ticket link and fix bug

revert indentation changes to execute() method

refactor `assert_single_element()`

revert addition of `narrow_relative_paths()`

add link to pantsbuild#5950

move common process invocation logic into NativeCompile

revert an unnecessary change

turn an assert into a full exception

revert unnecessary change

use get_local_platform() wherever possible

delete duplicate llvm subsystem

fix xcode cli tools resolution

change `ctypes_dylib=` to `ctypes_native_library=`

add a newline

move UnsupportedPlatformError to be a class field

remove unused codegen_types field

fix zinc-compiler options to be valid ones

Construct rule_graph recursively (pantsbuild#5955)

The `RuleGraph` is currently constructed iteratively, but can be more-easily constructed recursively.

Switch to constructing the `RuleGraph` recursively, and unify a few disparate diagnostic messages.

Helps to prepare for further refactoring in pantsbuild#5788.
Allow manylinux wheels when resolving plugins. (pantsbuild#5959)

Also plumb manylinux resolution support for the python backend, on by
default, but configurable via `python_setup.resolver_use_manylinux`.

Fixes pantsbuild#5958
`exclude-patterns` and `tag` should apply only to roots (pantsbuild#5786)

The `--exclude-patterns` flag currently applies to inner nodes, which causes odd errors. Moreover, tags should also apply only to roots. See pantsbuild#5189.

- added `tag` & `exclude_patterns` as params to `Specs`
- add tests for both
- modify changed tests to pass for inner node filtering

Fixes pantsbuild#5189.
Remove value wrapper on the python side of ffi. (pantsbuild#5961)

As explained in the comment in this change, the overhead of wrapping our CFFI "handle"/`void*` instances in a type that is shaped like the `Value` struct was significant enough to care about.

Since the struct has zero overhead on the rust side, whether we represent it as typedef or a struct on the python side doesn't make a difference (except in terms of syntax).

6% faster `./pants list ::` in Twitter's repo.
return an actual field

use temporary native-compile goal

Cobertura coverage: Include the full target closure's classpath entries for instrumentation (pantsbuild#5879)

Sometimes Cobertura needs access to the dependencies of class files being instrumented in order to rewrite them (pantsbuild#5878).

This patch adds an option that creates a manifest jar and adds an argument to the Cobertura call so that it can take advantage of it.

class files that need to determine a least upper bound in order to be rewritten can now be instrumented.

Fixes  pantsbuild#5878
add ticket link

fix xcode install locations and reduce it to a single dir_option

return the correct amount of path entries

Record start times per graph node and expose a method to summarize them. (pantsbuild#5964)

In order to display "significant" work while it is running on the engine, we need to compute interesting, long-running leaves.

Record start times per entry in the `Graph`, and add a method to compute a graph-aware top-k longest running leaves. We traverse the graph "longest running first", and emit the first `k` leaves we encounter.

While this will almost certainly need further edits to maximize it's usefulness, visualization can begun to be built atop of it.
Prepare the 1.8.0.dev4 release (pantsbuild#5969)

Mark a few options that should not show up in `./pants help`. (pantsbuild#5968)

`./pants help` contains core options that are useful to every pants command, and the vast majority of global options are hidden in order to keep it concise. A few non-essential options ended up there recently.

Hide them.
adding more documentation for python_app (pantsbuild#5965)

The python_app target doesn't have the documentation specific for it and has a documentation that is specific to jvm_app.

Added a few lines of documentation.

There is no system-wide change, only a documentation change.
Remove DeprecatedPythonTaskTestBase (pantsbuild#5973)

Use PythonTaskTestBase instead.

Fixes pantsbuild#5870
Chris first commit on fresh rebase

Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party

unrevert reverted fix (NEEDS FOLLOWUP ISSUE!)

put in a better fix for the strict_deps error until the followup issue is made

add ticket link

Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party

Shorten safe filenames further, and combine codepaths to make them readable. (pantsbuild#5971)

Lower the `safe_filename` path component length limit to 100 characters, since the previous 255 value did not account for the fact that many filesystems also have a limit on total path length. This "fixes" the issue described in pantsbuild#5587, which was caused by using this method via `build_invalidator.py`.

Additionally, merge the codepath from `Target.compute_target_id` which computes a readable shortened filename into `safe_filename`, and expand tests. This removes some duplication, and ensure that we don't run into a similar issue with target ids.

The specific error from pantsbuild#5587 should be prevented, and consumers of `safe_filename` should have safe and readable filenames.

Fixes pantsbuild#5587.
Whitelist the --owner-of option to not restart the daemon. (pantsbuild#5979)

Because the `--owner-of` option was not whitelisted as `daemon=False`, changing its value triggered unnecessary `pantsd` restarts.

Whitelist it.
Prepare the 1.8.0rc0 release. (pantsbuild#5980)

Robustify test_namespace_effective PYTHONPATH. (pantsbuild#5976)

The real problem is noted, but this quick fix should bolster against
interpreter cache interpreters pointing off to python binaries that
have no setuptools in their associated site-packages.

Fixes pantsbuild#5972
make_target upgrades sources to EagerFilesetWithSpec (pantsbuild#5974)

This better simulates how the engine parses BUILD files, giving a more
faithful experience in tests.

I'm about to make it a warning/error to pass a list of strings as the
sources arg, so this will make tests which use make_target continue to
work after that.

Also, make cloc use base class scheduler instead of configuring its own.
Lib and include as a dep-specifc location

source attribute is automatically promoted to sources (pantsbuild#5908)

This means that either the `source` or `sources` attribute can be used
for any rule which expects sources. Places that `source` was expected
still verify that the correct number of sources are actually present.
Places that `sources` was expected will automatically promote `source`
to `sources`.

This is a step towards all `sources` attributes being
`EagerFilesetWithSpec`s, which will make them cached in the daemon, and
make them easier to work with with both v2 remote execution and in the
v2 engine in general. It also provides common hooks for input file
validation, rather than relying on them being done ad-hoc in each
`Target` constructor.

For backwards compatibility, both attributes will be populated on
`Target`s, but in the future only the sources value will be provided.

`sources` is guaranteed to be an `EagerFilesetWithSpec` whichever of
these mechanisms is used.

A hook is provided for rules to perform validation on `sources` at build
file parsing time. Hooks are put in place for non-contrib rule types
which currently take a `source` attribute to verify that the correct
number of sources are provided. I imagine at some point we may want to
add a "file type" hook too, so that rules can error if files of the
wrong type were added as sources.

This is a breaking change for rules which use both the `source` and `sources` attributes (and where the latter is not equivalent to the former), or where the `source` attribute is used to refer to something other than a file. `source` is now becoming a
reserved attribute name, as `sources` and `dependencies` already are.

This is also a breaking change for rules which use the `source`
attribute, but never set `sources` in a Payload. These will now fail to
parse.

This is also a slightly breaking change for the `page` rule - before,
omitting the `source` attribute would parse, but fail at runtime. Now,
it will fail to parse.

This is also a breaking change in that in means that the source
attribute is now treated like a glob, and so if a file is specified
which isn't present, it will be ignored instead of error. This feels a
little sketchy, but it turns out we did the exact same thing by making
all sources lists be treated like globs...
Override get_sources for pants plugins (pantsbuild#5984)

1.7.0 release notes (pantsbuild#5983)

No additional changes, so it's a very short release note.
Fixups for native third party work

hardcode in c/c++ language levels for now

remove all the unnecessary code relating to file extensions

Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party

Caching tests are parsed through the engine (pantsbuild#5985)

reimplement a previous PR -- ignore this

This commit is a reimplementation of registering @rules for backends, because this PR began before
that one was split off.

add some simple examples to demonstrate how to use backend rules

...actually make the changes to consume backend rules in register.py

revert accidental change to a test target

remove extraneous log statement

fix lint errors

add native backend to release.sh

isolate native toolchain path and hope for the best

add config subdir to native backend

really just some more attempts

start trying to dispatch based on platform

extend EngineInitializer to add more rules from a backend

refactor Platform to use new methods in osutil.py

refactor the native backend to be a real backend and expose rules

register a rule in the python backend to get a setup.py environment

make python_dist() tests pass

make lint pass

create tasks and targets for c/c++ sources

- refactors the "native toolchain" and introduces the "binaries" subdirectory of subsystems

start by adding a new ctypes testproject

add c/c++ sources

add example BUILD file

add some native targets

add tasks dir

remove gcc

try to start adding tasks

clean some leftover notes in BuildLocalPythonDistributions

update NativeLibrary with headers

move DependencyContext to target.py

add native compile tasks

houston we have compilation

run:

./pants -ldebug --print-exception-stacktrace compile testprojects/src/python/python_distribution/ctypes:

for an idea

use the forbidden product request

change target names to avoid conflict with cpp contrib and flesh out cpp_compile

now we are compiling code

we can link things now

now we know how to infer headers vs sources

fix the test case and fix include dir collection

(un)?suprisingly, everything works, but bdist_wheel doesn't read MANIFEST.in

houston we have c++

bring back gcc so we can compile

halfway done with osx support

now things work on osx????????

ok, now it works on linux again too

first round of review

- NB: refactors the organization of the `results_dir` for python_dist()!
- move ctypes integration testing into python backend tests

revert some unnecessary changes

refactor native_artifact to be a datatype

fix some copyright years

add ctypes integration test

add assert_single_element method in collections.py

decouple the native tools for setup.py from the execution environment

streamline the selection of the native tools for setup.py invocation

make gcc depend on binutils on linux for the 'as' assembler

fix logging visibility by moving it back into the task

make the check for the external llvm url more clear

refactor local dist building a bit

- use SetupPyRunner.DIST_DIR as the source of truth
- add a separate subdir of the python_dist target's results_dir for the
  python_dist sources
- move shraed libs into the new subdir

fix imports

second round of review

- fixed bugs
- expanded error messages and docstrings

make a couple docstring changes

fix dist platform selection ('current' is not a real platform)

lint fixes

fix broken regex which modifies the `.dylib` extension for python_dist()

fix the ctypes integration test

respond to some review comments

clear the error message if we can't find xcode cli tools

refactor the compilation and linking pipeline to use subsystems

- also add `PythonNativeCode` subsystem to bridge the native and python backends

refactor the compilation and linking pipeline to use subsystems

add some notes

fix rebase issues

add link to pantsbuild#5788 -- maybe use variants for args for static libs

move `native_source_extensions` to a new `PythonNativeCode` subsystem

update native toolchain docs and remove bad old tests

move tgt_closure_platforms into the new `PythonNativeCode` subsystem

remove unnecessary logging

remove compile_settings_class in favor of another abstractmethod

refactor `NativeCompile` and add documentation

improve debug logging in NativeCompile

document NativeCompileSettings

refactor and add docstrings

convert provides= to ctypes_dylib= and add many more docstrings

remove or improve TODOs

improve or remove FIXMEs

improve some docstrings, demote a FIXME, and add a TODO

link FIXMEs to a ticket

add notes to the ctypes testproject

update mock object for strict deps -- test passes

fix failing integration test on osx

add hack to let travis pass

fix the system_id key in llvm and add a shameful hack to pass travis

swap the order of alias_types

remove unused EmptyDepContext class

remove -settings suffix from compile subsystem options scopes

add AbstractClass to NativeLibrary

bump implementation_version for python_dist() build

- we have changed the layout of the results_dir in this PR

add ticket link and fix bug

revert indentation changes to execute() method

refactor `assert_single_element()`

revert addition of `narrow_relative_paths()`

add link to pantsbuild#5950

move common process invocation logic into NativeCompile

revert an unnecessary change

turn an assert into a full exception

revert unnecessary change

use get_local_platform() wherever possible

delete duplicate llvm subsystem

fix xcode cli tools resolution

change `ctypes_dylib=` to `ctypes_native_library=`

add a newline

move UnsupportedPlatformError to be a class field

remove unused codegen_types field

fix zinc-compiler options to be valid ones

return an actual field

use temporary native-compile goal

add ticket link

fix xcode install locations and reduce it to a single dir_option

return the correct amount of path entries

unrevert reverted fix (NEEDS FOLLOWUP ISSUE!)

put in a better fix for the strict_deps error until the followup issue is made

add ticket link

hardcode in c/c++ language levels for now

remove all the unnecessary code relating to file extensions

fix osx failures and leave a ticket link

Add rang header-only lib for integration testing

Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party

Fix TestSetupPyInterpreter.test_setuptools_version (pantsbuild#5988)

Previously the test failed to populate the `PythonInterpreter` data
product leading to a fallback to the current non-bare interpreter which
allowed `setuptools` from `site-packages` to leak in.

Fixes pantsbuild#5467
Refactor conan grab into subsystem

Engine looks up default sources when parsing (pantsbuild#5989)

Rather than re-implementing default source look-up.

This pushes sources parsing of default sources through the engine, in parallel,
rather than being later synchronous python calls.

This also works for plugin types, and doesn't change any existing APIs.

It updates the Go patterns to match those that the engine currently performs,
rather than ones which aren't actually used by any code.
Add unit tests, refactor

C/C++ targets which can be compiled/linked and used in python_dist() with ctypes (pantsbuild#5815)

It is currently possible to expose native code to Python by compiling it in a `python_dist()` target, specifying C or C++ source files as a `distutils.core.Extension` in the `setup.py` file, as well as in the target's sources. `python_dist()` was introduced in pantsbuild#5141. We introduced a "native toolchain" to compile native sources for this use case in pantsbuild#5490.

Exposing Python code this way requires using the Python native API and `#include <Python.h>` in your source files. However, python can also interact with native code that does not use the Python native API, using the provided `ctypes` library. For this to work, the `python_dist()` module using `ctypes` needs to have a platform-specific shared library provided within the package. This PR introduces the targets, tasks, and subsystems to compile and link a shared library from native code, then inserts it into the `python_dist()` where it is easily accessible.

- Introduce the `ctypes_compatible_c_library()` target which covers C sources (`ctypes_compatible_cpp_library()` for C++), and can specify what to name the shared library created from linking the object files compiled from its sources and dependencies.
- Introduce `CCompile`, `CppCompile`, and `LinkSharedLibraries` to produce the shared libraries from the native sources. The compile tasks use options `CCompileSettings` or `CppCompileSettings` to define file extensions for "header" and "source" files.
- Introduce the `CCompileSettings` and `CppCompileSettings` subsystems to control compile settings for those languages.
- Convert `BuildLocalPythonDistributions` to proxy to the native backend through the new `PythonNativeCode` subsystem.
- Move all the `BinaryTool` subsystems to a `subsystems/binaries` subdirectory, and expose them to the v2 engine through `@rule`s defined in the subsystem's file.
- Move some of the logic in `pex_build_util.py` to `setup_py.py`, and expose datatypes composing the setup.py environment through `@rule`s in `setup_py.py`. `SetupPyRunner.for_bdist_wheel()` was created to set the wheel's platform, if the `python_dist()` target contains any native sources of its own, or depends on any `ctypes_compatible_*_library`s.

**Note:** the new targets are specifically prefixed with `ctypes_compatible_` because we don't yet eclipse the functionality of `contrib/cpp`. When the targets become usable for more than this one use case, the name should be changed.

To see how to link up native and Python code with `ctypes`, here's (most of) the contents of `testprojects/src/python/python_distribution/ctypes`:
*BUILD*:
```python
ctypes_compatible_c_library(
  name='c_library',
  sources=['some_math.h', 'some_math.c', 'src-subdir/add_three.h', 'src-subdir/add_three.c'],
  ctypes_dylib=native_artifact(lib_name='asdf-c'),
)

ctypes_compatible_cpp_library(
  name='cpp_library',
  sources=['some_more_math.hpp', 'some_more_math.cpp'],
  ctypes_dylib=native_artifact(lib_name='asdf-cpp'),
)

python_dist(
  sources=[
    'setup.py',
    'ctypes_python_pkg/__init__.py',
    'ctypes_python_pkg/ctypes_wrapper.py',
  ],
  dependencies=[
    ':c_library',
    ':cpp_library',
  ],
)
```
*setup.py*:
```python
setup(
  name='ctypes_test',
  version='0.0.1',
  packages=find_packages(),
  # Declare two files at the top-level directory (denoted by '').
  data_files=[('', ['libasdf-c.so', 'libasdf-cpp.so'])],
)
```
*ctypes_python_pkg/ctypes_wrapper.py*:
```python
import ctypes
import os

def get_generated_shared_lib(lib_name):
  # These are the same filenames as in setup.py.
  filename = 'lib{}.so'.format(lib_name)
  # The data files are in the root directory, but we are in ctypes_python_pkg/.
  rel_path = os.path.join(os.path.dirname(__file__), '..', filename)
  return os.path.normpath(rel_path)

asdf_c_lib_path = get_generated_shared_lib('asdf-c')
asdf_cpp_lib_path = get_generated_shared_lib('asdf-cpp')

asdf_c_lib = ctypes.CDLL(asdf_c_lib_path)
asdf_cpp_lib = ctypes.CDLL(asdf_cpp_lib_path)

def f(x):
  added = asdf_c_lib.add_three(x)
  multiplied = asdf_cpp_lib.multiply_by_three(added)
  return multiplied
```

Now, the target `testprojects/src/python/python_distribution/ctypes` can be depended on in a BUILD file, and other python code can freely use `from ctypes_python_pkg.ctypes_wrapper import f` to start jumping into native code.

1. pantsbuild#5933
2. pantsbuild#5934
3. pantsbuild#5949
4. pantsbuild#5950
5. pantsbuild#5951
6. pantsbuild#5962
7. pantsbuild#5967
8. pantsbuild#5977
Add simple integration test

Pull in master

Minor cleanup

Fix CI errors

Debug log stdout

Fix integration test

Fix integration test

Fix lint error on third party
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants