Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port build graph #10

Closed
wants to merge 39 commits into from

Commits on Jul 15, 2018

  1. Port build_graph tests

    Eric Arellano committed Jul 15, 2018
    Configuration menu
    Copy the full SHA
    9407682 View commit details
    Browse the repository at this point in the history
  2. Port build_graph

    Eric Arellano committed Jul 15, 2018
    Configuration menu
    Copy the full SHA
    37c846d View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2018

  1. Eliminate obsolete PANTS_ARGS from ci. (pantsbuild#6141)

    This only served to set interpreter constraints we already express in
    pants.ini, and at a higher precedence, making attempted overrides via
    environment variable silently fail.
    jsirois authored Jul 16, 2018
    Configuration menu
    Copy the full SHA
    886a481 View commit details
    Browse the repository at this point in the history
  2. Port core tasks to Python 3 (pantsbuild#6130)

    Eric-Arellano authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    5920dc0 View commit details
    Browse the repository at this point in the history
  3. Port backend graph info to Python 3 (pantsbuild#6131)

    Eric-Arellano authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    ddac26f View commit details
    Browse the repository at this point in the history
  4. Port backend project info to Python 3 (pantsbuild#6132)

    Eric-Arellano authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    117c614 View commit details
    Browse the repository at this point in the history
  5. Add missing future dependency to BUILD (pantsbuild#6135)

    Fixing issue from the port of buildgen of missing dependency in BUILD file from pantsbuild#6110.
    
    While technically everything runs correctly without this declaration, that is an implementation detail that should not be relied upon.
    Eric-Arellano authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    fdf5c10 View commit details
    Browse the repository at this point in the history
  6. Port findbugs to py3 (pantsbuild#6137)

    ### Problem
    
    update for py3 compatiibility
    
    ### Solution
    
    Just need future and builtins import.
    GoingTharn authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    c401137 View commit details
    Browse the repository at this point in the history
  7. Port goal package to Python 3 (pantsbuild#6138)

    Eric-Arellano authored and Stu Hood committed Jul 16, 2018
    Configuration menu
    Copy the full SHA
    c62ae87 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2018

  1. Port to mypy to py3 (pantsbuild#6140)

    just needs builtins imports
    
    ### Problem
    
    Port mypy to py3
    just imports at issue
    
    ### Solution
    Futurize script is it.
    GoingTharn authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    658cea0 View commit details
    Browse the repository at this point in the history
  2. Port option package to python3 (pantsbuild#6117)

    Followup for pantsbuild#6062. Almost all automated changes, the exception of the change from using `map()` to a for loop.
    ghthor authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    bf84716 View commit details
    Browse the repository at this point in the history
  3. Port binaries package to Python 3 (pantsbuild#6127)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    130a937 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5f36f15 View commit details
    Browse the repository at this point in the history
  5. Preserve output directories if process execution failed (pantsbuild#6152

    )
    
    Right now, this is in an and_then block, so doesn't get called if process execution fails.
    illicitonion authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    0428780 View commit details
    Browse the repository at this point in the history
  6. add the target fingerprint to the version of each local dist so that …

    …we don't use the first cached one (pantsbuild#6022)
    
    ### Problem
    
    I made pantsbuild#5479 because the dist resolved for a `python_dist()` would not change after changing any of the source files, and this persisted even after a `clean-all`. This issue is more thoroughly explained in pantsbuild#5449 (which I literally didn't see before making this PR, but describes almost exactly what this PR does, it seems -- great minds think alike?). We were building the `python_dist()` targets each time they were invalidated, but caching the package locally in `~/.cache/pants/` after we built it the first time and using that instead, because the version string was the same for both the first and subsequent builds of the `python_dist()` target.
    
    ### Solution
    
    - Move argv generation for setup.py invocations into `BuildLocalPythonDistributions`.
    - Append the `python_dist()` target's fingerprint to the version string using `--tag-build`. This conforms to the "local" version specifier format as per [PEP 440](https://www.python.org/dev/peps/pep-0440/#local-version-identifiers). This tagged version is then used in the synthetic `PythonRequirement` stitched into the build graph for the local dist.
    - Add an integration test `test_invalidation()` using `mock_buildroot()` (which fails on master, but not in this PR) where we run a binary depending on a `python_dist()`, rewrite the contents of a source file to change the output, then run pants again to verify that the new dist is resolved.
    
    *Separately:*
    
    - Made `PythonDistribution` subclass `PythonTarget` to avoid duplicated logic. It's not clear what the original reason for not subclassing it was at first, but it seems to be not be in effect anymore.
    - Changed the `dir` field name for the `Manager` namedtuple in `mock_buildroot()` to be `new_buildroot` so it doesn't get highlighted as a builtin in certain text editors.
    
    ### Result
    
    When we resolve local dists in any python requirements task, we get the dist corresponding to the target we just built. Resolves pantsbuild#5449.
    cosmicexplorer authored Jul 17, 2018
    Configuration menu
    Copy the full SHA
    dfd7f73 View commit details
    Browse the repository at this point in the history
  7. port scrooge to py3 (pantsbuild#6165)

    took futurize
    
    ### Problem
    
    Port scrooge to py3 -- took futurize suggestions
    
    ### Solution
    
    ported
    GoingTharn authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    42d19ea View commit details
    Browse the repository at this point in the history
  8. Port contrib/scalajs to py3 (pantsbuild#6164)

    ### Problem
    
    Porting to py3
    
    ### Solution
    
    futurize
    GoingTharn authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    2b8f295 View commit details
    Browse the repository at this point in the history
  9. Port java to Python 3 (pantsbuild#6159)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    e61a164 View commit details
    Browse the repository at this point in the history
  10. Port ivy to Python 3 (pantsbuild#6154)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    f5b8bd8 View commit details
    Browse the repository at this point in the history
  11. port cpp to py3 (pantsbuild#6116)

    ### Problem
    
    Porting contrib/cpp to py3
    
    ### Solution
    
    just added an import
    GoingTharn authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    baa68af View commit details
    Browse the repository at this point in the history
  12. Port invalidation to Python 3 (pantsbuild#6147)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    8225ebc View commit details
    Browse the repository at this point in the history
  13. Port help package to Python 3 (pantsbuild#6144)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    b0c86ce View commit details
    Browse the repository at this point in the history
  14. Port init package to Python 3 (pantsbuild#6145)

    Eric-Arellano authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    6c41ecf View commit details
    Browse the repository at this point in the history
  15. Port go to py3 (pantsbuild#6139)

    ### Problem
    
    Upgrading go to be compat w/py3
    
    ### Solution
    
    Very few, script handled 100% but I did revert a couple of excess lists.
    GoingTharn authored and Stu Hood committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    68bbe91 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2018

  1. Upgrade to isort 4.3.4. (pantsbuild#6166)

    This picks up python3 std lib module knowledge.
    
    In addition, remove the un-needed isort 3rdparty dep and the
    contrib/python/src/python/pants/contrib/python/isort binary. The isort
    pex is now directly built via script in pantsbuild/binaries.
    
    Fixes pantsbuild#6149
    jsirois authored and Stu Hood committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    1ad39c1 View commit details
    Browse the repository at this point in the history
  2. Port engine to Python 3 (pantsbuild#6133)

    Eric-Arellano authored and Stu Hood committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    2bf15d4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    aceff3e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fbb6175 View commit details
    Browse the repository at this point in the history
  5. rustfmt check more reliably works (pantsbuild#6172)

    We were running `--write-mode=diff` which used to exit 4 if your code as
    incorrectly formatted, and now doesn't. It now only exits non-0 if
    your code doesn't parse. `--write-mode=check` *does* exit non-zero if
    your code isn't correctly formatted, but provides no way of
    distinguishing "Your code is incorrectly formatted" from "your code
    doesn't parse". So that's nice of it.
    illicitonion authored Jul 18, 2018
    Configuration menu
    Copy the full SHA
    b902e7f View commit details
    Browse the repository at this point in the history
  6. Fix isort issues (pantsbuild#6174)

    Eric-Arellano authored and jsirois committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    d3f2005 View commit details
    Browse the repository at this point in the history
  7. Conan (third party) support for ctypes native libraries (pantsbuild#5998

    )
    
    ### Problem
    
    The new targets introduced in pantsbuild#5815 have no means of declaring, fetching, and using third party native dependencies.
    
    ### Solution
    
    Integrate the Conan package manager to fetch packages from a remote package store via a new task, copy the package data to the results directory of `third_party_native_library` targets in play, and create a product that the compile and link tasks can consume. Plumb the directory paths provided by the product through to the command lines of the compile and link steps.
    
    ### Result
    
    Users can now depend on third party native libraries in their `ctypes_compatible_cpp_library` targets from either conan-center or a remote URI that they specify via an option.
    
    ## Some notes
    - The conan home directory is currently under .pants.d, instead of ~ or ~/.cache/pants. I did this for the short term to make debugging cache problems and other issues as simple as a ./pants clean-all. I don't think the perf loss will be too bad (1-2 seconds).
    - Some string manipulation could be better done as regex, and I have TODOs for that
    - I am going to follow up with upstream Conan about getting a flag for cleaner client output so the parse method does not need to be so ugly.
    CMLivingston authored and cosmicexplorer committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    4aec12c View commit details
    Browse the repository at this point in the history
  8. Merge branch 'master' into port-build-graph

    Eric Arellano committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    f5e7691 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2018

  1. Fix incorrect use of bytes() when invoking the daemon in a tty (pants…

    …build#6181)
    
    ### Problem
    
    `pkill -f "pantsd \[" pantsd-runner && ./pants --enable-pantsd help` fails on master when run within a tty because we incorrectly try to pass an `int` directly to a `bytes`. This was introduced during pantsbuild#6159.
    
    ### Solution
    
    - Wrap the `bytes()` argument in a list in `NailgunProtocol.isatty_to_env()`.
    - Add unit testing which mocks out the tty querying to ensure the environment we return is valid.
    
    ### Result
    
    No more crashing when trying to create a pantsd in a tty.
    
    #### Thoughts
    
    This may be another argument for something like pantsbuild#6157. There may also be room for an integration test which opens up a "real" pty to catch issues like these in the future, but it's not clear to me how to do that right now.
    cosmicexplorer authored Jul 19, 2018
    Configuration menu
    Copy the full SHA
    7b04ad7 View commit details
    Browse the repository at this point in the history
  2. Prepare the 1.9.0rc0 release. (pantsbuild#6171)

    * add 1.9.x.rst copied from previous release notes
    
    * add dates and SHAs
    
    * bump VERSION file
    
    * learn how to use release-changelog-helper.sh
    
    * revert VERSION file changes
    
    * copy 1.9.0rc0 notes over into master.rst
    
    * Revert "revert VERSION file changes"
    
    This reverts commit d031577.
    
    * run ./build-support/bin/contributors.sh
    
    * convert ---- subheading to ~~~~~~
    
    * bump release candidate date
    
    * update changelog after rebase
    
    * add bugfix commit
    cosmicexplorer authored Jul 19, 2018
    Configuration menu
    Copy the full SHA
    3afceb9 View commit details
    Browse the repository at this point in the history
  3. Allow user to specify chunk size (pantsbuild#6173)

    Also, default is 3M not 10M, as grpc has a ~4M limit per-message
    illicitonion authored Jul 19, 2018
    Configuration menu
    Copy the full SHA
    c81d4c1 View commit details
    Browse the repository at this point in the history
  4. Convert fmt.isort to bootstrapping isort. (pantsbuild#6182)

    Previously we relied on a pre-deployed `isort.pex` binary; now we
    create an `isort.pex` locally as-needed.
    jsirois authored Jul 19, 2018
    Configuration menu
    Copy the full SHA
    a51b977 View commit details
    Browse the repository at this point in the history
  5. Ci deduplication (pantsbuild#6186)

    Deduplicate test config.
    Eric-Arellano authored and jsirois committed Jul 19, 2018
    Configuration menu
    Copy the full SHA
    6dec8de View commit details
    Browse the repository at this point in the history
  6. Merge branch 'master' of github.com:pantsbuild/pants into port-build-…

    …graph
    Eric Arellano committed Jul 19, 2018
    Configuration menu
    Copy the full SHA
    ae36579 View commit details
    Browse the repository at this point in the history
  7. Fix breaking tests!

    Eric Arellano committed Jul 19, 2018
    Configuration menu
    Copy the full SHA
    0813ea0 View commit details
    Browse the repository at this point in the history