Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python distribution task for user-defined setup.py + integration with ./pants {run/binary/test} #5141

Merged
merged 91 commits into from
Feb 7, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
3ec11c2
Python_distribution example
Nov 10, 2017
ca86bf6
First pass at distribution task and target
Nov 15, 2017
46eab74
Working package example for hello2
Nov 21, 2017
3aa0fe4
Changes to hello2 package
Nov 21, 2017
cd32058
Changes to python binary creation + add new target definition
Nov 21, 2017
4f9bc98
Working python dist example
Nov 21, 2017
5837f30
Modify example to use superhello
Nov 21, 2017
dd99194
Working superhello project equivalent to backends/tensorflow
Nov 21, 2017
4e2b1b9
Minor cleanup
Nov 21, 2017
74bfb8c
Finalized initial approach at packaging wheels w/ c sources into a pex
Nov 22, 2017
83be228
Add tensorflow dependency example to superhello; this puts ./pants bi…
Nov 24, 2017
c8a954e
Add cleanup of egg-info in pex build util
Nov 28, 2017
9cd32a4
Clean up + add some docs
Nov 28, 2017
1940c53
Clean up misc style issues and delete unnecessary file.
Nov 28, 2017
0ae435e
Remove excess __init__.py files
Nov 28, 2017
6585142
First stab at run task
Nov 29, 2017
194452b
pants run task implementation - first pass
Nov 29, 2017
e2f84cc
Add python_distribution to backend/targets build file
Nov 29, 2017
7297096
First pass at kwlzn change suggestions
Dec 1, 2017
9bb228f
PythonDistribution -> inherit Target instead of PythonTarget
Dec 5, 2017
c340abf
Create python distribution task and refactor example
Dec 6, 2017
4b65461
Debug session 1
Dec 6, 2017
e4c0556
Install Python Dist create task
Dec 7, 2017
7823d2a
Progress on pex_build_util
Dec 8, 2017
76aeda2
Working distribution creation task
Dec 8, 2017
4b64f2d
Further iteration on python distribution task, now functioning for py…
Dec 9, 2017
b6fa385
Minor cleanup + rename alias to be standard
Dec 12, 2017
8436dad
More cleanup of docs
Dec 12, 2017
1c981f7
Run/test
Dec 12, 2017
6881a72
Revert pants run mods and fix bug in pex build util
Dec 12, 2017
a859f46
Remove cruft
Dec 12, 2017
314ecfb
pants test example
Dec 13, 2017
23f6999
Working distribution create task and integration for pants run/binary…
Dec 13, 2017
fb1c903
Fix minor BUILD file error to pass CI
Dec 13, 2017
59ff560
Add guard statement for case where pants test run does not require da…
Dec 14, 2017
51e9c64
Remove unused imports and add guard from consuming python dist products
Dec 14, 2017
7b8e8be
Remove unused imports created from refactor
Dec 15, 2017
177c379
Whitespace lint fixes and unused import
Dec 15, 2017
31d3a91
Another whitespace error
Dec 15, 2017
8c7b658
Easy nits and whitespace issues
Dec 15, 2017
edf110d
More cleanup + move stuff in task execution under invalidated
Dec 19, 2017
dacd672
Working caching under vt.results dir. Moving to tests.
Dec 19, 2017
53a1355
Solid working state based off of vt.results dir caching
Dec 20, 2017
401a10d
Working goals using invalidated blocks
Dec 20, 2017
056e267
Add integration testing and simple unit test for python create distri…
Dec 21, 2017
5f4e07c
Add detection of multiple setup.py files and throw an error.
Dec 21, 2017
365e708
Style fixes and cruftslaying per rjiang's comments
Dec 21, 2017
3e1739f
Fix merge conflict
Dec 21, 2017
e2bf445
Clean up comments, docstrings, and fix broken testprojects integratio…
Dec 21, 2017
022b1e7
Add rjiang suggestion for counting setup.py files
Dec 22, 2017
be2583a
Addresses a few changes
Jan 6, 2018
f1ed6da
Fix install directory clobbering and setup.py positioning
Jan 6, 2018
4e9bb6b
Fix install directory clobbering and setup.py positioning
Jan 6, 2018
03bc4c0
Remove unneccessary checks for invalid targets and streamline method …
Jan 9, 2018
a2116c4
Remove cruft
Jan 9, 2018
9e0b2d6
Add TODO with github link for package conflict case in python dist ba…
Jan 9, 2018
e8f6b98
Fix multiple binary target case and add integration test
Jan 10, 2018
4fd1567
Cleanup integration test and move superhello test project to examples…
Jan 10, 2018
d41e7ee
Remove unnecessary targets from BUILD file in superhello_testproject
Jan 11, 2018
a6377e4
Remove tests that break testprojects integration testing
Jan 12, 2018
aabf88f
Update TODO github issue link
Jan 12, 2018
fe8dd55
Simplify a few lines, add check for ambiguous python dists, and fix c…
Jan 12, 2018
2925530
Edge case impl for same setup.py package name/version as a binary dep
Jan 13, 2018
c6edc08
remove unneeded dependency test
Jan 13, 2018
ffae43d
Remove crufty files
Jan 16, 2018
cae065c
Add integration tests for targets that conflict with transitive deps …
Jan 16, 2018
178bcbd
Fix issues with CI and failing testprojects target
Jan 17, 2018
a4150d0
Remove duplicate functions to enforce DRY
Jan 18, 2018
22b72f0
Remove add_labels from PythonDistribution object
Jan 18, 2018
654c20e
Remove unused import
Jan 18, 2018
eb8598d
Fix lint
Jan 19, 2018
3024a34
Add xfail test to testprojects tests
Jan 19, 2018
7f757c0
Disallow dependencies on a python_dist target
Jan 22, 2018
0b82067
Resolve merge conflicts from removal of tasks2
Jan 23, 2018
a787717
Slightly modify test assertion for conflicting deps test
Jan 24, 2018
8ac832b
Add clean-all statements to integration tests to gauge flakiness
Jan 24, 2018
59164c1
Add remove command to travis.yml to remove problematic file from fail…
Jan 24, 2018
b4d367c
Rename superhello to fasthello
Jan 25, 2018
1a04e80
Rebase with master
Jan 30, 2018
9e3d2ca
Remove mod to travis yml
Jan 30, 2018
d6dca9c
Simplify guard clause for built dists
Feb 1, 2018
f5d3a52
Formatting changes
Feb 6, 2018
46bd7c0
Add PythonDistribution.alias to register
Feb 6, 2018
c231963
Remove redundant py_dist adding and simplify req lib injection
Feb 6, 2018
8c6a4b9
Use cache keys for addressing synthetic targets
Feb 6, 2018
445cb98
Change resolve requirements call site and signature
Feb 6, 2018
1dc8724
Add support and test case for binary task python dist isolation
Feb 6, 2018
de1939b
Fix CI failures
Feb 6, 2018
f65978e
Fix lint error
Feb 6, 2018
bf2b553
Refactor synthetic req injection method
Feb 7, 2018
4b2fae5
Synthetic address modifications
Feb 7, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

# Like Hello world, but built with a python_dist.
# python_dist allows you to use setup.py to depend on C/C++ extensions.

python_dist(
name='fasthello',
sources=[
'super_greet.c',
'hello_package/hello.py',
'hello_package/__init__.py',
'setup.py'
]
)

python_binary(
name='main',
source='main.py',
dependencies=[
':fasthello',
]
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import super_greet

def hello():
print(super_greet.super_greet())
return super_greet.super_greet()
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

# hello_package is a python module within the fasthello python_distribution
from hello_package import hello


if __name__ == '__main__':
hello.hello()
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

from setuptools import setup, find_packages
from distutils.core import Extension


c_module = Extension(str('super_greet'), sources=[str('super_greet.c')])

setup(
name='fasthello',
version='1.0.0',
ext_modules=[c_module],
packages=find_packages(),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#include <Python.h>

static PyObject * super_greet(PyObject *self, PyObject *args) {
return Py_BuildValue("s", "Super hello");
}

static PyMethodDef Methods[] = {
{"super_greet", super_greet, METH_VARARGS, "A super greeting"},
{NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initsuper_greet(void) {
(void) Py_InitModule("super_greet", Methods);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

# Example of defining a test target that depends on a python_dist target.

python_tests(
name='fasthello',
sources=[
'test_fasthello.py'
],
dependencies=[
'examples/src/python/example/python_distribution/hello/fasthello:fasthello'
]
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

# hello_package is a python module within the fasthello python_distribution.
from hello_package import hello


# Example of writing a test that depends on a python_dist target.
def test_fasthello():
assert hello.hello() == "Super hello"
5 changes: 5 additions & 0 deletions src/python/pants/backend/python/register.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@
from pants.backend.python.python_requirement import PythonRequirement
from pants.backend.python.python_requirements import PythonRequirements
from pants.backend.python.targets.python_binary import PythonBinary
from pants.backend.python.targets.python_distribution import PythonDistribution
from pants.backend.python.targets.python_library import PythonLibrary
from pants.backend.python.targets.python_requirement_library import PythonRequirementLibrary
from pants.backend.python.targets.python_tests import PythonTests
from pants.backend.python.tasks.build_local_python_distributions import \
BuildLocalPythonDistributions
from pants.backend.python.tasks.gather_sources import GatherSources
from pants.backend.python.tasks.pytest_prep import PytestPrep
from pants.backend.python.tasks.pytest_run import PytestRun
Expand All @@ -34,6 +37,7 @@ def build_file_aliases():
PythonBinary.alias(): PythonBinary,
PythonLibrary.alias(): PythonLibrary,
PythonTests.alias(): PythonTests,
PythonDistribution.alias(): PythonDistribution,
'python_requirement_library': PythonRequirementLibrary,
Resources.alias(): Resources,
},
Expand All @@ -51,6 +55,7 @@ def build_file_aliases():

def register_goals():
task(name='interpreter', action=SelectInterpreter).install('pyprep')
task(name='build-local-dists', action=BuildLocalPythonDistributions).install('pyprep')
task(name='requirements', action=ResolveRequirements).install('pyprep')
task(name='sources', action=GatherSources).install('pyprep')
task(name='py', action=PythonRun).install('run')
Expand Down
1 change: 1 addition & 0 deletions src/python/pants/backend/python/targets/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
python_library(
sources = [
'python_binary.py',
'python_distribution.py',
'python_library.py',
'python_requirement_library.py',
'python_target.py',
Expand Down
63 changes: 63 additions & 0 deletions src/python/pants/backend/python/targets/python_distribution.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

from pex.interpreter import PythonIdentity
from twitter.common.collections import maybe_list

from pants.base.exceptions import TargetDefinitionException
from pants.base.payload import Payload
from pants.base.payload_field import PrimitiveField
from pants.build_graph.target import Target


class PythonDistribution(Target):
"""A Python distribution target that accepts a user-defined setup.py."""

default_sources_globs = '*.py'

@classmethod
def alias(cls):
return 'python_dist'

def __init__(self,
address=None,
payload=None,
sources=None,
compatibility=None,
**kwargs):
"""
:param address: The Address that maps to this Target in the BuildGraph.
:type address: :class:`pants.build_graph.address.Address`
:param payload: The configuration encapsulated by this target. Also in charge of most
fingerprinting details.
:type payload: :class:`pants.base.payload.Payload`
:param sources: Files to "include". Paths are relative to the
BUILD file's directory.
:type sources: ``Fileset`` or list of strings. Must include setup.py.
:param compatibility: either a string or list of strings that represents
interpreter compatibility for this target, using the Requirement-style
format, e.g. ``'CPython>=3', or just ['>=2.7','<3']`` for requirements
agnostic to interpreter class.
"""
payload = payload or Payload()
payload.add_fields({
'sources': self.create_sources_field(sources, address.spec_path, key_arg='sources'),
'compatibility': PrimitiveField(maybe_list(compatibility or ()))
})
super(PythonDistribution, self).__init__(address=address, payload=payload, **kwargs)

if not 'setup.py' in sources:
raise TargetDefinitionException(
self, 'A setup.py in the top-level directory relative to the target definition is required.'
)

# Check that the compatibility requirements are well-formed.
for req in self.payload.compatibility:
try:
PythonIdentity.parse_requirement(req)
except ValueError as e:
raise TargetDefinitionException(self, str(e))
2 changes: 2 additions & 0 deletions src/python/pants/backend/python/tasks/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ python_library(
'3rdparty/python:pex',
'3rdparty/python/twitter/commons:twitter.common.collections',
'3rdparty/python/twitter/commons:twitter.common.dirutil',
'src/python/pants/backend/python:python_requirement',
'src/python/pants/backend/python:python_requirements',
'src/python/pants/backend/python:interpreter_cache',
'src/python/pants/backend/python:python_chroot',
'src/python/pants/backend/python/subsystems',
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# coding=utf-8
# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import glob
import os
import shutil

from pex.interpreter import PythonInterpreter

from pants.backend.python.tasks.pex_build_util import is_local_python_dist
from pants.backend.python.tasks.setup_py import SetupPyRunner
from pants.base.build_environment import get_buildroot
from pants.base.exceptions import TargetDefinitionException, TaskError
from pants.base.fingerprint_strategy import DefaultFingerprintStrategy
from pants.task.task import Task
from pants.util.dirutil import safe_mkdir


class BuildLocalPythonDistributions(Task):
"""Create python distributions (.whl) from python_dist targets."""

options_scope = 'python-create-distributions'
PYTHON_DISTS = 'user_defined_python_dists'

@classmethod
def product_types(cls):
return [cls.PYTHON_DISTS]

@classmethod
def prepare(cls, options, round_manager):
round_manager.require_data(PythonInterpreter)

@property
def cache_target_dirs(self):
return True

def execute(self):
dist_targets = self.context.targets(is_local_python_dist)
built_dists = set()

if dist_targets:
with self.invalidated(dist_targets,
fingerprint_strategy=DefaultFingerprintStrategy(),
invalidate_dependents=True) as invalidation_check:
for vt in invalidation_check.all_vts:
if vt.valid:
built_dists.add(self._get_whl_from_dir(os.path.join(vt.results_dir, 'dist')))
else:
if vt.target.dependencies:
raise TargetDefinitionException(
vt.target, 'The `dependencies` field is disallowed on `python_dist` targets. List any 3rd '
'party requirements in the install_requirements argument of your setup function.'
)
built_dists.add(self._create_dist(vt.target, vt.results_dir))

self.context.products.register_data(self.PYTHON_DISTS, built_dists)

def _create_dist(self, dist_tgt, dist_target_dir):
"""Create a .whl file for the specified python_distribution target."""
interpreter = self.context.products.get_data(PythonInterpreter)

# Copy sources and setup.py over to vt results directory for packaging.
# NB: The directory structure of the destination directory needs to match 1:1
# with the directory structure that setup.py expects.
for src_relative_to_target_base in dist_tgt.sources_relative_to_target_base():
src_rel_to_results_dir = os.path.join(dist_target_dir, src_relative_to_target_base)
safe_mkdir(os.path.dirname(src_rel_to_results_dir))
abs_src_path = os.path.join(get_buildroot(),
dist_tgt.address.spec_path,
src_relative_to_target_base)
shutil.copyfile(abs_src_path, src_rel_to_results_dir)
# Build a whl using SetupPyRunner and return its absolute path.
setup_runner = SetupPyRunner(dist_target_dir, 'bdist_wheel', interpreter=interpreter)
setup_runner.run()
return self._get_whl_from_dir(os.path.join(dist_target_dir, 'dist'))

def _get_whl_from_dir(self, install_dir):
"""Return the absolute path of the whl in a setup.py install directory."""
dists = glob.glob(os.path.join(install_dir, '*.whl'))
if len(dists) == 0:
raise TaskError('No distributions were produced by python_create_distribution task.')
if len(dists) > 1:
raise TaskError('Ambiguous local python distributions found: {}'.format(dists))
return dists[0]
48 changes: 47 additions & 1 deletion src/python/pants/backend/python/tasks/pex_build_util.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,16 @@
from pex.resolver import resolve
from twitter.common.collections import OrderedSet

from pants.backend.python.python_requirement import PythonRequirement
from pants.backend.python.subsystems.python_setup import PythonSetup
from pants.backend.python.targets.python_binary import PythonBinary
from pants.backend.python.targets.python_distribution import PythonDistribution
from pants.backend.python.targets.python_library import PythonLibrary
from pants.backend.python.targets.python_requirement_library import PythonRequirementLibrary
from pants.backend.python.targets.python_tests import PythonTests
from pants.base.build_environment import get_buildroot
from pants.base.exceptions import TaskError
from pants.build_graph.address import Address
from pants.build_graph.files import Files
from pants.python.python_repos import PythonRepos

Expand All @@ -34,6 +37,10 @@ def has_python_sources(tgt):
return is_python_target(tgt) and tgt.has_sources()


def is_local_python_dist(tgt):
return isinstance(tgt, PythonDistribution)


def has_resources(tgt):
return isinstance(tgt, Files) and tgt.has_sources()

Expand Down Expand Up @@ -113,7 +120,6 @@ def dump_requirements(builder, interpreter, req_libs, log, platforms=None):

# Resolve the requirements into distributions.
distributions = _resolve_multi(interpreter, reqs_to_build, platforms, find_links)

locations = set()
for platform, dists in distributions.items():
for dist in dists:
Expand Down Expand Up @@ -158,3 +164,43 @@ def _resolve_multi(interpreter, requirements, platforms, find_links):
allow_prereleases=python_setup.resolver_allow_prereleases)

return distributions


def inject_synthetic_dist_requirements(build_graph, local_built_dists, synthetic_address, binary_tgt=None):
"""Inject a synthetic requirements library from a local wheel.

:param build_graph: The build graph needed for injecting synthetic targets.
:param local_built_dists: A list of paths to locally built wheels to package into
requirements libraries.
:param synthetic_address: A generative address for addressing synthetic targets.
:param binary_tgt: An optional parameter to be passed only when called by the `python_binary_create`
task. This is needed to ensure that only python_dist targets in a binary target's closure are included
in the binary for the case where a user specifies mulitple binary targets in a single invocation of
`./pants binary`.
:return: a :class: `PythonRequirementLibrary` containing a requirements that maps to a locally-built wheels.
"""
def should_create_req(bin_tgt, loc):
if not bin_tgt:
return True
# Ensure that a target is in a binary target's closure. See docstring for more detail.
return any([tgt.id in loc for tgt in bin_tgt.closure()])

def python_requirement_from_wheel(path):
base = os.path.basename(path)
whl_dir = os.path.dirname(path)
whl_metadata = base.split('-')
req_name = '=='.join([whl_metadata[0], whl_metadata[1]])
return PythonRequirement(req_name, repository=whl_dir)

local_whl_reqs = [
python_requirement_from_wheel(whl_location)
for whl_location in local_built_dists
if should_create_req(binary_tgt, whl_location)
]

if not local_whl_reqs:
return []

addr = Address.parse(synthetic_address)
build_graph.inject_synthetic_target(addr, PythonRequirementLibrary, requirements=local_whl_reqs)
return [build_graph.get_target(addr)]
Loading