build: validate the source directory passed to ProjectBuilder #260

pfmoore · 2021-03-18T18:04:27Z

No description provided.

pfmoore · 2021-03-18T18:06:22Z

Fixes #259.

Initial version, for some reason the tox -e fix command doesn't work on my PC (it doesn't recognise my copy of Visual Studio when building the pre-commit environments) so I can't run the linting locally. I'll let CI do it for me...

pfmoore · 2021-03-18T18:42:05Z

I don't know why the tests are failing, they work locally.

FFY00

I think checking for either pyproject.toml or setup.py is fine, even though I don't like that people with a setup.cfg will continue with the current behavior, making things slightly inconsistent, so I am not gonna block this.

FFY00 · 2021-03-18T18:44:51Z

src/build/__init__.py

+ pyproject_toml = os.path.join(srcdir, "pyproject.toml")
+ setup_py = os.path.join(srcdir, "setup.py")


You are not running pre-commit. Please use single quotes here 😛

Thanks. Is the right way to run pre-commit to use the tox -e fix command? As I said, that's failing for me :-( Is there a contributing guide that explains how to set up for development, run tests, etc? I wasn't able to find one.

I tried running it by hand, which seems to have worked. So I've pushed a fix.

It should, I don't know exactly what went wrong in your case 😕. Perhaps @gaborbernat might have more insight?

We should have a contributor guide, but right now I'm ETIME.

No worries, I completely understand.

Thanks. Is the right way to run pre-commit to use the tox -e fix command? As I said, that's failing for me

Can you provide more exact details for this?

I'll try to later, but basically it said I didn't have Visual C version 14 installed (I have VS 2019) when building the virtualenvs. But when I ran pre-commit outside of tox, it worked, and now tox -e fix works.

Weird. Was it trying to build some c-extension package (such as black?)

Yes. I can't recall which one, I'll let you know once I manage to reproduce the issue.

src/build/__init__.py

FFY00 · 2021-03-18T19:35:47Z

You seem to have found an issue with our integration tests 😅. We test building a few popular projects in the CI as a sanity check and it looks like we are invoking build in the directory where the github tarballs are extracted, not the folder that is inside them and contains the project.

I will open a separate PR, you can then rebase on top of it.

FFY00 · 2021-03-18T19:39:07Z

Btw, as the integration tests take a long time and are only included as a sanity check, you need to pass --only-integration or --run-integration to pytest. You can do this in tox as tox -e py39 -- --only-integration.

Sorry for the trouble!

gaborbernat · 2021-03-18T19:48:48Z

I don't know why the tests are failing, they work locally.

This is why I'm not the biggest fan of not running integration tests locally by default 😆 People think if it passed locally they are done.

gaborbernat

The logic looks ok, but we need tests for these cases.

pfmoore · 2021-03-18T21:30:30Z

Yep, once the current tests pass I'll add tests for the new functionality.

FFY00 · 2021-03-18T22:21:31Z

The tests should now be fixed now, @pfmoore can you rebase your branch on main?

pfmoore · 2021-03-18T22:31:36Z

Thanks for the quick fix, rebase done.

FFY00 · 2021-03-18T22:51:32Z

Thanks, when you come back to add the tests, could you also add an entry in the changelog?

pfmoore · 2021-03-18T23:05:22Z

Done.

By the way, the symlink from docs/changelog.rst to CHANGELOG.rst doesn't get extracted properly on Windows, and messes up the git repo. Twice, I nearly uploaded a commit which changed that file to a plain text file containing the text ../CHANGELOG.rst 🙁

For portability, it might be better not to include symlinks in the repository...

FFY00 · 2021-03-19T00:58:37Z

Uh, I thought symlinks were fixed on Windows, I guess that's not the case.

gaborbernat · 2021-03-19T01:00:44Z

I mean could be Paul doesn't have symlinks enabled 🤷‍♂️

FFY00

Other than this, it should be good! Thanks 😊

CHANGELOG.rst

pfmoore · 2021-03-19T07:52:20Z

I do have symlinks enabled, and they do work. I suspect that the problem is that git might not handle them. I'm using a very vanilla git setup.

pfmoore · 2021-03-19T08:15:50Z

@gaborbernat I've raised #262 with details of the pre-commit failure.

@FFY00 I think I've done everything needed now. I've no idea why the coverage has gone down (or what I need to do to fix that). The coverage report seems to be flaggking lines that I haven't touched as uncovered 🙁

Let me know what (if anything) more I need to do.

FFY00 · 2021-03-19T13:00:33Z

Thanks! The coverage is a bit fiddly, it is fine now.

pfmoore · 2021-03-19T14:12:39Z

Thanks @FFY00 and @gaborbernat for the help getting this PR through 🙂 Much appreciated.

gaborbernat

Otherwise looks good.

gaborbernat · 2021-03-19T15:26:49Z

tests/test_projectbuilder.py

@@ -136,6 +136,18 @@ def test_check_dependency(monkeypatch, requirement_string, expected):
 assert next(build.check_dependency(requirement_string), None) == expected


+def test_bad_project(test_no_project_path):
+ # Passing a nonexistent project directory
+ with pytest.raises(build.BuildException):


Can you split each test into its own test please? 👌

The code already went in. This is not a big deal though.

henryiii · 2021-03-24T15:07:05Z

Sorry I missed this. It's also valid to have setup.cfg but no setup.py or pyproject.toml.

henryiii · 2021-03-24T15:08:25Z

See: https://github.com/joerick/cibuildwheel/blob/96aa38b56c241773f612ee702d638eb76c80576b/cibuildwheel/__main__.py#L164-L169

layday · 2021-03-24T16:27:24Z

It's not for legacy projects - how are you going to be able to invoke sdist or bdist_wheel? If it works in pip it must be because it calls setup() on the fly.

pfmoore · 2021-03-24T16:37:29Z

If there's no pyproject.toml then (in certain circumstances - it's complicated with all sorts of compatibility cruft) pip will use the setuptools.build_meta:__legacy__ backend, as suggested in PEP 517.

Also, see my comment here. In terms of PEP 517, there are only two sorts of source tree - ones with pyproject.toml and ones with setup.py. Nothing else is technically a Python source tree, if you accept PEP 517's definition.

henryiii · 2021-03-24T20:16:38Z

Not having a pyproject.toml is not something I do, but setup.cfg only projects are valid, and currently work in build:

pipx run cookiecutter gh:scikit-hep/cookie
rm pyproject.toml
rm setup.py
pipx run build

This correctly produces a wheel and SDist, just like pip would. (Technically, I'm not sure how you would make an SDist without build, actually). Current master would now be broken. We suggest setup.cfg only projects on https://packaging.python.org/tutorials/packaging-projects/ (with a pyproject.toml too, of course, but technically it can be omitted and pip and current build work).

henryiii · 2021-03-24T20:32:30Z

Quick followup, pip checks this:

python3 -m pip wheel .
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

So being just as picky in build is perfectly fine. Though it is a removal of something that used to work.

henryiii · 2021-05-26T04:01:12Z

As of pypa/pip#9945, we've now swapped with Pip. Pip 21.1.2 does support just setup.cfg now, and now we don't. 🤣

$ pip wheel .
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/Users/henryschreiner/tmp/quickcheck/venv/bin/python3.9 -m pip install --upgrade pip' command.
$ pip install --upgrade pip
$ pip wheel .
Processing /Users/henryschreiner/tmp/quickcheck
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting numpy>=1.13.3
  File was already downloaded /Users/henryschreiner/tmp/quickcheck/numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl
Building wheels for collected packages: quickcheck
  Building wheel for quickcheck (setup.py) ... done
  Created wheel for quickcheck: filename=quickcheck-0.0.0-py3-none-any.whl size=3372 sha256=57acfb84109eedde21404ffdea6ef0e87d6ef7663498e3220bb0c4a72b066d35
  Stored in directory: /private/var/folders/_8/xtbws09n017fbzdx9dmgnyyr0000gn/T/pip-ephem-wheel-cache-62ksr1uu/wheels/9a/34/3d/e8d9298f424ae357bac078a56990fce39937c7ed9018594f47
Successfully built quickcheck

pfmoore requested review from FFY00, gaborbernat and layday as code owners March 18, 2021 18:04

FFY00 approved these changes Mar 18, 2021

View reviewed changes

gaborbernat requested changes Mar 18, 2021

View reviewed changes

pfmoore added 3 commits March 18, 2021 22:30

Validate the source directory passed to ProjectBuilder

20e2ffd

Lint fixes

2a12106

Rewrite test logic

8c37d99

pfmoore force-pushed the check_setup_py branch from e39f212 to 8c37d99 Compare March 18, 2021 22:31

pfmoore added 3 commits March 18, 2021 22:55

Add tests for the new functionality

f027d23

Add a changelog entry

1400a34

Lint fixes

1bfe27d

FFY00 approved these changes Mar 19, 2021

View reviewed changes

CHANGELOG.rst Outdated Show resolved Hide resolved

Changelog fixes

3fb27c7

FFY00 changed the title ~~Validate the source directory passed to ProjectBuilder~~ build: validate the source directory passed to ProjectBuilder Mar 19, 2021

FFY00 merged commit 2c1da83 into pypa:main Mar 19, 2021

pfmoore deleted the check_setup_py branch March 19, 2021 14:12

gaborbernat requested changes Mar 19, 2021

View reviewed changes

henryiii mentioned this pull request Mar 24, 2021

build does not validate that the "source directory" argument is a valid source directory #259

Closed

This was referenced May 26, 2021

Restore setup.cfg only packages? #302

Closed

Better diagnose when setup.py/cfg cannot be found pypa/pip#9945

Merged

AndydeCleyre mentioned this pull request May 25, 2022

Replace direct usage of the pep517 module with the build module, for loading project metadata jazzband/pip-tools#1629

Merged

4 tasks

		pyproject_toml = os.path.join(srcdir, "pyproject.toml")
		setup_py = os.path.join(srcdir, "setup.py")

build: validate the source directory passed to ProjectBuilder #260

build: validate the source directory passed to ProjectBuilder #260

Conversation

pfmoore commented Mar 18, 2021

pfmoore commented Mar 18, 2021

pfmoore commented Mar 18, 2021

FFY00 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FFY00 commented Mar 18, 2021

FFY00 commented Mar 18, 2021 • edited Loading

gaborbernat commented Mar 18, 2021

gaborbernat left a comment

Choose a reason for hiding this comment

pfmoore commented Mar 18, 2021

FFY00 commented Mar 18, 2021

pfmoore commented Mar 18, 2021

FFY00 commented Mar 18, 2021

pfmoore commented Mar 18, 2021

FFY00 commented Mar 19, 2021

gaborbernat commented Mar 19, 2021

FFY00 left a comment

Choose a reason for hiding this comment

pfmoore commented Mar 19, 2021

pfmoore commented Mar 19, 2021

FFY00 commented Mar 19, 2021

pfmoore commented Mar 19, 2021

gaborbernat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii commented Mar 24, 2021

henryiii commented Mar 24, 2021

layday commented Mar 24, 2021

pfmoore commented Mar 24, 2021 • edited Loading

henryiii commented Mar 24, 2021 • edited Loading

henryiii commented Mar 24, 2021

henryiii commented May 26, 2021

FFY00 commented Mar 18, 2021 •

edited

Loading

pfmoore commented Mar 24, 2021 •

edited

Loading

henryiii commented Mar 24, 2021 •

edited

Loading