Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude test and docs from package #378

Merged
merged 1 commit into from
Jun 1, 2023
Merged

Conversation

mikelolasagasti
Copy link
Contributor

After the migration to pyproject.toml done in 3.4.0, creating the package for Fedora fails with the following error:

RPM build errors:
error: Installed (but unpackaged) file(s) found:
   /usr/lib/python3.11/site-packages/docs/imgs/logo-horizontal.png
   /usr/lib/python3.11/site-packages/docs/imgs/logo-horizontal.svg
   /usr/lib/python3.11/site-packages/docs/imgs/logo.svg
   /usr/lib/python3.11/site-packages/docs/imgs/screenshot.png
   /usr/lib/python3.11/site-packages/docs/man/build-man.sh
   /usr/lib/python3.11/site-packages/docs/man/pg_activity.1
   /usr/lib/python3.11/site-packages/docs/man/pg_activity.pod
   /usr/lib/python3.11/site-packages/tests/__pycache__/conftest.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/conftest.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_activities.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_activities.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_config.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_config.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_data.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_data.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_types.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_types.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_views.cpython-311.opt-1.pyc
   /usr/lib/python3.11/site-packages/tests/__pycache__/test_views.cpython-311.pyc
   /usr/lib/python3.11/site-packages/tests/conftest.py
   /usr/lib/python3.11/site-packages/tests/data/local-processes-input.json
   /usr/lib/python3.11/site-packages/tests/test_activities.py
   /usr/lib/python3.11/site-packages/tests/test_config.py
   /usr/lib/python3.11/site-packages/tests/test_data.py
   /usr/lib/python3.11/site-packages/tests/test_scroll.txt
   /usr/lib/python3.11/site-packages/tests/test_types.py
   /usr/lib/python3.11/site-packages/tests/test_ui.txt
   /usr/lib/python3.11/site-packages/tests/test_views.py
   /usr/lib/python3.11/site-packages/tests/test_views.txt
   /usr/lib/python3.11/site-packages/tests/test_widgets.txt

Those files were not installed before 3.4.0 and are missing the pgactivity package name.

Excluding tests and docs package can be built again.

@dlax
Copy link
Member

dlax commented Jun 1, 2023

Thank you for the report; indeed this is problematic, and I can confirm the issue with python -m installer dist/pg_activity-3.4.1-py3-none-any.whl.

However, after re-reading setuptools documentation about automatic discovery, I wonder if we shouldn't just remove the packages configuration entry:

diff --git a/pyproject.toml b/pyproject.toml
index c67455f..2fbf803 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -79,9 +79,6 @@ Homepage = "https://github.com/dalibo/pg_activity"
 [tool.setuptools.dynamic]
 version = { attr = "pgactivity.__version__" }
 
-[tool.setuptools.packages.find]
-where = ["."]
-
 [tool.black]
 line-length = 88
 include = '\.pyi?$'

Can you try if that works for you?

@mikelolasagasti
Copy link
Contributor Author

It fails with:

+ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3 --wheeldir /builddir/build/BUILD/pg_activity-3.4.1/pyproject-wheeldir --output /builddir/build/BUILD/pg_activity-3.4.1-1.fc39.x86_64-pyproject-buildrequires -r -x tests
Handling setuptools from build-system.requires
Requirement satisfied: setuptools
   (installed: setuptools 67.7.2)
/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:66: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
No `packages` or `py_modules` configuration, performing automatic discovery.
`flat-layout` detected -- analysing .
discovered packages -- ['pgactivity', 'SPECPARTS', 'pgactivity.queries']
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 546, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 448, in generate_requires
    generate_build_requirements(backend, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 276, in generate_build_requirements
    new_reqs = get_requires()
               ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
    self.run_setup()
  File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
    exec(code, locals())
  File "<string>", line 3, in <module>
  File "/usr/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 159, in setup
    dist.parse_config_files()
  File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 908, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 66, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 143, in read_configuration
    return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 198, in expand_configuration
    return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 246, in expand
    self._expand_all_dynamic(dist, package_dir)
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 291, in _expand_all_dynamic
    version=self._obtain_version(dist, package_dir),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 340, in _obtain_version
    return _expand.version(self._obtain(dist, "version", package_dir))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 329, in _obtain
    return self._expand_directive(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 323, in _expand_directive
    return _expand.read_attr(directive["attr"], package_dir, root_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 187, in read_attr
    _parent_path, path, module_name = _find_module(module_name, package_dir, root_dir)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 228, in _find_module
    if package_dir:
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 459, in __len__
    return len(self._target())
               ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 452, in _target
    self._value = self._obtain()
                  ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 422, in _get_package_dir
    self()
  File "/usr/lib/python3.11/site-packages/setuptools/config/expand.py", line 412, in __call__
    self._dist.set_defaults(name=False)  # Skip name, we can still be parsing
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/discovery.py", line 351, in __call__
    self._analyse_package_layout(ignore_ext_modules)
  File "/usr/lib/python3.11/site-packages/setuptools/discovery.py", line 384, in _analyse_package_layout
    or self._analyse_flat_layout()
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/discovery.py", line 441, in _analyse_flat_layout
    return self._analyse_flat_packages() or self._analyse_flat_modules()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/discovery.py", line 447, in _analyse_flat_packages
    self._ensure_no_accidental_inclusion(top_level, "packages")
  File "/usr/lib/python3.11/site-packages/setuptools/discovery.py", line 477, in _ensure_no_accidental_inclusion
    raise PackageDiscoveryError(cleandoc(msg))
setuptools.errors.PackageDiscoveryError: Multiple top-level packages discovered in a flat-layout: ['SPECPARTS', 'pgactivity'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

@dlax
Copy link
Member

dlax commented Jun 1, 2023

It fails with:

+ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3 --wheeldir /builddir/build/BUILD/pg_activity-3.4.1/pyproject-wheeldir --output /builddir/build/BUILD/pg_activity-3.4.1-1.fc39.x86_64-pyproject-buildrequires -r -x tests
Handling setuptools from build-system.requires
Requirement satisfied: setuptools
   (installed: setuptools 67.7.2)
[...]
setuptools.errors.PackageDiscoveryError: Multiple top-level packages discovered in a flat-layout: ['SPECPARTS', 'pgactivity'].
[...]

Where does this SPECPARTS come from, is this a left-over of a previous build?

@mikelolasagasti
Copy link
Contributor Author

It's created by Fedora's build system:

Building target platforms: x86_64
Building for target x86_64
setting SOURCE_DATE_EPOCH=1685577600
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.MqqlUk
+ umask 022
+ cd /builddir/build/BUILD
+ cd /builddir/build/BUILD
+ rm -rf pg_activity-3.4.1
+ /usr/lib/rpm/rpmuncompress -x /builddir/build/SOURCES/v3.4.1.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd pg_activity-3.4.1
+ /usr/bin/mkdir -p SPECPARTS
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ /usr/lib/rpm/rpmuncompress /builddir/build/SOURCES/0001-Exclude-test-and-docs-from-package.patch
+ /usr/bin/patch -s --fuzz=0 --no-backup-if-mismatch -f
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(%generate_buildrequires): /bin/sh -e /var/tmp/rpm-tmp.7x2S6e
+ umask 022
+ cd /builddir/build/BUILD
+ cd pg_activity-3.4.1
(...)

You can check a full log here:

https://download.copr.fedorainfracloud.org/results/mikelo2/psycopg/fedora-rawhide-x86_64/05992267-pg_activity/builder-live.log.gz

@dlax
Copy link
Member

dlax commented Jun 1, 2023

Ok; can you try that?

diff --git a/pyproject.toml b/pyproject.toml
index c67455f..5c7fbcc 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -79,8 +79,8 @@ Homepage = "https://github.com/dalibo/pg_activity"
 [tool.setuptools.dynamic]
 version = { attr = "pgactivity.__version__" }
 
-[tool.setuptools.packages.find]
-where = ["."]
+[tool.setuptools]
+packages = ["pgactivity"]
 
 [tool.black]
 line-length = 88

(Last resort attempt; I'd rather be explicit on what to include than listing what should be excluded.)

@mikelolasagasti
Copy link
Contributor Author

Fails:

+ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3 --wheeldir /builddir/build/BUILD/pg_activity-3.4.1/pyproject-wheeldir --output /builddir/build/BUILD/pg_activity-3.4.1-1.fc39.x86_64-pyproject-buildrequires -r -x tests
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 546, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 447, in generate_requires
    backend = get_backend(requirements)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 227, in get_backend
    pyproject_data = toml_load(f)
                     ^^^^^^^^^^^^
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 217, in toml_load
    return toml_module.load(load_from)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/tomllib/_parser.py", line 66, in load
    return loads(s, parse_float=parse_float)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/tomllib/_parser.py", line 113, in loads
    pos, header = create_dict_rule(src, pos, out)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/tomllib/_parser.py", line 290, in create_dict_rule
    raise suffixed_err(src, pos, f"Cannot declare {key} twice")
tomllib.TOMLDecodeError: Cannot declare ('tool', 'setuptools', 'packages', 'find') twice (at line 85, column 31)
error: Bad exit status from /var/tmp/rpm-tmp.opxXUo (%generate_buildrequires)
    Bad exit status from /var/tmp/rpm-tmp.opxXUo (%generate_buildrequires)

I'd rather be explicit on what to include than listing what should be excluded

I pushed a new change using include rather than excluding. It works for me.

@dlax
Copy link
Member

dlax commented Jun 1, 2023

tomllib.TOMLDecodeError: Cannot declare ('tool', 'setuptools', 'packages', 'find') twice (at line 85, column 31)

The message is about the toml file being invalid because it contains a duplicated tool.setuptools.packages.find section. Can you double-check the patch you applied? (The diff I posted removed that section.)

@mikelolasagasti
Copy link
Contributor Author

tomllib.TOMLDecodeError: Cannot declare ('tool', 'setuptools', 'packages', 'find') twice (at line 85, column 31)

The message is about the toml file being invalid because it contains a duplicated tool.setuptools.packages.find section. Can you double-check the patch you applied? (The diff I posted removed that section.)

Sorry, you're right, did not realize about packages.find being removed.

Works fine with your proposal. I updated the PR with that change, but as solution was yours feel free to close this and push a patch in your name.

Automatic discovery, a.k.a. `packages.find` option, requires some tweak
on include/exclude so only required files are installed. Declaring
`packages` directly simplifies this by just adding the required package.
@dlax dlax merged commit 8a5fa22 into dalibo:master Jun 1, 2023
@dlax
Copy link
Member

dlax commented Jun 1, 2023

Just published version 3.4.2 including this change. Thank you.

@hroncok
Copy link

hroncok commented Jun 5, 2023

It's created by Fedora's build system...

I've reported this at rpm-software-management/rpm#2532

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants