Add [tool.maturin.include] and [tool.maturin.exclude]

[mbrobbel]

Based on the discussion in #1253, this is another attempt to allow users to override ignore files in Python modules by adding [tool.maturin.include] and [tool.maturin.exclude] similar to Poetry. This deprecates [tool.maturin.sdist-include].

After some feedback on this approach, I'll add:

• Documentation updates
• Integration tests

Closes #1253

[netlify]

✅ Deploy Preview for maturin-guide ready!

Name	Link
🔨 Latest commit	66e3421
🔍 Latest deploy log	https://app.netlify.com/sites/maturin-guide/deploys/636bc1db300a53000863e22d
😎 Deploy Preview	https://deploy-preview-1255--maturin-guide.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[messense]

Thanks a lot for looking into this!

[messense]

This already looks pretty good, should be ready soon when you are done with tests and documentation changes.

BTW don't forget to add a changelog entry: https://github.com/PyO3/maturin/blob/main/Changelog.md#unreleased

[messense]

I think we are missing a file list check for wheel, unfortunately we don't have a existing infrastructure for this kind of test yet.

[mbrobbel]

I think we are missing a file list check for wheel, unfortunately we don't have a existing infrastructure for this kind of test yet.

I added a simple wheel file check for the test crate that was added in this PR. It helped to find an error in the target paths for files to be included in wheels.

[messense]

We can simplify this test crate.

[messense]

Looks good to me, thanks!

[messense]

bors r+

[messense]

There are some test failures: https://github.com/PyO3/maturin/actions/runs/3427030079/jobs/5709531358

[bors]

Canceled.

[messense]

bors delegate+

[bors]

✌️ mbrobbel can now approve this pull request. To approve and merge a pull request, simply reply with bors r+. More detailed instructions are available here.

[messense]

bors r+

[bors]

Already running a review

[messense]

Still need to fix the sdist test: https://github.com/PyO3/maturin/actions/runs/3427943992/jobs/5711566272, sdist shouldn't include .pyc or .so files.

[bors]

Build failed:

• conclusion

[mbrobbel]

Still need to fix the sdist test: https://github.com/PyO3/maturin/actions/runs/3427943992/jobs/5711566272, sdist shouldn't include .pyc or .so files.

I'm confused about how these files end up in the source distribution and that this fails for the added test but not for the one this was based on? Shouldn't the root .gitignore prevent those files from being added?

[messense]

I'm not sure, perhaps you can try add the ignores to https://github.com/PyO3/maturin/pull/1255/files#diff-130021135ea4ccb1cd3ba6e252477d65142317ffba1e4a10ad7af791d30c5b12 test-crates/pyo3-mixed-include-exclude/pyo3_mixed_include_exclude/.gitignore

If you want to debug it, use the alpine:edge docker image.

[messense]

You can also try modify .github/workflows/test.yml and add

    - name: Setup tmate session
      uses: mxschmitt/action-tmate@v3

to test-alpine job and get a ssh shell.

[messense]

I wasn't able to reproduce it locally using alpine:edge docker container, looks like we have to debug it with action-tmate.

[messense]

cargo package --list --allow-dirty didn't filter them

[
    "Cargo.toml",
    "Cargo.toml.orig",
    "README.md",
    "check_installed/check_installed.py",
    "pyo3_mixed_include_exclude/__init__.py",
    "pyo3_mixed_include_exclude/__pycache__/__init__.cpython-310.pyc",
    "pyo3_mixed_include_exclude/exclude_this_file",
    "pyo3_mixed_include_exclude/include_this_file",
    "pyo3_mixed_include_exclude/pyo3_mixed_include_exclude.cpython-310-x86_64-linux-gnu.so",
    "pyo3_mixed_include_exclude/python_module/__init__.py",
    "pyo3_mixed_include_exclude/python_module/__pycache__/__init__.cpython-310.pyc",
    "pyo3_mixed_include_exclude/python_module/__pycache__/double.cpython-310.pyc",
    "pyo3_mixed_include_exclude/python_module/double.py",
    "pyproject.toml",
    "src/lib.rs",
    "tests/test_pyo3_mixed_include_exclude.py",
    "tox.ini",
]

Seems to go back to this

maturin/src/source_distribution.rs

Lines 519 to 522 in 21ea2a4

	// Technically, `ignore` crate should handle this,
	// but somehow it doesn't on Alpine Linux running in GitHub Actions,
	// so we do it manually here.
	// See https://github.com/PyO3/maturin/pull/1187#issuecomment-1273987013

[messense]

bors r+

[bors]

Build succeeded:

• conclusion
• Test (arm64 Linux)
• Test (arm64 macOS)
• Test (x86_64 FreeBSD)

[mbrobbel]

Thank you for the help and quick feedback to get this in so quickly!