More packaging fixes

[ssbarnea]

• Use modern build packaging
• Correct errors from Manifest.in which caused accidental inclusion
  of .mypy_cache folders from various locations in source tree
• Avoid tox bug with editale when setup.py is missing
• Final archive should now be less than 300kb

Related: tox-dev/tox#2197

[webknjaz]

There's a lot of unnecessary lines in the manifest. Drop them.

[webknjaz]

Why do you want to exclude it from the sdist?

[ssbarnea]

I do not see any use to include CI config in sdist, same with .github folder. On the other hand I kept test related files so you can still run tests from source archive.

[webknjaz]

The whole point of the sdist is to provide the sources for stuff, as much as possible. It's right there in the name. The users needing a small thing will use the wheel anyway (and it's configured separately). Many things use sdists as the source of truth, in particular — the downstream packagers. So don't exclude things without a really compelling reason.

[webknjaz]

Maintaining such exclusions is rather thankless. It's not worth the maintenance burden. Plus if you exclude something — add a comment with the motivation for each entry.

[webknjaz]

Looks like this line is the only entry that actually makes sense to be in this file.

[ssbarnea]

That is far from true, i tried this myself but the result caused problems in multiple places, one of them was the check-manifest. The other was that extra built docs were included and that desired json/yaml were not included from inside src/molecule. On the other hand if you put generic recursive includes for those two extensions you may endup getting stuff like src/molecule/test/.mypy_cache/.../foo.json which are excluded by default.

I know that testing behavior of Manifest.in is quite tricky as it is highly dependent on what you may have on disk, like temp files, stuff that are not VCS tracked.

check-manifest command is very useful for debugging as it reports both kind of unexpected exclusions or inclusions but still fails to give correct recommendations or to spot the bug I mentioned.

[webknjaz]

Oh, so your problem was because of the recursive inclusion and not a bug in setuptools-scm. That makes sense, then.

[ssbarnea]

@webknjaz If you can find some reliable source like official packaging docs claiming that CI/CD config should be included in sdist I will do that. AFAIK, the size is still important and we should not include stuff in bulk. I am not saying I need to remove every file but at least those folders that can influence the final size in a significant way.

I really doubt any downstream maintainer relies on sdist archives instead of having a fork of the upstream repository. It would be insane to maintain an active package using only sdists.

In fact most people strip even the tests from sdist, but we do not.

[webknjaz]

@webknjaz If you can find some reliable source like official packaging docs claiming that CI/CD config should be included in sdist I will do that. AFAIK, the size is still important and we should not include stuff in bulk. I am not saying I need to remove every file but at least those folders that can influence the final size in a significant way.

I'm a reliable source. A lot of things are still tribal knowledge, unfortunately. But I prefer to follow what the setuptools maintainer does for reducing the maintenance burden of looking after hundreds of projects on PyPI.
I agree that some extra excludes can be useful but if it doesn't cause problems reported by the actual users, I'd rather not give up the ease of maintaining this packaging setup.

I really doubt any downstream maintainer relies on sdist archives instead of having a fork of the upstream repository. It would be insane to maintain an active package using only sdists.

Well, I was speaking based on my experience and prior research. I can only speak about Gentoo and Fedora but they do exactly this — they point at sdists in the corresponding package specs of the respective ecosystems: https://github.com/gentoo/gentoo/blob/39729ace/app-admin/ansible-lint/ansible-lint-5.2.1.ebuild#L12 + https://src.fedoraproject.org/rpms/python-molecule/blob/c93b515b/f/python-molecule.spec#_20.
I attended PackagingCon three weeks ago and there were packagers from a lot of ecosystems, including distros and they all were in agreement on this.

In fact most people strip even the tests from sdist, but we do not.

And when they do, the downstream maintainers get especially unhappy about this. They come to the upstream projects asking to fix this problem. This was specifically emphasized in the talk of the Fedora's Python packagers, brought up and confirmed in the conversations on matrix, and I've heard this in the past from downstream maintainers in other places on the internet.

[webknjaz]

Duplicate of #3324