Allow multiple inclusion of symlinks

[agoose77]

In the process of porting a package to Hatch, I noticed that it was failing to include a directory that was symlinked to elsewhere in the project. At the moment we have a first-seen-wins approach that means the source directory (in this instance) is being ignored.

In general, I think we should want users to be able to treat symlinks are distinct (rather than looking at the inode). Do you agree?

hatch/backend/src/hatchling/builders/utils.py

Lines 16 to 26 in 05f17ca

	def safe_walk(path):
	seen = set()
	for root, dirs, files in os.walk(path, followlinks=True):
	stat = os.stat(root)
	identifier = stat.st_dev, stat.st_ino
	if identifier in seen:
	del dirs[:]
	continue
	seen.add(identifier)
	yield root, dirs, files

See scikit-hep/pyhf#1888 for the project in question!

[ofek]

Can we do that while also preventing infinite loops?

[agoose77]

I think so, let me come up with a patch.

[ofek]

hatch/tests/backend/builders/plugin/test_interface.py

Lines 152 to 171 in 6ed86c1

	@pytest.mark.requires_unix
	def test_infinite_loop_prevention(self, temp_dir):
	project_dir = temp_dir / 'project'
	project_dir.ensure_dir_exists()
	with project_dir.as_cwd():
	config = {'tool': {'hatch': {'build': {'include': ['foo', 'README.md']}}}}
	builder = Builder(str(project_dir), config=config)
	(project_dir / 'README.md').touch()
	foo = project_dir / 'foo'
	foo.ensure_dir_exists()
	(foo / 'bar.txt').touch()
	(foo / 'baz').symlink_to(project_dir)
	assert [f.path for f in builder.recurse_included_files()] == [
	str(project_dir / 'README.md'),
	str(project_dir / 'foo' / 'bar.txt'),
	]

[ofek]

Does that project currently work with setuptools? Seems like it does the same thing: https://github.com/pypa/setuptools/blob/78cb747d66bda1a6f6649e82690aaf5083a89d69/setuptools/_distutils/filelist.py#L275-L299

[agoose77]

In pyhf (which is not my project, but I did look at contributing a new pyproject.toml for), the wheel building does currently work. There is a symlink from docs/.../a-symlink to src/pyhf/schemas. Hatch seems to find docs first, so by the time it encounters src it cannot visit schemas. The reason it's a problem here and not in setuptools is that I assume setuptools is not visiting the docs, so it only ever sees one instance of the (dev, inode) pair

[matthewfeickert]

Does that project currently work with setuptools?

Yes, pyhf currently uses setuptools as our build backend.

Hatch seems to find docs first, so by the time it encounters src it cannot visit schemas

I haven't looked at what is actually being done here yet, but why does hatch need to look at all of the directories in the repository when a project is very probably only packaging a subset?

edit: Seems like that shouldn't be necessary/a problem with tool.hatch.build.targets.

[ofek]

Hatchling inclusion is based mostly on pattern filtering rather than raw paths. Let me think about this.

[ofek]

https://hatch.pypa.io/latest/config/build/#explicit-selection