Symlinks inside package directory

[Lunarequest]

I've been working on moving the archinstall library from setup-tools to flit. currently using setup.cfg we include profiles to the wheel and sdist like so

archinstall = 
    examples/*.py
    profiles/*.py
    profiles/applications/*.py

I have been unable to do the same with flit nor find any documentation about if this is possible, could flit add this functionality?

[takluyver]

I think you're asking for what setuptools calls package_data. It shouldn't be necessary - Flit includes everything that's inside your package in the source tree.

It looks like archinstall uses symlinks to bring files from outside into the package. I think that should be OK, but it's possible something doesn't work with symlinks.

[Lunarequest]

I think you're asking for what setuptools calls package_data. It shouldn't be necessary - Flit includes everything that's inside your package in the source tree.

It looks like archinstall uses symlinks to bring files from outside into the package. I think that should be OK, but it's possible something doesn't work with symlinks.

the symlinks are with in the repo and using flit they break, using setup tools everything works as intended

[takluyver]

I think I've figured this out. We're using os.walk() to find the files to go in the wheel, which doesn't follow directory symlinks by default. You can set followlinks=True, but if we do that we need to take care to avoid infinite loops if there's a a symlink cycle.

It's also more complex for sdists, where I imagine the 'right' thing is to include the symlinks as symlinks, along with the files they point to. But what if they point outside the project directory? 🤔

For now, symlinks aren't supported. We can look at supporting them, but there's a fair bit of complexity that would have to be worked out about how they should behave.

[merwok]

It's also more complex for sdists, where I imagine the 'right' thing is to include the symlinks as symlinks

This isn’t an easy question; sdists so far are implementation-defined, and I don’t remember if distutils supported symlinks in any way.

[takluyver]

I'm guessing it works in some way with distutils/setuptools, or archinstall wouldn't be arranged that way. Of course, that doesn't necessarily mean it's supported. 🙂

[takluyver]

This was raised again in #683.

I might in the future explicitly disallow symlinks, i.e. error on trying to build sdists or wheels from source files containing a symlink. Symlinking to something outside the project directory could be (part of) a security issue - e.g. if I craft an sdist or git repo containing a symlink and get someone else's system to build a wheel from it, it might grab a sensitive file like an SSH private key. I'm not sure there's a compelling use case for symlinking one location inside the project to another. Flit also doesn't try to cover every possible use case, so even if there are rare scenarios where one wants symlinks, it may still be out of scope.

This would be in Flit 4.0, just in case someone is doing something with symlinks that currently works, despite the lack of deliberate support.