pip install with target doesn't work for optional sub-packages

[matham]

• Pip version: 9.0.1
• Python version: 2.7-x64
• Operating System: Win10

Currently, when using the target option, --target, if a directory/file already exists and would be overwritten it errors out requesting the you use upgrade. However, that is impossible when trying to install an optional sub-package to a package that was also installed to a different location with --target.

Imagine you have the package foo installed and it has its __init__.py and other modules. Now I would like to install foo.bar to foo (i.e. sub-package bar in the foo directory). When installing to site-packages, like normal, it works and pip doesn't care that foo directory already exists (in fact it doesn't seem to care even if foo.bar exists as long as the *.dist_info for foo.bar is removed). The point is that pip is happy to overwrite files.

Now imagine foo was installed elsewhere with --target. Now I'd like to install foo.bar also to the other location. But now pip errors out saying Target directory foo already exists. Specify --upgrade to force replacement. --upgrade would not work here is at would just delete the previous contents of foo messing up the main package.

This change happened in #2007, while I see that there were issues that required this, these changes made sub-packages impossible with --target.

Would perhaps a --overwrite option be in order? And maybe the --overwrite option should be automatically set to be in line with current pip behavior when installing to site-packages?

[lmazuel]

+1 got the same problem and was looking to open an issue and found this one.

This is EXACTLY our situation.

FYI @brettcannon @annatisch

[pradyunsg]

I'm not sure what we want to do here. Maybe other @pypa/pip-committers have inputs on this?

[tlelson]

Also have the exact problem.

This kind of local install is only getting more popular with increasing use of such things as AWS lambda and GCP AppEngine.

[automartin5000]

I think I have the same problem? I actually thought #2007 might fix the problem, but I guess not. I posted on SO just to confirm

[pradyunsg]

Could someone provide a reproduction case for this?

[peacepirate]

Sounds like an issue similar to what I ran into. My issue was trying to install subpackages under a common package hierarchy that already exists at the time of installation. example package names: dog.german, dog.british. I added a temporary fix for my use case: using --root [location] instead of -t to install in a predetermined location. This creates the sites-package sub directories, but that location is static and I copy over the dir from final location for further use.

[mehmetfazil]

I am currently trying to build a deployment package for AWS Lambda, with using amazonlinux image and python3-pip, having this issue as well. Has not find any workarounds yet.

[pfmoore]

As @pradyunsg said, we'll need a reproduction case for this. I'd also add that I would want to see more context on what the logic is for doing this and why it's a reasonable behaviour to rely on (descriptions involving artificial packages like foo.bar are great for explaining the problem, but not so good for explaining why you want to do things in the first place).

My inclination is to say that you shouldn't be trying to do this, as it's not how I'd expect --target to be used, which is why I want to better understand what you're trying to achieve here.

[mojganaz]

@pfmoore one usecase for this is when you'd want to ship multiple python packages to your spark application. You'd pip install with -t option into a folder. Zip the folder and pass it to spark-submit using --py-files option.

[pfmoore]

@mojganaz Thanks for the example, but I don't really see why you couldn't just pip install everything, submit the zipped folder, and then remove the folder. Next time you want to submit, do the same. What's the requirement for doing pip install -t into an existing, non-empty folder in this scenario?

Sorry if the answer is something straightforward about how spark works, I'm not at all familiar with spark, so feel free to explain in idiot-level terms 🙂

[pradyunsg]

Closing this because no one has provided a reproducer for this, and there has been no activity on this for two years now.