Speed up unpack_file_url

[msabramo]

E.g.: pip install /path/to/dir

by building an sdist and then unpacking it instead of doing shutil.copytree. shutil.copytree may copy files that aren't included in the sdist, which means that:

1. If those files are large, a pip install could take much longer than it should.
2. Folks that are using python setup.py install to test their package are not fully testing it, because it may copy over files that wouldn't be there if they built and sdist and installed that.

So this method building an sdist is both more accurate and faster.

Fixes #2195

Cc: @tomprince, @pfmoore, @dstufft, @fungi, @rbtcollins

[pfmoore]

👍

[msabramo]

This one should probably require a few thumbs up before merging because my first attempt at this messed pbr and had to be reverted.

So I'd love for OpenStack folks to give this a try and weigh in.

[msabramo]

This could replace #2533

[emonty]

@msabramo thanks! I'll run it through some pbr paces this weekend.

[msabramo]

Thanks, @emonty! I hope this change works better for pbr than my earlier one. It would be really nice to make pip install /path/to/dist work a lot faster (and be a more realistic test of what will happen when you build an sdist and publish it).

[dstufft]

Talked to @emonty on IRC and it appears like this works for pbr now.

[Ivoz]

For posterity, atm, it looks like it was supposed to be

     if os.path.isdir(link_path):
-        if os.path.isdir(location):
-            rmtree(location)
-        shutil.copytree(link_path, location, symlinks=True)
+        _copy_dist_from_dir(link_path, location)

at Line 696

That said, when a function called unpack_file_url is using subprocess calls to execute setup.pys things get semantically confusing for me. Seems like things could use a little re-(naming/factoring). Also might have to be careful this isn't on a codepath which allows setuptools to not be installed (for wheels, for instance).

[henrykironde]

In case it helps: I compared pip and python setup.py install. I found that when I included data in the directory structure, pip would try to copy all this data. This would take over 5 minutes since my project can create lots of data when "testing under development". python setup.py install would not include the files since I defined the folders and files to include(or packages). Am wondering if pip does not clearly detect the required files in any corrupted directory structure.