Fallback to default vcs ref and determine package name from the pip line where possible

[matteius]

• Fallback to default vcs ref when no ref is supplied
• More proactively determine package name from the pip line where possible, fallback to the existing file scanning logics when unable to determine name.

The issue

There are so many edge cases of trying to get the package name from setup.py and surprisingly many packages still rely on it. A great example is requests: https://github.com/psf/requests/blob/main/setup.py

In the requests setup.py example, the name comes from reading another file on the filesystem, and the ast parser will never be able to accomplish that. Prior to this change, it was not possible to install the requests vcs into the Pipfile without manually creating the Pipfile entry with the correct package name and locking.

Fixes issue #5857

The fix

Allow not supplying a default ref (falls back to repository default thanks to pip internals)
Allows user to specify using the new style pip line name @ syntax the package name:

Compare these two examples:
1.) Not supplying a name in the install (works as before):

$ pipenv  install "git+https://github.com/requests/requests.git"
Installing git+https://github.com/requests/requests.git...
Resolving git+https://github.com/requests/requests.git...
[    ] Installing...INFO:pipenv.patched.pip._internal.vcs.git:Cloning https://github.com/requests/requests.git to c:\users\matte\appdata\local\temp\tmpkedl0h8p
INFO:pip.subprocessor:Running command git clone --filter=blob:none https://github.com/requests/requests.git 'C:\Users\matte\AppData\Local\Temp\tmpkedl0h8p'
[    ] Installing None...INFO:pip.subprocessor:Cloning into 'C:\Users\matte\AppData\Local\Temp\tmpkedl0h8p'...
[    ] Installing None...INFO:pip.subprocessor:Updating files:  47% (48/101)
INFO:pip.subprocessor:Updating files:  48% (49/101)
INFO:pip.subprocessor:Updating files:  49% (50/101)
INFO:pip.subprocessor:Updating files:  50% (51/101)
INFO:pip.subprocessor:Updating files:  51% (52/101)
INFO:pip.subprocessor:Updating files:  52% (53/101)
INFO:pip.subprocessor:Updating files:  53% (54/101)
INFO:pip.subprocessor:Updating files:  54% (55/101)
INFO:pip.subprocessor:Updating files:  55% (56/101)
INFO:pip.subprocessor:Updating files:  56% (57/101)
INFO:pip.subprocessor:Updating files:  57% (58/101)
INFO:pip.subprocessor:Updating files:  58% (59/101)
INFO:pip.subprocessor:Updating files:  59% (60/101)
INFO:pip.subprocessor:Updating files:  60% (61/101)
INFO:pip.subprocessor:Updating files:  61% (62/101)
INFO:pip.subprocessor:Updating files:  62% (63/101)
INFO:pip.subprocessor:Updating files:  63% (64/101)
INFO:pip.subprocessor:Updating files:  64% (65/101)
INFO:pip.subprocessor:Updating files:  65% (66/101)
INFO:pip.subprocessor:Updating files:  66% (67/101)
INFO:pip.subprocessor:Updating files:  67% (68/101)
INFO:pip.subprocessor:Updating files:  68% (69/101)
INFO:pip.subprocessor:Updating files:  69% (70/101)
INFO:pip.subprocessor:Updating files:  70% (71/101)
INFO:pip.subprocessor:Updating files:  71% (72/101)
INFO:pip.subprocessor:Updating files:  72% (73/101)
INFO:pip.subprocessor:Updating files:  73% (74/101)
INFO:pip.subprocessor:Updating files:  74% (75/101)
INFO:pip.subprocessor:Updating files:  75% (76/101)
INFO:pip.subprocessor:Updating files:  76% (77/101)
INFO:pip.subprocessor:Updating files:  77% (78/101)
INFO:pip.subprocessor:Updating files:  78% (79/101)
INFO:pip.subprocessor:Updating files:  79% (80/101)
INFO:pip.subprocessor:Updating files:  80% (81/101)
INFO:pip.subprocessor:Updating files:  81% (82/101)
INFO:pip.subprocessor:Updating files:  82% (83/101)
INFO:pip.subprocessor:Updating files:  83% (84/101)
INFO:pip.subprocessor:Updating files:  84% (85/101)
INFO:pip.subprocessor:Updating files:  85% (86/101)
INFO:pip.subprocessor:Updating files:  86% (87/101)
INFO:pip.subprocessor:Updating files:  87% (88/101)
INFO:pip.subprocessor:Updating files:  88% (89/101)
INFO:pip.subprocessor:Updating files:  89% (90/101)
INFO:pip.subprocessor:Updating files:  90% (91/101)
INFO:pip.subprocessor:Updating files:  91% (92/101)
INFO:pip.subprocessor:Updating files:  92% (93/101)
INFO:pip.subprocessor:Updating files:  93% (94/101)
INFO:pip.subprocessor:Updating files:  94% (95/101)
INFO:pip.subprocessor:Updating files:  95% (96/101)
INFO:pip.subprocessor:Updating files:  96% (97/101)
INFO:pip.subprocessor:Updating files:  97% (98/101)
INFO:pip.subprocessor:Updating files:  98% (99/101)
INFO:pip.subprocessor:Updating files:  99% (100/101)
INFO:pip.subprocessor:Updating files: 100% (101/101)
INFO:pip.subprocessor:Updating files: 100% (101/101), done.
[   =] Installing None...INFO:pipenv.patched.pip._internal.vcs.git:Resolved https://github.com/requests/requests.git to commit 881281250f74549f560408e5546d95a8cd73ce28
Added --title-- to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock (120f0d) out of date, updating to (3873be)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Locking Failed!
[    ] Locking...False
Usage: resolver.py [options]
Traceback (most recent call last):
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1480, in _process_long_opt
    opt = self._match_long_opt(opt)
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1465, in _match_long_opt
    return _match_abbrev(opt, self._long_opt)
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1670, in _match_abbrev
    raise BadOptionError(s)
optparse.BadOptionError: no such option: --title--@
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 377, in _parse_file
    args_str, opts = self._line_parser(line)
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 409, in parse_line
    opts, _ = parser.parse_args(options, defaults)
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1389, in parse_args
    self.error(str(err))
  File "c:\users\matte\.pyenv\pyenv-win\versions\3.9.13\lib\optparse.py", line 1569, in error
    self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 452, in parser_exit
    raise OptionParsingError(msg)
pipenv.patched.pip._internal.req.req_file.OptionParsingError: resolver.py: error: no such option: --title--@
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\matte\Projects\pipenv\pipenv\resolver.py", line 675, in <module>
    main()
  File "C:\Users\matte\Projects\pipenv\pipenv\resolver.py", line 661, in main
    _main(
  File "C:\Users\matte\Projects\pipenv\pipenv\resolver.py", line 645, in _main
    resolve_packages(
  File "C:\Users\matte\Projects\pipenv\pipenv\resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
  File "C:\Users\matte\Projects\pipenv\pipenv\resolver.py", line 592, in resolve
    return resolve_deps(
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 897, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 670, in actually_resolve_deps
    resolver.resolve()
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 442, in resolve
    constraints = self.constraints
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 400, in constraints
    possible_constraints_list = self.possible_constraints
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 387, in possible_constraints
    possible_constraints_list = [
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 387, in <listcomp>
    possible_constraints_list = [
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 148, in parse_requirements
    for parsed_line in parser.parse(filename, constraint):
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 335, in parse
    yield from self._parse_and_recurse(filename, constraint)
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 340, in _parse_and_recurse
    for line in self._parse_file(filename, constraint):
  File "C:\Users\matte\Projects\pipenv\pipenv\patched\pip\_internal\req\req_file.py", line 381, in _parse_file
    raise RequirementsFileParseError(msg)
pipenv.patched.pip._internal.exceptions.RequirementsFileParseError: Invalid requirement: --title--@ git+https://github.com/requests/requests.git
resolver.py: error: no such option: --title--@

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\matte\AppData\Local\Programs\Python\Python311\Scripts\pipenv.exe\__main__.py", line 7, in <module>
    # when invoked as python -m pip <command>
                 ^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\vendor\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\matte\Projects\pipenv\pipenv\routines\install.py", line 297, in do_install
    raise e
  File "C:\Users\matte\Projects\pipenv\pipenv\routines\install.py", line 281, in do_install
    do_init(
  File "C:\Users\matte\Projects\pipenv\pipenv\routines\install.py", line 648, in do_init
    do_lock(
  File "C:\Users\matte\Projects\pipenv\pipenv\routines\lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 838, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\matte\Projects\pipenv\pipenv\utils\resolver.py", line 707, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

2.) Providing the name of the install:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/vcs_reqs3
$ pipenv  install "requests@ git+https://github.com/requests/requests.git"
Installing requests@ git+https://github.com/requests/requests.git...
Resolving requests@ git+https://github.com/requests/requests.git...
Added requests to Pipfile's [packages] ...
Installation Succeeded
Installing dependencies from Pipfile.lock (120f0d)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

The checklist

• Associated issue
• A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

[oz123]

I want to cry---

name=about["__title__"],

Such a thing, a very essential python package, which has not been removed in 6 (!!!) years...
Someone should just make PR to convert reuests to use pyproject....

[oz123]

As menition in my other comment to PR #5919 I would like to add a test case for this.
Now, I know that requests is already, inlcuded in many cases, but if they fix their setup then how do we know (with out browsing through past disscussion on PRs) why this code path was added?

If there is no test case, can we at least add a docsting explaining what is this all about?

[matteius]

Ill add tests in #5853 and a doc string now.