Allow spaces in path requirements

[charliermarsh]

Summary

This PR modifies our parsing to allow spaces in URLs. I don't know if this is a correct change... But we now parse URLs until we see:

• A newline.
• A semicolon (marker) or hash (comment), preceded by a space. We parse the URL until the last
  non-whitespace character (inclusive).
• A semicolon (marker) or hash (comment) followed by a space. We treat this as an error, since
  the end of the URL is ambiguous (e.g., https://foo.com; marker) would be a URL that ends in ;).

Closes #6032.

[charliermarsh]

These rules are definitely somewhat dubious.

[ofek]

I haven't reviewed this in full but note that the Python packaging library requires a space before the semicolon as shown in this example: https://hatch.pypa.io/latest/config/dependency/#complex-syntax

[zanieb]

I think that matches

A semicolon (marker) or hash (comment), preceded by a space. We parse the URL until the last non-whitespace character (inclusive).

[charliermarsh]

Yes we also require that.

[charliermarsh]

(Although I don't believe pip requires that, either for PEP 508 or unnamed URL requirements.)

[ofek]

Great thanks!