"pip search" should use PackageFinder, not XML-RPC API

[carljm]

Using the XML-RPC API has several disadvantages:

• It lies. It tells you what releases are registered at PyPI, but if those releases have no sdists uploaded (or if there are linked sdists for newer releases than those actually registered at PyPI), it doesn't actually tell you what release will be installed by "pip install." If "search" used the same method for finding a package as "install", this would not be a problem.
• Related to the first; you can't incorporate --find-links or --extra-index-url, etc with a "pip search".
• PyPI mirrors do not support the XML-RPC API, so "pip search" is less robust than the other parts of pip by depending on it.

[ogirardot]

as #368 will be fixed by this ticket, it's a +1

[ogirardot]

+2 via #207 and #243 (both duplicates)

[equake]

any news about this issue? i'm having to work with pip behind a firewalled machine and it's not working...

[ogirardot]

what's not working "pip search" or "pip install" ?

2012/2/22 equake <
reply@reply.github.com

any news about this issue? i'm having to work with pip behind a firewalled
machine and it's not working...

---

Reply to this email directly or view it on GitHub:
#395 (comment)

[eproxus]

Any progress on this? Pip still doesn't allow search behind a proxy even though the --proxy flag is given (which works correctly when using install, for example).

[carljm]

I'm not aware that anyone is actively working on this. I don't have any plans to work on it myself, though I'd certainly collaborate and review the work of anyone motivated to tackle it.

[TiagoTT]

This bug is still true with pip 1.4.1.

[Ivoz]

I think this can be closed now, #1902 solves the proxy issue; if we wish to use a different method to return search results for perhaps the new Warehouse PyPI infra then a new issue is probably best to discuss on a clean slate.

[sergray]

The description of the ticket is about disadvantages of using XML-RPC API in pip search, but it somehow became mixed with the proxy problems, which are not related to the original message

[dstufft]

Reopening, I still think it is valid to use the PackageFinder instead of XMLRPC even if the proxy issue is fixed.

[pfmoore]

AIUI, PackageFinder wouldn't include a description whereas search does. Certainly getting a usable set of results with no description is better than getting irrelevant results with descriptions, but in an ideal world, both would be better :-)

[pradyunsg]

@brainwane @di FYI.

IMO this would need to be fixed before Warehouse disables XML-RPC APIs.

[dstufft]

Or we'd need to expose a new search API that isn't built on top of XML-RPC.

[nussbrot]

Hi!

I'm having an issue with pip search and landed here on detours.
A short description of my problem:
I have my own PyPI server hosting some Python packages and i configured this server in pip.conf with the extra-index-url so that i can use pip install <pkg_on_my_server> <pkg_on_pypi> and it just works.

I wanted to do the same with pip search, but I can't. I have to either search PyPI or my server, which is tedious.
I was redirected by Stackoverflow to issue #332 and from there to this issue.

This issue is very old, so I'm not sure if there is anything going on to solve this problem or if you have planned to solve this problem.

So in the meantime: do you know a workaround to this issue? I suspected more people having such problems, but I could not find solutions.

[n0npax]

Hi,
I check the code and fixing this issue may be not trivial due to following issues:

• PackageFinder does not provide any option to search for packages. Implementing new function require using old deprecated api.
  ** XML-RPC API will be marked as deprecated https://warehouse.pypa.io/api-reference/xml-rpc/
  ** Json api doesn't provide an option to search packages https://warehouse.pypa.io/api-reference/json/

I believe this issue should be marked as blocked by pypi/warehouse#3436

[cjerdonek]

Can someone sketch what it would mean to use PackageFinder for search functionality (i.e. what an implementation might look like), and what the tricky parts might be and how much work?

[uranusjr]

Since this is brought up again (latest issue reference), here are my unorganised thoughts on this:

• PackageFinder is based on several formats, mainly PEP 503 and find-links.
• The latter does not really fit into pip search since it is artifact-based, not project-based.
  • Theoretically it is possible to group artifacts into projects and generate a similar format, but there are too many edge cases (e.g. multiple versions, different artifacts have different metadata) to make this worthwhile IMO. Since pip search does not currently support find-links anyway I think it is reasonable to punt this until someone brings it up (if ever).
  • So, focusing on PEP 503.
• Currently pip search would 1. fetch a list of packages matching the query, and 2. display each package’s name, latest version, and description (likely from the latest version).
• The fetching part is possible, although maybe inefficient. PEP 503 has a package listing page (pip search only considers the package name IIUC), but it would be a performance hit to download and parse the whole package list (pypi.org/simple is about 10 MB text).
• But the display part is trickier. PEP 503 (and thus PackageFinder) is only capable of providing the name and latest version. It is also theoretically possible to collect the description from PEP 503 (by downloading the selected artifact), but that is way too costly IMO.
• One (reasonable IMO) compromise is to extend PEP 503 to include this information. This can be done by adding an optional text block (say <div id="meta_description"></div>). Index servers can choose to implement it, or pip would simply display an empty string if the block is not found. The needed Warehouse implementation is hopefully not too difficult, since the same information already presents in the JSON API.
• Or… maybe we should just standardise the JSON API and use that instead? This has also been raised multiple times, but I am not sure what the current consensus on it is, if any.

[pfmoore]

The fetching part is easy

Is it? Fetching the packages that match the query is the requirement. Currently queries search in both the package name and the description (and possibly elsewhere - the search functionality is a bit weird!). I'm not sure replicating that would be "easy". OTOH, if we redefine search to just search on package name, it should be OK.

One (reasonable IMO) compromise is to extend PEP 503 to include this information

-1 on this. The simple repository API should remain just that - simple. Being able to have a web server just serve a static directory using its default functionality is a key feature of the API. While extensions like this are optional, the more we add, the more we risk getting people complaining that things don't work when actually they rely on optional features which simply aren't present.

Or… maybe we should just standardise the JSON API and use that instead?

That sounds like a better option. We're going to need to do this at some point when we expose static metadata from indexes, so why not start now by standardising the JSON API, and a means to query a repository to confirm what version (if any) of the JSON API it supports? I don't know if tools like devpi support the JSON API at the moment, but I'd expect them to be more likely to do so if it were standardised.

[gutsytechster]

I have a few questions regarding this.

1. Is the PackageFinder mentioned in above discussions refer to this here https://github.com/pypa/pip/blob/master/src/pip/_internal/index/package_finder.py#L597?
2. As currently, the Warehouse JSON API doesn't support search functionality, how can we use that to standardise the result?

[pradyunsg]

Honestly, I'd much rather we remove the command entirely: #5216

[pradyunsg]

Closing this since I don't think we can implement this as-is.

Or we'd need to expose a new search API that isn't built on top of XML-RPC.

That would be pypi/warehouse#284 -- we can revisit this if/when that API is implemented.