Speed up pip list --outdated

[CrafterKolyan]

Fixes #7964

This thing is working way too slow now. The problem is that it fetches all versions in one thread.

[McSinyx]

I suggest to be a bit lazier, e.g.

something like this

    def iter_packages_latest_infos(self, packages, options):
        with self._build_session(options) as session:
            finder = self._build_package_finder(options, session)

            def latest_infos(dist):
                typ = 'unknown'
                all_candidates = finder.find_all_candidates(dist.key)
                if not options.pre:
                    # Remove prereleases
                    all_candidates = [candidate for candidate in all_candidates
                                      if not candidate.version.is_prerelease]

                evaluator = finder.make_candidate_evaluator(
                    project_name=dist.project_name,
                )
                best_candidate = evaluator.sort_best_candidate(all_candidates)
                if best_candidate is None:
                    return None

                remote_version = best_candidate.version
                if best_candidate.link.is_wheel:
                    typ = 'wheel'
                else:
                    typ = 'sdist'
                # This is dirty but makes the rest of the code much cleaner
                dist.latest_version = remote_version
                dist.latest_filetype = typ
                return dist

            pool = Pool(DEFAULT_POOLSIZE)
            for dist in pool.imap_unordered(latest_infos, packages):
                if dist is not None:
                    yield dist

We still need to consider the pool size and the chunk size and whether to optionize them.

[McSinyx]

I don't think we need to close/join the pool though, since after the iterations all jobs are finished.

[CrafterKolyan]

I don't think we need to close/join the pool though, since after the iterations all jobs are finished.

Even though the jobs are done it's better to clean all resources. For example there are extra threads which may become zombie and unaccessible. Also there are not only worker threads but some extra utility threads.

https://github.com/python/cpython/blob/ea9e240aa02372440be8024acb110371f69c9d41/Lib/multiprocessing/pool.py#L662-L666

And I had one test failed without this closing code.

[pfmoore]

I'm concerned about the complexity vs benefit tradeoff here. Adding multithreaded code adds a significant possibility for extra failures/issues, and you haven't really given any details as to what the time savings are like. Can you give some examples of before/after timings (preferably with examples that others could reproduce)? I have never personally found pip list --outdated too slow, so I'm lukewarm about this change.

What do other @pypa/pip-committers think?

[CrafterKolyan]

@pfmoore Here is reproducible MINIMAL example. You need python3, pip and virtualenv installed.

python3 -m venv venv
source venv/bin/activate
pip install -U setuptools
pip install -U pip
pip install numpy scipy pandas matplotlib seaborn statsmodels sklearn jupyter tqdm requests six
git clone https://github.com/CrafterKolyan/pip.git --depth=1 --branch=patch-1
cd pip
time pip list --outdated
pip install .
time pip list --outdated
cd .. && deactivate

Cleaning:

rm -rf venv pip

Output example:

Here we have only about 67 packages installed with just a few popular Python packages and this fix already saves 2 seconds of real time. In Data Science problems I usually have at least 100 packages installed and I get about two times boost with this fix. Also I don't see any problem in multi-threading programs especially parts with requesting something from the Internet.

[CrafterKolyan]

Some real results (no toy problem like in my example) from the person I know (and who wanted to stay anonymized):

All text is saved in original language. If you want I can translate it for you into English.

[nikitabobko]

Also I believe that all performance fixes should be documented right in the code via comment. So if it's going to be decided that performance is more important in this performance vs simplicity tradeoff then it's worth to sum up @CrafterKolyan benchmarks and write them into comment (or at least in commit message)

[CrafterKolyan]

One more example from Mac OS X:
Before:

After:

[CrafterKolyan]

@nikitabobko I've added comment in code and also created an issue with example which is referenced in pull request name and eventually will be in commit name.

[McSinyx]

I can also confirm the speedup on my Debian testing system with 286 Python packages installed:

On master

real	0m36.089s
user	0m15.676s
sys	0m0.306s

On this branch

real	0m11.451s
user	0m11.653s
sys	0m0.369s

@CrafterKolyan, could you please switch back to multiprocessing.dummy.Pool? multiprocessing.pool.ThreadPool is an undocumented feature. Also, to close an issue when this gets merged, the fix keyword needs to be in the PR's description (not title).

[uranusjr]

This will need a test to ensure the indentation thread safty thing is not accidentally broken in the future.

[CrafterKolyan]

@uranusjr added two tests. Actually they are more about working the same in created threads as in main thread of IndentingFormatter. But they are failing without this fix. Or should I test get_indentation and indent_log as independent functions (without any context for what they are actually were written) and think of some tests where some race conditions may happen?

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[pradyunsg]

LGTM.

With my 20.1 release manager hat on, this can be included in this month's release, if once the news fragment is updated.

I'll wait for a few days to let other @pypa/pip-committers chime in. :)

[pradyunsg]

Thanks @CrafterKolyan for the PR and for the prompt responses to review comments. :)

[xavfernandez]

I'm not totally confident that pip's codebase is thread safe but this seems simple enough 👍

[pradyunsg]

Yea I'm OK because this seems to not be touching anything other than logging and the specific code at hand.

[pradyunsg]

Looks like at least our download progress indicator isn't thread safe.

https://github.com/pradyunsg/pip/pull/5/checks?check_run_id=579142888#step:6:2757

[CrafterKolyan]

But this code doesn't use any progress bars

[pfmoore]

But this code doesn't use any progress bars

Understood. But you had to patch logging to fix thread safety. This PR could mean that future changes (for example, adding a progress bar to pip list!) need to do further thread safety patching in the same way.

The fact is, that we don't guarantee pip's code is thread safe. Thread safety has always been an unnecessary maintenance burden to require, given that pip doesn't use threads itself, This PR changes that, meaning that we now do have to guarantee thread safety, at least for the parts of pip's code called by this command.

As I said above, I'm concerned about the cost vs benefits of this change. It seems to me that we could be taking on a non-trivial maintenance cost to fix an issue that doesn't appear to be more than an inconvenience to a relatively small number of users.

[CrafterKolyan]

My changes are not about thread safety but about not throwing exception that _log_state doesn't have indentation field in another thread. This code should have been there before. And if it is not intended to use threads at pip (and you actually don't use them) then why are there threading.local() variables?

[McSinyx]

@CrafterKolyan, I think once upon a time pip planned to be multithreading, but then other works got in the way and such speed enhancement is not yet implemented. BTW could you please remove _log_state.indentation = 0 since IMHO it is a bit misleading?

@pfmoore, with GH-825 under consideration, I think we should gradually prepare the codebase to be thread-safe, since Python 2 support AFAIK is not going to be dropped any time soon so asyncio is not a immediate option. I applied for GSoC with parallel download as the goal and would try to make the display-relevant part thread-safe this summer, if you pip fellows accept the proposal. That being said, there are considerations over the cost of such change and I would love to be given it a try. I cannot be 100% sure that I will stick around forever to be responsible of the thread-safety, but if the project succeed, it's very likely that I will have a stronger bond with pip and be able to help for a long-term.

The above is just my opinion and I hope you can make a decision on this matter in the way that works out best for both the maintainers and the users.

[pradyunsg]

@pfmoore's concerns are reasonable and appropriate. Thinking about this a bunch more, I personally still feel that the benefits here could be worth it... at least enough to make it worth giving this a shot.

---

I think this will be a reasonable thing to include in pip 20.1-beta1 and to see if we get any user reports of problems due to this. That gives us the opportunity to communicate "hey, this is the first instance of pip actually utilizing parallelization within itself, so there may be rough edges". If we don't have any user reports of issues due to this, that's great. If we do, we'd treat it like any other regression in a new functionality -- if it's bad-enough, we'll revert the change and then decide if/how to reintroduce it.

[pfmoore]

+1 on the plan to have it available in 20.1 beta, and see if we get any issues in the wild.

[pradyunsg]

Awesome. Looks like everyone is on board with the plan, so I'll go ahead and merge this in. :)

[pradyunsg]

Thanks for the PR @CrafterKolyan! ^>^