Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up JudgeHandler's processing of problem list #328

Merged
merged 1 commit into from
Sep 19, 2023

Conversation

hieplpvip
Copy link
Contributor

Description

Type of change: improvement

What

Speed up JudgeHandler's processing of problem list

Why

OJ has more than 40k+ problems now. Judges sometimes stall waiting for bridged to process the problem list.

When test data for any problem changes, judges send the codes of ALL judge-able problems to bridged, which then maps these codes to primary keys:

judge.problems.set(Problem.objects.filter(code__in=list(self.problems.keys())))

This is extremely slow because:

  1. This retrieves all fields, not just primary keys.
  2. This creates a model instance for every returned row just to be discarded right after set runs.

Mitigated by only querying the primary keys as values list.

A proper fix for this could be caching the mapping, but it's not straightforward as problems can be renamed.

How Has This Been Tested?

On my PC, I simulate processing all problems:

_problems = Problem.objects.values_list('code', flat=True).all()
_problems = list(zip(_problems, [0] * len(_problems)))
self._problems = _problems

Previously, this takes 0.351s. After this patch, it only takes 0.129s :)

Be aware that my PC is faster than our server, and I only run 1 judge instead of 4. The processing time would be further multiplied by the number of judges.

Checklist

  • I have explained the purpose of this PR.
  • I have performed a self-review of my own code

By submitting this pull request, I confirm that my contribution is made under the terms of the AGPL-3.0 License.

@hieplpvip hieplpvip force-pushed the hiep/speed-up-bridged branch 2 times, most recently from 3b05f5b to 0376c38 Compare September 18, 2023 07:58
@hieplpvip hieplpvip force-pushed the hiep/speed-up-bridged branch from 0376c38 to 54a5282 Compare September 19, 2023 07:03
@hieplpvip hieplpvip merged commit a860684 into master Sep 19, 2023
@hieplpvip hieplpvip deleted the hiep/speed-up-bridged branch September 19, 2023 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant