-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider replacing kingfisher-process with ocds-merge-rs in pipeline #292
Labels
Comments
This was referenced Jun 12, 2023
Noting that since upgrading to yapw 0.1.0 in open-contracting/kingfisher-process@3b7dfc5 (i.e. since using async consumers, as appropriate), we haven't had RabbitMQ issues. The main reason for a Rust version, then, is that for large collections, we end up using a lot of CPU (triggering alerts), with RabbitMQ queueing over 1M messages in the release compiler. A Rust version means we use less CPU and finish faster. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was working on a Rust rewrite of OCDS Merge (need to add a CLI and compare performance to the Python version). Once that's done, I can also explore replacing Kingfisher Process (that would mean using OCDS Kit to upgrade the few collections that are OCDS 1.0 – if that's too slow, I can write a Rust version).
Having fewer items in the queues should increase the stability of RabbitMQ. (If RabbitMQ continues to fall over, maybe we switch to Redis, where enabling its durability features won't be as much of a performance concern, as we'll just have 1 message per job task per publication).
Similar issue for data support server: open-contracting/deploy#437
The text was updated successfully, but these errors were encountered: