Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider replacing kingfisher-process with ocds-merge-rs in pipeline #292

Open
jpmckinney opened this issue Jun 12, 2023 · 1 comment
Open
Labels

Comments

@jpmckinney
Copy link
Member

I was working on a Rust rewrite of OCDS Merge (need to add a CLI and compare performance to the Python version). Once that's done, I can also explore replacing Kingfisher Process (that would mean using OCDS Kit to upgrade the few collections that are OCDS 1.0 – if that's too slow, I can write a Rust version).

Having fewer items in the queues should increase the stability of RabbitMQ. (If RabbitMQ continues to fall over, maybe we switch to Redis, where enabling its durability features won't be as much of a performance concern, as we'll just have 1 message per job task per publication).

Similar issue for data support server: open-contracting/deploy#437

@jpmckinney
Copy link
Member Author

jpmckinney commented May 6, 2024

Noting that since upgrading to yapw 0.1.0 in open-contracting/kingfisher-process@3b7dfc5 (i.e. since using async consumers, as appropriate), we haven't had RabbitMQ issues.

The main reason for a Rust version, then, is that for large collections, we end up using a lot of CPU (triggering alerts), with RabbitMQ queueing over 1M messages in the release compiler. A Rust version means we use less CPU and finish faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant