-
-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to set workers message treatment concurrency ? #77
Comments
Hi, PoiraudVictor! I'm not sure that I fully understand, but there's no control worker_threads or promise in celery.node/src/kombu/brokers/redis.ts Lines 138 to 150 in efc4334
Sorry for not being of much help. |
Thanks a lot actumn for your quick answer. I have probably been unclear in my first message. Actually I have 2 services, the first one (service A) sends messages to the second one (service B). When service B has finished processing a message, it sends it to a new queue (q1). Service B is listening to a queue (q0) for messages from service A. Although, if A sends a message m1 to q0 while B is still processing a previous message m0, B will process m1 and m0 in parallel. How can I achieve this with Hereafter, service B code : import { createClient as createPublisher } from "celery-node";
import { createWorker as createConsumer } from "celery-node";
const { CELERY_BROKER, CELERY_BACKEND, Q0, Q1 } = process.env;
const celeryConsumer: CeleryConsumer = createConsumer(
CELERY_BROKER,
CELERY_BACKEND,
Q0
);
const celeryPublisher: CeleryPublisher = createPublisher(
CELERY_BROKER,
CELERY_BACKEND,
Q1
);
celeryConsumer.register("some.task", async (message: CustomMessageType) => {
<...message-processing-code...>;
const task: Task = celeryPublisher.createTask("some.other.task");
task.applyAsync([someArgs]);
});
await celeryConsumer.start(); I run my code locally on both macos Monterey v12.1 and Ubuntu container. |
were you able to solve this? |
I implemented solo execution in my fork but only for redis right now, I will try to add more options next week, |
came over this issue and I have a cool solution that may be very helpful for rabbitMQ.
I don't think its the best solution but after using it after your |
Description
Hey guys, thanks for the good work on this project, it rocks.
Although, I have not been able to play with my workers message treatment concurrency.
In python you can pass the
--concurrency
flag to thecelery worker
command. How can I achieve that with celery-node ?The best scenario for me would be to set my workers message treatment concurrency to 1. I am using k8s and want to have full controle on scaling. Not being able to set concurrency to an arbitrary value goes against it.
Thanks !
The text was updated successfully, but these errors were encountered: