FEATURE REQUEST: RabbitMQ capabilities exploration and configuration

[smokeyScraper]

🌟 Feature Description

Explore RabbitMQ capabilities and implement a production-ready Event Bus System for Devr.AI, building upon the existing event-driven architecture to support reliable message queuing, event distribution, and inter-agent communication.

🔍 Problem Statement

Current limitations:

• In-memory event handling lacks persistence and reliability
• No message queuing for handling high loads or failures
• Missing retry mechanisms and dead letter queues
• No support for distributed agent deployment
• Limited event routing capabilities
• Lack of event replay and audit trail

🎯 Expected Outcome

• Robust event bus with RabbitMQ integration
• Event serialization with schema validation
• Reliable message delivery with retry logic
• Dead letter queue handling
• Event routing based on type and platform
• Support for agent scaling and distribution
• Event monitoring and debugging capabilities

📷 Screenshots and Design Ideas

To be added later.
Current focus on assessing capabilities and designing a system aligned for agentic interactions.

Record

• I agree to follow this project's Code of Conduct
• I want to work on implementing this feature

[ShivamMenda]

Hi can I contribute/ research on this?

[smokeyScraper]

Hi @ShivamMenda,
Thanks for taking an interest in this issue!

Basically, this issue is based on a standalone part of the codebase, so yeah could be taken up and worked on easily without any intervention from the rest of the codebase. Preferably, this scopes out everything needed to be done to migrate backend/app/core/orchestration to preferably a locally hosted message queue.
The current orchestration logic is not maintained properly and uses Async Queues for scheduling the messages (entirely vibe code fr xD)

I haven't documented this feature yet, but I have a few designs related to this in my proposal, which I can share with you.

PS: You can refer to Docs, would be a good read. Probably, can scope out everything that makes the most sense of the AMQP protocols and our system. Refer Notion too.

[ShivamMenda]

Hi @smokeyScraper I did take a look at the queue_manager.py and I see the logic written, good for now but getting a queue setup will be a better way to keep messages in queue and makes sure that the agent is not overburdened. Let me setup a POC where we can send messages and see if it is coming up in a queue correctly

[smokeyScraper]

Yeah, that would be awesome @ShivamMenda. POC would be good to go and can be populated later on.
PS: current code for orchestration/ is majorly vibe-coded as it had to be changed so I didn't write that lol :p