Do we need to use noDelay?

[jpwilliams]

senecajs/seneca-amqp-transport#63

Seem to be getting limited to 25 messages per second when contacting a remote Ubuntu RabbitMQ server, though setting noDelay doesn't seem to make any difference to that at all?

[jpwilliams]

Need to see if this is still an issue with a remote RabbitMQ.

[jacktuck]

I'm confident #58 is the bottleneck for RPC.

[jpwilliams]

Aye this is a separate issue though. As in, limits it to exactly 25 messages/s. :D

[tejkumar51]

Apologies if it is not the right place to post this comment.
I have just begin to use RabbitMQ as mediator for my webcrawler and I am observing the same behavior with "remote Ubuntu RabbitMQ server" (not more than 25msg/sec consumption rate - Using basic_get)
Used delivery_mode =2 for publish

1. channel.basic_publish(exchane,target_queue,message,
   pika.BasicProperties(content_type='text/plain',delivery_mode=2,headers=header_params))
2. channel.basic_get(
   queue=target_queue,
   no_ack=False) )

While I haven't seen any delivery issues in local rabbitmq server (seen 800 msg/sec for my app ).
I initially thought this is the issue with the application processing but seems it is with RabbitMQ (If I am not wrong)
You found any solutions for this ?

[jpwilliams]

@tejkumar51 No problem! Watch this thread as we'll be trying to solve it here.

Same issue here though - locally it's at lightning speed but for remote instances we're getting that nasty 25 per-second lock. Does look like it's something to do with the underlying AMQP client. What are you using to communicate with RabbitMQ?

The issue referenced at the top of this thread seems to have solved the problem using a noDelay option, but no luck here.

[tejkumar51]

Thanks @jpwilliams !
Using 'Pika' - Python AMQP Client Library, I am not sure where to update this noDelay option but with little knowledge from documentation ( https://www.rabbitmq.com/networking.html ) updated tcp buffer size and saw slight improvement (40 per-sec) but it's not considerable.

[jpwilliams]

@tejkumar51 Mm I've seen the same slight increases here. Contrary to my tests when this issue was created (9 months ago now!), the noDelay option is helping the speed some our side. Negligibly, grabbing about 31 msg/sec instead of 25, but it fluctuates and seems to be free of the lock. Still, it's severely impeded.

Looks like Pika already disables Nagle's algorithm (that line is the same as noDelay: true on our side), so it can't be that that's causing the issue.

Is this just how slow RabbitMQ can be over a network? I knew it wasn't intended to be communicated with over WAN, but the slow-down seems excessive, no?

[tejkumar51]

@jpwilliams Yes over the network I feel it has very slow rate , reference from https://stackoverflow.com/questions/20516450/ridiculously-slow-simultaneous-publish-consume-rate-with-rabbitmq I have observed one thing from my side:

• I am fetching messages one at a time but processing asynchronously (so fetching and processing phases are asynchronous) then acknowledging when I am finished - May be I need to look for consumers with prefetching count options rather than basic_get()

[jpwilliams]

@tejkumar51 Aye using basic_consume() should give you a huge performance boost if consistently consuming, so that ay well be your issue. Prefetch count can help too, but the performance gains dwindle pretty quickly. RabbitMQ have a good blog that has a good short section about prefetch count about half way through.

You could also try making sure that you have separate channels for publishing/subscribing and removing durability/persistence options should help, though usually not to a huge extent.

In regards to this lib:

• Already using consume over get
• Disabled Nagle's algorithm via noDelay: true
• Already have separate channels for publishing/subscribing (per consumer)
• Using a default prefetch count of 48
• No message persistence
• No queue durability (not in current build, but tested)

I'm starting to think that maybe RabbitMQ over WAN is just much, much slower. Seems mad that it's that bad though. I guess our problem is seeing these rates for RPC (publisher -> server -> consumer -> server -> publisher) so there's a lot of time needed for packets travelling across the WAN.

[tejkumar51]

Thanks for the info @jpwilliams !
Sure I need to check for the option of creating consumers, keeping your suggestions above in mind. (At present my messages are persistent and queue is also durable and using single channel)
but for now I am planning to run my crawler on the same server until new design implemented (with non-blocking consumers)
Thanks for your suggestions !

[jpwilliams]

No worries, @tejkumar51. I'll close this as we've fixed the issue with this lib, but feel free to contact me if you need help with anything.