Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When not to use enableAutoPipelining? #1377

Open
stavalfi opened this issue Jul 3, 2021 · 3 comments
Open

When not to use enableAutoPipelining? #1377

stavalfi opened this issue Jul 3, 2021 · 3 comments

Comments

@stavalfi
Copy link

stavalfi commented Jul 3, 2021

  1. From the docs, it sounds like there are only advantages. So why ioredis does not internally use it without any user-configurations?
  • I have a single node cluster with 1 shard (master). Maybe in my specific use case there are only advatanges for enableAutoPipelining?
  1. Also, If I send multi, will the commands inside this multi will be send in a "bigger" multi (incase there are some commands waiting in the queue)?

  2. With enableAutoPipelining=true, does ioredis send at most only 1 request at the same time?

  3. maybe the answers should be posted in the documentation.


Thanks for this super great feature!! Good job!!

@stavalfi stavalfi changed the title when not to use enableAutoPipelining? When not to use enableAutoPipelining? Jul 3, 2021
@TysonAndre
Copy link
Collaborator

TysonAndre commented Aug 1, 2021

From the docs, it sounds like there are only advantages. So why ioredis does not internally use it without any user-configurations?

Not a maintainer, but there were still some open issues, some of which were recently fixed in 4.27.7 (#1391) - maybe they're waiting for more real-world testing or a major release before making it a default

Second, if 100 commands are automatically pipelined, and there's a (EDIT: Networking, not redis) error (EDIT: and you get disconnected before the first response is sent), you don't know how many failed, which may be fine in some applications but undesirable in others (e.g. if there were 100 increment commands, you wouldn't know how many (0-100) of the increments actually succeeded or actually failed - having 100 requests not knowing if they succeeded or failed after a spurious networking error is worse (in some applications) than just 1 request with an unknown result (reconnecting then sending the other 99))

  • Autopipelining is generally the best choice for high throughput or bursts of requests to process in my experience, though
  1. If I send multi, will the commands inside this multi will be send in a "bigger" multi (incase there are some commands waiting in the queue)?

I think the multi is atomic and can't be added in a pipeline together with other commands from briefly reading the lib/autopipelining code (array of commands that forbid autopipelining). I'm not certain if that's correct.

  • I'm likely misreading that, though. Maybe the multi is excluded because they aren't actually requests yet
  1. With enableAutoPipelining=true, does ioredis send at most only 1 request at the same time?

No, that describes what happens when it's false. It sends many requests at a time without waiting for a response when it's true

@stavalfi
Copy link
Author

stavalfi commented Aug 2, 2021

Thanks for your time @TysonAndre!

if 100 commands are automatically pipelined, and there's an error, you don't know how many failed

How can it be? I will get the result-status of every one of my requests so I will know if a specific request succeed or not. Do I have any mistake?

With enableAutoPipelining=true, does ioredis send at most only 1 request at the same time?

No, that describes what happens when it's false. It sends many requests at a time without waiting for a response when it's true

I'm confused. it's the point of autopipeline to wait for more commands (while there is an ongoing request) and when there will be a response, collect all the pending requests and then go to redis only once?

It seems like I understand the opposite from your explanantion.

@TysonAndre
Copy link
Collaborator

With enableAutoPipelining=true, does ioredis send at most only 1 request at the same time?

Actually, I was confused by the name in my previous response. In both cases it's sending multiple requests at the same time before waiting for a response - i.e. https://en.wikipedia.org/wiki/Protocol_pipelining is enabled in both cases

What enableAutoPipelining is doing is grouping the multiple writes of small packets into a single stream write of a larger amount of data, making more efficient use of the network link and sending fewer total tcp packets https://github.com/luin/ioredis#autopipeautopipelininglining

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants