-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FixedCircularBuffer isFull(), isEmpty() and wrapping off by one #53139
Comments
Your example is demonstrating how buffer overflow exist. When people is pushing to a buffer stack without caring the state, then all the information will be messed up. In a proper case, the buffer should throw when pushing new data in full state. So, the For example, the below usage shows it is properly skip pushing when a node/lib/internal/fixed_queue.js Lines 99 to 117 in ff659fa
|
The expected circular buffer semantic is not to throw when full. Making it throw will follow a fixed queue semantic. As I said, the current usage is not triggering the issue. And the buggy implementation is not exported. I understand the branching less optimization for a specific use case. But the tradeoff is debatable with for example a 100k sized fixed queue. And I do not find any benchmarking code on fixed queue implementation in the repo. Before I start to test optimization in that code with appropriate benchmarking, I want to make sure the semantic issues on well defined data structure are understood. The naming fixed queue on in fact a queue using several fixed arrays for storage is also debatable. |
Reference from https://wiki.c2.com/?CircularBuffer
Despite it is expected for throwing in circular buffer or not. For the usage of
I believe it is acceptable to be The original design is date back from PR 18617, throughout the conversation. The benefit are
Note: I am not a member from Node.js, I just want to discuses about it since I am interested in the implementation too. |
A fixed queue can be implemented with a circular buffer design, at the cost of losing one 'space'. In that case, it's still called a fixed queue, not a circular buffer. The underlying design choice to implement a fixed queue semantic is not expected to change the name of the initial abstraction. My concerns are more about the performance optimization of the implementation without benchmarking the alternatives:
I plan when time permits to add a benchmark for the fixed queue. Then start testing with performance regression detection some micro optimizations. |
On that one, the current implementation is the most efficient and cannot change the size of the fixed size array without noticing.
On that one, it's not that clear, the integer binary representation significant bits size seems to impact badly bitwise operation at a first glance, when the ternary operator behaves most consistently whatever its value is. Benchmark results to come. |
Not enough FOSS time to dig further. |
Version
Not relevant
Platform
Not relevant
Subsystem
Not relevant
What steps will reproduce the bug?
No response
How often does it reproduce? Is there a required condition?
No response
What is the expected behavior? Why is that the expected behavior?
No response
What do you see instead?
Implementation copied from lib/internal/fixed_queue.js:
One way to fix it to count the number of elements in
list
atpush()
andshift()
and test it against zero or the 'list` fixed array size.It's current usage in FixedQueue does not expose the issue with wrapping when full and the resulting incorrect FIFO semantic at
shift()
because a new buffer is created. But the trade off at losing one array space each 2047 elements is debatable vs. integer property increment/decrement + branching one more.Additional information
No response
The text was updated successfully, but these errors were encountered: