Queue Size parameter does not work when using timeouts in Node.next

[chetan-parthiban]

Describe the bug
It seems that the queue_size and timeout parameters don't play nicely together. When they are used in tandem there seems to be some sort of implicit queuing that happens creating increasing delays in the node subscribing to messages.

To Reproduce
Here is a minimal example to reproduce:

dataflow:

nodes:
  - id: time_publisher
    custom:
      source: time_publisher.py
      inputs:
        keepalive: dora/timer/millis/100000
      outputs:
        - ts

  - id: time_subscriber
    custom:
      source: time_subscriber.py
      inputs:
        ts: 
          source: time_publisher/ts
          queue_size: 1

time_publisher.py:

from dora import Node
import pyarrow as pa
import time


def main() -> None:

    dora_node = Node()
    i = 0
    while True:
        dora_node.send_output("ts", pa.array([time.perf_counter_ns()]))
        i+= 1
        print(f"Sent {i} times", flush=True)
        time.sleep(0.001)



if __name__ == "__main__":
    main()

time_subscriber.py

from dora import Node
import time

def main() -> None:
    dora_node = Node()

    i = 0
    while True:
        message = dora_node.next(timeout=0.001)

        if message is None:
            break

        if message["type"] != "INPUT":
            continue
        sent = message["value"][0].as_py() // 1000
        received = time.perf_counter_ns()  // 1000

        i += 1
        print(f"[{i}] Sent: {sent}, Received: {received}, Difference: {received - sent}")
        time.sleep(0.1)



if __name__ == "__main__":
    main()

Sample logs from subscriber node:

[1] Sent: 175851788624, Received: 175851989164, Difference: 200540
[2] Sent: 175851879962, Received: 175852089537, Difference: 209575
[3] Sent: 175851881062, Received: 175852189798, Difference: 308736
[4] Sent: 175851882208, Received: 175852290081, Difference: 407873
[5] Sent: 175851883322, Received: 175852390332, Difference: 507010
[6] Sent: 175851884428, Received: 175852490576, Difference: 606148
[7] Sent: 175851885503, Received: 175852590809, Difference: 705306
[8] Sent: 175851886575, Received: 175852691048, Difference: 804473
[9] Sent: 175851887647, Received: 175852791274, Difference: 903627
[10] Sent: 175851888713, Received: 175852891510, Difference: 1002797
[11] Sent: 175851889790, Received: 175852991776, Difference: 1101986
[12] Sent: 175851890882, Received: 175853092049, Difference: 1201167
[13] Sent: 175851891960, Received: 175853192288, Difference: 1300328
[14] Sent: 175851893074, Received: 175853292526, Difference: 1399452
[15] Sent: 175851894173, Received: 175853392695, Difference: 1498522
[16] Sent: 175851895252, Received: 175853492852, Difference: 1597600
[17] Sent: 175851896374, Received: 175853593075, Difference: 1696701
[18] Sent: 175851897479, Received: 175853693273, Difference: 1795794
[19] Sent: 175851898587, Received: 175853793456, Difference: 1894869
[20] Sent: 175851899693, Received: 175853893628, Difference: 1993935
[21] Sent: 175851900800, Received: 175853993856, Difference: 2093056
[22] Sent: 175851901907, Received: 175854094144, Difference: 2192237
[23] Sent: 175851903019, Received: 175854194436, Difference: 2291417
[24] Sent: 175851904129, Received: 175854294672, Difference: 2390543

Expected behavior
The printed time differences should remain somewhat stable since the queue should just give the node the most recent message

Screenshots or Video
If applicable, add screenshots to help explain your problem.

Environments (please complete the following information):

• Running on ubuntu 24.04
• uname (GNU coreutils) 9.4
• Dora version 0.3.9

Additional context
Add any other context about the problem here.

[haixuanTao]

Hey :)

That's an interesting case!

Thanks for reporting it!

Going to create a test out of it.