Responses to a query are sometimes not received

[robojan]

Describe the bug

Sometimes the responses to queries are not received, when this happens the following message is logged in the zenoh log:

rx-1 ThreadId(07) zenoh::net::routing::dispatcher::queries: Route reply Face{1, 2b2c6100603ba8e1238a98fda14685ce}:12 from Face{1, 2b2c6100603ba8e1238a98fda14685ce}: Query not found!

After doing some digging it seems to be related to reordering of packets. Initially I was using UDP, where I had this problem a lot more often, but this problem is still present when using TCP only less often.
Normally the order of network packets is request -> response -> responseFinal, but when it goes wrong the order is request -> responseFinal -> response

Attached is a wireshark trace and the corresponding log. Note that I am using zenoh here with TCP and UDP transports on port 7200.
zenoh_reorder_filtered.txt
zenoh_reorder_filtered.pcapng.gz

To reproduce

I have a reliable way to trigger this, but I cannot easily share it.
But my situation is as follows.

• I have a zenoh peer that has 16 publishers with publication cache.
• Then I have another zenoh peer that subscribes on all 16 publishers with a querying subscriber
• When the second peer joins it makes a lot of requests to get the latest data, here sometimes a query fails. On my x86 machine I see it almost never fail, but on the jetson it fails about 50% of the runs.

System info

• Platform: Nvidia jetson Orin NX
• CPU: 8 core ARM64
• Zenoh version: 1.1.0