Data delivery issues

[pavloburykh]

Currently, we are facing data delivery issues.

Here is an example caught in one of the PRs:
on video below you can observe that messages sent from User A to User B (and vica-verse) are not delivered until app re-login.

There are no clear steps for reproduction, it happens randomly.

messages_not_delivered1.mp4

User_Android.zip
User_IOS.zip

Also, part of e2e tests are constantly failing because of delivery issues.

Additional Information

• Status version: nightly
• Operating System: Android, iOS

[cammellos]

We have all the information on the devices, I have checked, so no status-go changes are necessary.
We are not tracking envelope ids on the fleet. This will require a change in nim-waku and a deployment, I will take a look at it.

[cammellos]

Checking some logs, so far i have:

Message was sent by device A:

t=2023-06-28T09:06:37+0000 lvl=info msg="publishing message via relay" envelopeHash=0xcc314453d9ddfd6ea8ee51d84bdb46ccc4832452fb9c83ba874dfc7cc178bef6

It was relayed by the node:

docker.log:2023-06-28T09:06:36.543412+00:00 docker/nim-waku[865]: TRC 2023-06-28 09:06:36.543+00:00 waku.relay received                        topics="waku node" tid=1 file=waku_node.nim:239 peerId=16U*TCnsXV pubsubTopic=/waku/2/default-waku/proto hash=0xcc314453d9ddfd6ea8ee51d84bdb46ccc4832452fb9c83ba874dfc7cc178bef6 receivedTime=1687943196543115008 payloadSizeBytes=1052

But it only made it to the other device after pulling it from the store node, ~20 minutes later (after relogin most likel):

t=2023-06-28T09:24:29+0000 lvl=info msg="received waku2 store message" envelopeHash=0xcc314453d9ddfd6ea8ee51d84bdb46ccc4832452fb9c83ba874dfc7cc178bef6

So it was dropped/not forwarded to the client for some reason.

[cammellos]

The receiver peer had very low connectivity:
t=2023-06-28T10:00:17+0000 lvl=dbug msg="connected peers" connected=[{UniqueID:16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D}]

as it wasn't able to discover any peers:

t=2023-06-28T08:19:19+0000 lvl=warn msg="dns discovery error " error="lookup prod.nodes.status.im on [::1]:53: read udp [::1]:52156->[::1]:53: read: connection refused"
t=2023-06-28T08:19:19+0000 lvl=warn msg="failed to restart discv5" error="failed to fetch bootnodes"

[cammellos]

same for the sender:

t=2023-06-28T09:16:28+0000 lvl=warn msg="dns discovery error " error="lookup prod.nodes.status.im on [::1]:53: read udp [::1]:49807->[::1]:53: read: connection refused"
t=2023-06-28T09:16:28+0000 lvl=warn msg="failed to restart discv5" error="failed to fetch bootnodes"

[cammellos]

Closing this issue for now, as that particular error should be fixed, feel free to re-open or open a new one if the issue persist (it's likely that there are more issues :) )