Significant number of uplink packets are not being transmitted

[CanEmreUnal]

We are experiencing a noticeable decrease in packet arrivals to our ChirpStack LNS over the last three days. Specifically, the packet reception rate has dropped by approximately 1/3 compared to previous levels.

Here is the current configuration for reference:

{
  "org": {
    "oui": 26,
    "owner": "13Js6uZSrSU9bnKeGGw6YM91j4rWEhvBuNkQj7evKHT1XiTsM47",
    "payer": "112tbU2H7bwDrRPHK8vGpC1VnVHBg914WKenEbhjiyZZHg8qEJgr",
    "delegate_keys": [
      "112tbU2H7bwDrRPHK8vGpC1VnVHBg914WKenEbhjiyZZHg8qEJgr"
    ],
    "locked": false
  },
  "net_id": "000024",
  "devaddr_constraints": [
    {
      "start_addr": "480008D8",
      "end_addr": "480008DF"
    }
  ]
}

[macpie]

Could you provide your ROUTE_ID? As well as location (US/EU) ?

[CanEmreUnal]

ae92ae20-8872-11ee-90ee-5369153d8ed3, server location is in EU,

…

On Fri, Dec 20, 2024 at 11:06 PM macpie ***@***.***> wrote: Could you provide your ROUTE ID ? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG36QKDM5AKWYXD5BZ3XJD2GR2CPAVCNFSM6AAAAABT63USDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJXGY2DQOBWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[macpie]

Checking

[macpie]

I think it should be fixed now, for some reason the Config Service did not provide your devaddr_range ...

[CanEmreUnal]

Is it something that should take time ? Unfortunately, I didn't notice any progress yet.

…

On Fri, Dec 20, 2024 at 11:12 PM macpie ***@***.***> wrote: I think it should be fixed now, for some reason the Config Service did not provide your devaddr_range ... — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG36QJ4XFGM5MR5SWMODLT2GR2ZRAVCNFSM6AAAAABT63USDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJXGY2TKOBRGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[macpie]

It should be immediate, but your devices will need to transmit of course.

[CanEmreUnal]

We are certain that the devices with DevEUI 0016C001F0007036 and 0016C001F0007010 are transmitting. (we're seeing in packet-forwarder logs) These represent just a small portion of the devices, but we are still not receiving uplinks from them. Could you please check on your end? Could it possibly be caused by waf blocking?

…

On Fri, Dec 20, 2024 at 11:25 PM macpie ***@***.***> wrote: I should be immediate, but your devices will need to transmit of course. — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG36QNC6GLVCYVFBZRVTQ32GR4LNAVCNFSM6AAAAABT63USDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJXGY3TENBZGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[macpie]

I just ran an update on all the servers and I can confirm that your route is everywhere now (just in case the hotspot transmitting your device's data was routed threw another server than one in the EU).

If data is still not showing up soonish, I can try to track your data:

1. Uplink via devaddr.
2. Joins via dev EUI ( DO NOT POST your APP EUI).
3. Or via a hotspot name that the device usually goes threw.

[CanEmreUnal]

It seems like they are gradually coming back online. Thank you for your efforts.
However, we are still wondering about the root cause and whether this issue might recur in the future.

[macpie]

This is most likely a disconnect between HPR and Config Service (that holds all the routing infos). I have seen this happen once or twice.

I will keep this open and try to see if we can at least add metrics around this specific issue (missing devaddr range in route) and see if we can fix this by retrying in case we don't get one (range).

Note that this is an infrequent hedge case but nonetheless annoying and that should be fixed.

[pangguanyu]

We are facing similar issue now. Could you please take a look? Uplink log confirmed successfully to router on Gateway, but not sent to chirpstack.BTW, we have 2 sensors. We are testing them together. 1 is sending data every 20s, another one is sending every 5minutes/30minutes. If we keep the 20s sensor online, everything looks good. But if we stop sending data from it, the sensor sending data every 5min/30min will have data loss. @macpie

[pangguanyu]

e6410b10-e2a3-4f70-bb18-7045c5ade0cd

[macpie]

We are facing similar issue now. Could you please take a look? Uplink log confirmed successfully to router on Gateway, but not sent to chirpstack.BTW, we have 2 sensors. We are testing them together. 1 is sending data every 20s, another one is sending every 5minutes/30minutes. If we keep the 20s sensor online, everything looks good. But if we stop sending data from it, the sensor sending data every 5min/30min will have data loss. @macpie

How do you stop the 1st device from sending? It sounds like to me that when you disable that device you also disable the 2nd one.

FYI: I cannot debug anything from a LNS' device ID. Please see #308 (comment)