Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LoRaWAN packet routing (HIP draft) #9

Closed
wants to merge 12 commits into from
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
301 changes: 301 additions & 0 deletions 0008-lorawan-join-routing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,301 @@
- Start Date: 2020-02-11
- HIP PR: <!-- leave this empty -->
- Tracking Issue: <!-- leave this empty -->

# Summary
[summary]: #summary
Many LoRaWAN sensors come pre-provisioned and as such AppEUI and AppKey cannot be updated. [Our current LoRaWAN implementation](https://github.com/helium/HIP/commit/8e233ac37b9c39c3083be014d7f4b25aa40b1a18) requires updating these fields for a device to work on the Helium network.

We would like to implement a scheme that takes AppEUI, AppKey, and DevEUI as fixed parameters, but still enables hotspots to route data to the proper endpoint.
This HIP proposes to add [XOR filter](https://lemire.me/blog/2019/12/19/xor-filters-faster-and-smaller-than-bloom-filters/)
based routing tables to the blockchain ledger to aid in routing LoRaWAN join
packets to the correct destination. A related HIP describes the routing of
packets once the device has joined.

# Motivation
[motivation]: #motivation

#Why are we doing this? What use cases does it support? What problems does it solve? What is the expected outcome?

With the addition of LoRaWAN support, we have inherited some of the routing
problems inherent in the LoRaWAN network model. LoRaWAN assumes that the network
has a centralized network server (or set of them) (similar to a LAN) and our model
is much closer to internet routing. Additionally, it turns out many LoRaWAN devices
do not allow end-user configuration of DevEUI/AppEUI/AppKey. Thus the existing
model of co-opting the AppEUI to route the join packet is unsuitable and
something new is needed.

# Stakeholders
[stakeholders]: #stakeholders

* LoRaWAN device users

# Detailed Explanation
[detailed-explanation]: #detailed-explanation

A XOR filter is a probabalistic data structure that allows for fast checking for
key membership It never has false negatives but can have false positives. XOR
filters come in 2 flavors, xor8 and xor16. xor8 has a false positive rate of
about 0.3 percent and xor16 has a false positive rate of about 0.002%. We
propose that the space tradeoff (2x) for xor16 is worth it because it results in
much less ambiguous routing for large numbers of devices.

XOR filters require inputs be hashed to a 64 bit integer before being stored or
compared against the XOR filter. We propose the use of
[xxhash](http://cyan4973.github.io/xxHash/)'s 64 bit mode. This is one of the
fastest hash functions available and seems to provide good distribution for the
XOR filter. Both the DevEUI and the AppEUI (totaling 128 bits together) would be
hashed to a 64 bit key, this should help in cases of DevEUI collisions.

A list of XOR filters would be attached to the OUI record in the ledger that
would contain the routing filters for that OUI. A single filter would suffice
for small OUIs, only larger OUIs whose amount of devices could exceed the
maximum filter size would require more than one routing filter.

XOR16 filters for a range of sizes are presented below for illustration purposes:

| Devices | Bytes on chain |
|-----------|----------------|
| 100 | 322 bytes |
| 1,000 | 2.48 kbytes |
| 10,000 | 24.09 kbytes |
| 100,000 | 240.21 kbytes |
| 1,000,000 | 2.34 mbytes |

With the largest XOR filter, we can check for key membership in about 14
milliseconds on an raspberry pi 3b+ in 32 bit mode. The smallest xor filter can
be checked in well under a millisecond.

In a more complete test, 10,000 OUIs holding device space for 50 million devices
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we doing anything clever here, or just checking them one by one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to do a linear scan, as far as I know.

was able route an address to its destination in 1.1 seconds, again on a
raspberry pi 3b+ in 32 bit mode. 64 bit mode is expected to be slightly faster.
The same test on a pi 4 is about 200 milliseconds faster, and the same test on
an intel i7-7700HQ runs in about 96 milliseconds.

Some more benchmarks are presented below. The memory and database sizes are
approximate.

On an Intel i7-7700HQ running Void linux:

```
running xor8_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.21Mb, max 0.51Mb, min 0.10Mb
Approximate database size 9.99Mb
Average errors 39.300, max 59, min 20
Average lookup 0.056s, max 0.150s, min 0.034s
Lookup misses 0


running xor16_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.54Mb, max 0.96Mb, min 0.23Mb
Approximate database size 23.76Mb
Average errors 0.155, max 2, min 0
Average lookup 0.077s, max 0.202s, min 0.054s
Lookup misses 0


running bloom with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.55Mb, max 1.02Mb, min 0.24Mb
Approximate database size 38.65Mb
Average errors 0.036, max 2, min 0
Average lookup 0.101s, max 0.218s, min 0.068s
Lookup misses 0
```

On a Raspberry Pi 3b+ running 32 bit Raspbian:

```
running xor8_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.12Mb, max 0.25Mb, min -0.06Mb
Approximate database size 9.99Mb
Average errors 39.300, max 59, min 20
Average lookup 0.360s, max 0.426s, min 0.278s
Lookup misses 0


running xor16_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.27Mb, max 0.84Mb, min 0.08Mb
Approximate database size 23.76Mb
Average errors 0.155, max 2, min 0
Average lookup 0.533s, max 0.629s, min 0.487s
Lookup misses 0


running bloom with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.14Mb, max 0.88Mb, min 0.02Mb
Approximate database size 38.65Mb
Average errors 0.030, max 1, min 0
Average lookup 0.700s, max 0.838s, min 0.651s
Lookup misses 1000
```

On a Raspberry Pi 4 running 32 bit Rasbian:

```
running xor8_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.15Mb, max 0.26Mb, min -0.02Mb
Approximate database size 9.99Mb
Average errors 39.300, max 59, min 20
Average lookup 0.269s, max 0.386s, min 0.227s
Lookup misses 0


running xor16_xxhash with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.24Mb, max 0.37Mb, min 0.08Mb
Approximate database size 23.76Mb
Average errors 0.155, max 2, min 0
Average lookup 0.391s, max 0.436s, min 0.355s
Lookup misses 0


running bloom with 10000 OUIs and 8000000 Devices
Attempting 1000 random lookups
Average memory 0.20Mb, max 0.40Mb, min 0.02Mb
Approximate database size 38.65Mb
Average errors 0.030, max 1, min 0
Average lookup 0.536s, max 0.583s, min 0.500s
Lookup misses 1000
```

As we can see, xor16-xxhash seems to be the best combination of speed, size and
accuracy. Additionally the bloom filter seems to have some portability issue
(likely related to the 32/64 bit switch) which would need to be resolved.

Finally, a more ambitious test, on the same 3 machines, in the same order:

```
running xor16_xxhash with 10000 OUIs and 50000000 Devices
Attempting 1000 random lookups
Average memory 0.53Mb, max 1.58Mb, min 0.04Mb
Approximate database size 118.54Mb
Average errors 0.179, max 2, min 0
Average lookup 0.096s, max 0.137s, min 0.079s
Lookup misses 0

running xor16_xxhash with 10000 OUIs and 50000000 Devices
Attempting 1000 random lookups
Average memory -0.01Mb, max 1.58Mb, min -0.09Mb
Approximate database size 118.54Mb
Average errors 0.179, max 2, min 0
Average lookup 1.112s, max 1.322s, min 0.945s
Lookup misses 0

running xor16_xxhash with 10000 OUIs and 50000000 Devices
Attempting 1000 random lookups
Average memory 0.17Mb, max 1.99Mb, min 0.07Mb
Approximate database size 118.54Mb
Average errors 0.179, max 2, min 0
Average lookup 0.916s, max 1.009s, min 0.864s
Lookup misses 0
```

The benchmark code can be found
[here](https://github.com/Vagabond/bloom_router). These runs were generated by
the command in the README. The benchmark was first run on the Intel machine to
generate the routing tables, run a second time to use the generated routing
tables, and then the tables were copied to the Pi to run the test with the same
databases (generating the tables can be quite RAM intensive, so it is unsuitable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will tables be updated then? Every time someone registers a set of IDs, someone has to do the work to generate and distribute the new table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, so you'd likely want to batch updates.

to be done on the Pi).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see some extension briefly discussing the process of generating the table or tables. Who do we trust to do this? How do we certify the validity of a table? How do we keep a customer from shooting themselves in the foot?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added something

The process of generating the filter is roughly as follows: the complete list of
{DevEUI, AppEUI} pairs are hashed into a 64 bit keys and then fed into the XOR
filter generator. The filter then does some bit twiddling to set some bits in a
list of 'fingerprints'. A fairly readable example can be found
[here](https://github.com/juanbono/xor-filter/blob/master/src/lib.rs).

# Drawbacks
[drawbacks]: #drawbacks

## Why should we *not* do this?

No matter how you slice it, this adds quite a lot of information to the ledger.
However, storing every possible DevEUI in a list would be 32Gb ( 2^32 * 8 bytes)
of data. Storing AppEUIs or the associated routing destination just makes it
worse. XOR filters compress massively more, but they're not free.

One flaw with this scheme will be 'route posioning' where I will be able to get
join packets for devices I want to see routed to myself simply by adding that
device to my bloom filter. In theory this could be used as a denial of service
attack by interfering with the receive window for the legitimate join response
packet.

# Rationale and Alternatives
[alternatives]: #rationale-and-alternatives

## Why is this design the best in the space of possible designs?

XOR16 filters using xxhash64 were evaluated against bloom filters using a 1 in
200 million false positive rate, XOR8 filters using erlang:phash2 and xxhash64 and
xor16 filters using erlang:phash2. XOR16 with xxhash64 provided the best all
around speed and memory usage, and were very competitive for lookup times.
phash2 has too many collisions (it's only a 60 bit hash) for this application,
so it was rejected.

Other alternatives include cuckoo filters, but they're expected to be slower
than xor16+xxhash64 and require a more complicated implementation. Simply
storing a complete routing table, as discussed before, is not reasonable
because of size constraints.

## What other designs have been considered and what is the rationale for not choosing them?

Suggestions for other designs are welcome, routing arbitrary addresses without
being able to use prefix routing seems hard.

## What is the impact of not doing this?

We will be unable to support routing join packets for arbitrary LoRaWAN
join packets and thus unable to support LoRaWAN devices that are
already provisioned or unable to have their credentials modified.

# Unresolved Questions
[unresolved]: #unresolved-questions

## What parts of the design do you expect to resolve through the HIP process before this gets merged?

We need to nail down the permissible sizes of the XOR filters we want to store.

## What parts of the design do you expect to resolve through the implementation of this feature?

We need to define where/how these routing entries are stored, how they're
created/updated, etc.

## What related issues do you consider out of scope for this HIP that could be addressed in the future independently of the solution that comes out of this HIP?

Pricing and availability of routing table entries for organizations operating
their own OUI.

# Deployment Impact
[deployment-impact]: #deployment-impact

## How will current users be impacted?

All existing LoRaWAN devices will either have to have their credentials added
to the routing table, or they will need new credentials.

## How will existing documentation/knowlegebase need to be supported?

We don't have a lot of documentation for the existing thing, so we can probably
forego this.

## Is this backwards compatible?

It can be made to be, we can probably generate routing tables out of console's
DB, for the devices we know the DevEUI for.

# Success Metrics
[success-metrics]: #success-metrics

What metrics can be used to measure the success of this design?

## What should we measure to prove an acceptance of this by it's users?

A user should be able to onboard any arbitrary LoRaWAN device without altering
its DevEUI/AppEUI/AppKey.
Loading