Skip to content

Commit

Permalink
Remove zlib compression gossip query support
Browse files Browse the repository at this point in the history
Gossip query compression is not very useful - it was added for
mobile clients to, in theory, sync the gossip data directly from
P2P peers, but to my knowledge no mobile clients actually use it
for that, or at least use it where the gossip *query* data is a
substantial portion of their overall bandwidth usage.

Further, because of the semantics of `gossip_timestamp_filter`, its
impractical to ensure you receive a reliable, full view of the
gossip data without re-downloading large portions of the gossip
data on startup.

Ultimately, gossip queries are a pretty non-optimal method of
synchronizing the gossip data. If someone wants highly optimized
gossip data synchronization a new method based on set
reconciliation needs to be propose.

Finally, the current gossip query encoding semantics do not allow
for negotiation and instead require all lightning implementations
take a zlib dependency in some form or another. Given the recent
zlib decoding memory corruption vulnerability, this seems like an
opportune time to simply remove the zlib support, requiring that
nodes stop sending compressed gossip query data (though they can
support reading such gossip query data as long as they wish).

This is an alternative to the suggested gossip query encoding
support in lightning#825.
  • Loading branch information
TheBlueMatt committed Apr 20, 2022
1 parent cf4fddd commit f2c8588
Showing 1 changed file with 1 addition and 11 deletions.
12 changes: 1 addition & 11 deletions 07-routing-gossip.md
Original file line number Diff line number Diff line change
Expand Up @@ -576,16 +576,10 @@ rest contains the data.

Encoding types:
* `0`: uncompressed array of `short_channel_id` types, in ascending order.
* `1`: array of `short_channel_id` types, in ascending order, compressed with zlib deflate<sup>[1](#reference-1)</sup>
* `1`: previously used for zlib compression, no longer recommended

This encoding is also used for arrays of other types (timestamps, flags, ...), and specified with an `encoded_` prefix. For example, `encoded_timestamps` is an array of timestamps than can be either compressed (with a `1` prefix) or uncompressed (with a `0` prefix).

Note that a 65535-byte zlib message can decompress into 67632120
bytes<sup>[2](#reference-2)</sup>, but since the only valid contents
are unique 8-byte values, no more than 14 bytes can be duplicated
across the stream: as each duplicate takes at least 2 bits, no valid
contents could decompress to more than 3669960 bytes.

Query messages can be extended with optional fields that can help reduce the number of messages needed to synchronize routing tables by enabling:

- timestamp-based filtering of `channel_update` messages: only ask for `channel_update` messages that are newer than the ones you already have.
Expand Down Expand Up @@ -1129,10 +1123,6 @@ A->D's `update_add_htlc` message would be:
And D->C's `update_add_htlc` would again be the same as B->C's direct payment
above.

## References

1. <a id="reference-1">[RFC 1950 "ZLIB Compressed Data Format Specification version 3.3](https://www.ietf.org/rfc/rfc1950.txt)</a>
2. <a id="reference-2">[Maximum Compression Factor](https://zlib.net/zlib_tech.html)</a>

![Creative Commons License](https://i.creativecommons.org/l/by/4.0/88x31.png "License CC-BY")
<br>
Expand Down

0 comments on commit f2c8588

Please sign in to comment.