Advertize compression algorithms support in `init` (features 32/33) #825

t-bast · 2020-12-14T14:52:10Z

Add a list of supported compression algorithms to the init message.
These compression algorithms may be used in several places in the future, but currently only in extended gossip queries.

Without this explicit field, it's impossible to opt-out of zlib compression.
If your node implementation does not support zlib, it will only be discovered when your peer sends you a query_short_channel_ids that asks for a compressed response. Your only choices are to close the connection or send back uncompressed data, which may or may not be accepted by your peer.

Fixes #811

ariard

Thanks for working on this.

09-features.md

t-bast · 2020-12-23T15:40:18Z

I've thought about this more, and I'm not a fan of any kind of option_raw_format: feature bits should not be used to advertise a lack of functionality. But I'm not a fan of an explicit zlib_compress feature bit either.

Another angle that could be explored for this would be a new tlv field in the init message:

1. type: 3 (`compression_methods`)
2. data:
        * [`tu64`:`supported_algorithms`]

Each bit indicates support for a specific compression algorithm.
The only supported values for now would be 0x00 (only uncompressed data) and 0x01 (support for zlib compression).
I don't expect us to ever support a lot of compression options, but since we have truncated uint64 available in all implementations, this gives us 64 potential algorithms that can be used (more than enough).

Nodes then need to remember what compression options are supported by their peers and act accordingly when doing gossip queries. If our peer set 0x00, we should only use the raw format. If he set 0x01 we can use either raw or zlib format.

What do you think @TheBlueMatt @ariard @bmancini55? Should I amend this PR to go in that direction?

bmancini55 · 2020-12-23T21:30:32Z

Thanks @t-bast!

I agree, option_raw_format seems clunky. I do think zlib_compress could work if we follow similar rules that you just outlined, basically defaulting to using raw in the absence of feature agreement.

That said, I like your new TLV proposal. It makes sense that supported compression would be indicated in the same way the encoding flags are used.

One suggestion would be to put the TLV and associated logic under an option_compression feature. This way it would be clear which nodes are legacy and which support these cascading compression rules. Peer selection could then make determinations beforehand to connect to nodes that support these rules.

t-bast · 2020-12-28T09:41:19Z

One suggestion would be to put the TLV and associated logic under an option_compression feature. This way it would be clear which nodes are legacy and which support these cascading compression rules. Peer selection could then make determinations beforehand to connect to nodes that support these rules.

Done in 673c912

01-messaging.md

TheBlueMatt

Looks good to me.

t-bast · 2021-01-05T13:59:03Z

Rebased and clarified the case where no compression algorithm is supported in c4322b7

01-messaging.md

ysangkok · 2021-04-01T18:56:33Z

09-features.md

+| 18/19 | `option_support_large_channel`    | Can create large channels                                | IN       |                           | [BOLT #2](02-peer-protocol.md#the-open_channel-message)              |
+| 20/21 | `option_anchor_outputs`           | Anchor outputs                                           | IN       | `option_static_remotekey` | [BOLT #3](03-transactions.md)                                        |
+| 22/23 | `option_anchors_zero_fee_htlc_tx` | Anchor commitment type with zero fee HTLC transactions   | IN       |                           | [BOLT #3][bolt03-htlc-tx], [lightning-dev][ml-sighash-single-harmful]|
+| 24/25 | `option_compression`              | Compression algorithms advertised in `init`              | IN       |                           | [BOLT #1](01-messaging.md#the-init-message)                          |


Electrum version 4.1.0 uses bit 25 for Trampoline: https://github.com/spesmilo/electrum/blob/92226b077ad9d248e0f3325f3de89b34b1778ea5/RELEASE-NOTES#L76

That sounds...risky. There's no guarantee this will be the final spec feature bit.

This adds the new feature bit from lightning/bolts#825

TheBlueMatt · 2021-07-23T20:16:44Z

I have an implementation of this at lightningdevkit/rust-lightning#1015, happy to test with any other implementations :).

t-bast · 2021-08-13T07:00:09Z

I have an implementation of this at lightningdevkit/rust-lightning#1015, happy to test with any other implementations :).

I have a very old branch with early eclair support for it, but it needs many changes now, I'll get back to it when I have some time available.

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

t-bast · 2021-09-15T16:09:46Z

I have an implementation of this at lightningdevkit/rust-lightning#1015, happy to test with any other implementations :).

Here is a PR to support this feature on eclair: ACINQ/eclair#1956
If you can test it against rust-lightning it would be great!

This adds the new feature bit from lightning/bolts#825

TheBlueMatt · 2021-11-13T17:03:03Z

Needs rebase.

This adds the new feature bit from lightning/bolts#825

01-messaging.md

07-routing-gossip.md

vincenzopalazzo

LGTM, I have only some small comments

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

This adds the new feature bit from lightning/bolts#825

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

TheBlueMatt · 2021-12-06T18:57:48Z

Needs rebase. @t-bast and I successfully tested this between LDK and eclair.

Add a tlv field in `init` to list supported compression algorithms. This compression will be used in several places, currently in extended gossip queries. Fixes #811

TheBlueMatt · 2021-12-06T19:23:37Z

Meeting discussion pivoted to just dropping zlib - it only adds so much and the complexity overhead of this PR maybe isn't worth it vs just deprecating zlib to begin with.

t-bast · 2021-12-07T09:38:16Z

I'll reach out to a few mobile wallets to see if they are fine with deprecating zlib.

Gossip query compression is not very useful - it was added for mobile clients to, in theory, sync the gossip data directly from P2P peers, but to my knowledge no mobile clients actually use it for that, or at least use it where the gossip *query* data is a substantial portion of their overall bandwidth usage. Further, because of the semantics of `gossip_timestamp_filter`, its impractical to ensure you receive a reliable, full view of the gossip data without re-downloading large portions of the gossip data on startup. Ultimately, gossip queries are a pretty non-optimal method of synchronizing the gossip data. If someone wants highly optimized gossip data synchronization a new method based on set reconciliation needs to be propose. Finally, the current gossip query encoding semantics do not allow for negotiation and instead require all lightning implementations take a zlib dependency in some form or another. Given the recent zlib decoding memory corruption vulnerability, this seems like an opportune time to simply remove the zlib support, requiring that nodes stop sending compressed gossip query data (though they can support reading such gossip query data as long as they wish). This is an alternative to the suggested gossip query encoding support in lightning#825.

t-bast · 2022-04-26T07:08:06Z

Replaced by #981 (a.k.a death to zlib)

t-bast mentioned this pull request Dec 14, 2020

Lightning Specification Meeting 2020/12/21 #827

Closed

18 tasks

ariard reviewed Dec 14, 2020

View reviewed changes

09-features.md Outdated Show resolved Hide resolved

t-bast force-pushed the zlib-feature-flag branch from 35c479d to 673c912 Compare December 28, 2020 09:40

t-bast changed the title ~~Advertize zlib support via feature bits~~ Advertize compression algorithms support in init Dec 28, 2020

t-bast force-pushed the zlib-feature-flag branch from 8e2257d to bc87bda Compare December 28, 2020 09:53

t-bast mentioned this pull request Dec 28, 2020

Lightning Specification Meeting 2021/01/04 #830

Closed

16 tasks

bmancini55 reviewed Dec 29, 2020

View reviewed changes

01-messaging.md Outdated Show resolved Hide resolved

TheBlueMatt approved these changes Dec 30, 2020

View reviewed changes

t-bast force-pushed the zlib-feature-flag branch from 8548f2f to c4322b7 Compare January 5, 2021 13:58

t-bast mentioned this pull request Jan 11, 2021

Lightning Specification Meeting 2021/01/18 #831

Closed

17 tasks

t-bast mentioned this pull request Jan 27, 2021

Lightning Specification Meeting 2021/02/01 #838

Closed

21 tasks

Roasbeef requested changes Feb 1, 2021

View reviewed changes

01-messaging.md Outdated Show resolved Hide resolved

t-bast force-pushed the zlib-feature-flag branch from c4322b7 to 5071b93 Compare February 2, 2021 08:12

t-bast mentioned this pull request Feb 9, 2021

Lightning Specification Meeting 2021/02/15 #840

Closed

23 tasks

rustyrussell mentioned this pull request Feb 18, 2021

BOLT 2: option_shutdown_anysegwit (Feature 26/27) #672

Merged

ysangkok reviewed Apr 1, 2021

View reviewed changes

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jul 23, 2021

Define feature for option_compression advertisement

1004b08

This adds the new feature bit from lightning/bolts#825

TheBlueMatt mentioned this pull request Jul 23, 2021

Implement Compression Algorithm Advertisement lightningdevkit/rust-lightning#1015

Closed

TheBlueMatt mentioned this pull request Aug 12, 2021

Lightning Specification Meeting 2021/08/16 #893

Closed

18 tasks

t-bast mentioned this pull request Aug 27, 2021

Lightning Specification Meeting 2021/08/30 #901

Closed

15 tasks

t-bast mentioned this pull request Sep 8, 2021

Lightning Specification Meeting 2021/09/13 #909

Closed

19 tasks

t-bast added a commit to ACINQ/eclair that referenced this pull request Sep 15, 2021

Add support for option_compression

94f6c64

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

t-bast mentioned this pull request Sep 15, 2021

Add support for option_compression ACINQ/eclair#1956

Closed

t-bast mentioned this pull request Oct 6, 2021

Lightning Specification Meeting 2021/10/11 #920

Closed

19 tasks

t-bast mentioned this pull request Oct 21, 2021

Lightning Specification Meeting 2021/10/25 #929

Closed

21 tasks

t-bast mentioned this pull request Nov 5, 2021

Lightning Specification Meeting 2021/11/08 #933

Closed

21 tasks

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 10, 2021

Define feature for option_compression advertisement

e1a8cca

This adds the new feature bit from lightning/bolts#825

t-bast changed the title ~~Advertize compression algorithms support in init~~ Advertize compression algorithms support in init (features 32/33) Nov 12, 2021

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 13, 2021

Define feature for option_compression advertisement

868f6d7

This adds the new feature bit from lightning/bolts#825

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 13, 2021

Define feature for option_compression advertisement

3e22a96

This adds the new feature bit from lightning/bolts#825

vincenzopalazzo reviewed Nov 15, 2021

View reviewed changes

01-messaging.md Show resolved Hide resolved

vincenzopalazzo reviewed Nov 15, 2021

View reviewed changes

07-routing-gossip.md Show resolved Hide resolved

vincenzopalazzo reviewed Nov 15, 2021

View reviewed changes

t-bast mentioned this pull request Nov 22, 2021

Lightning Specification Meeting 2021/11/22 #936

Closed

21 tasks

t-bast force-pushed the zlib-feature-flag branch from e0a61bd to 3cf46fd Compare November 23, 2021 10:36

t-bast added a commit to ACINQ/eclair that referenced this pull request Nov 30, 2021

Add support for option_compression

8f9480c

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

t-bast added a commit to ACINQ/eclair that referenced this pull request Nov 30, 2021

Add support for option_compression

8f0229f

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

t-bast added a commit to ACINQ/eclair that referenced this pull request Nov 30, 2021

Add support for option_compression

7c1f245

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 30, 2021

Define feature for option_compression advertisement

5ba5cc4

This adds the new feature bit from lightning/bolts#825

t-bast added a commit to ACINQ/eclair that referenced this pull request Dec 1, 2021

Add support for option_compression

84a6a8b

Implement lightning/bolts#825 This lets us specify which compression algorithms we support and use one that our peer supports as well when syncing the network graph.

t-bast mentioned this pull request Dec 6, 2021

Lightning Specification Meeting 2021/12/06 #943

Closed

21 tasks

Advertize known compression algorithms

192bfc3

Add a tlv field in `init` to list supported compression algorithms. This compression will be used in several places, currently in extended gossip queries. Fixes #811

t-bast force-pushed the zlib-feature-flag branch from 3cf46fd to 192bfc3 Compare December 6, 2021 19:00

TheBlueMatt mentioned this pull request Apr 20, 2022

Remove zlib compression gossip query support #981

Merged

t-bast closed this Apr 26, 2022

t-bast deleted the zlib-feature-flag branch April 26, 2022 07:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Advertize compression algorithms support in `init` (features 32/33) #825

Advertize compression algorithms support in `init` (features 32/33) #825

t-bast commented Dec 14, 2020 •

edited

Loading

ariard left a comment

t-bast commented Dec 23, 2020

bmancini55 commented Dec 23, 2020

t-bast commented Dec 28, 2020

TheBlueMatt left a comment

t-bast commented Jan 5, 2021

ysangkok Apr 1, 2021

t-bast Apr 1, 2021

TheBlueMatt commented Jul 23, 2021

t-bast commented Aug 13, 2021

t-bast commented Sep 15, 2021

TheBlueMatt commented Nov 13, 2021

vincenzopalazzo left a comment

TheBlueMatt commented Dec 6, 2021

TheBlueMatt commented Dec 6, 2021

t-bast commented Dec 7, 2021

t-bast commented Apr 26, 2022

Advertize compression algorithms support in init (features 32/33) #825

Advertize compression algorithms support in init (features 32/33) #825

Conversation

t-bast commented Dec 14, 2020 • edited Loading

ariard left a comment

Choose a reason for hiding this comment

t-bast commented Dec 23, 2020

bmancini55 commented Dec 23, 2020

t-bast commented Dec 28, 2020

TheBlueMatt left a comment

Choose a reason for hiding this comment

t-bast commented Jan 5, 2021

ysangkok Apr 1, 2021

Choose a reason for hiding this comment

t-bast Apr 1, 2021

Choose a reason for hiding this comment

TheBlueMatt commented Jul 23, 2021

t-bast commented Aug 13, 2021

t-bast commented Sep 15, 2021

TheBlueMatt commented Nov 13, 2021

vincenzopalazzo left a comment

Choose a reason for hiding this comment

TheBlueMatt commented Dec 6, 2021

TheBlueMatt commented Dec 6, 2021

t-bast commented Dec 7, 2021

t-bast commented Apr 26, 2022

Advertize compression algorithms support in `init` (features 32/33) #825

Advertize compression algorithms support in `init` (features 32/33) #825

t-bast commented Dec 14, 2020 •

edited

Loading