Question: expected resource usage for lnd

[brndnmtthws]

I've noticed the resource usage (CPU and mem) seem rather high for lnd. Far higher than bitcoind or parity, for example. Given that the number of transactions passing through my lnd node is ~0 on average, this doesn't make much sense to me. I'd be worried about the scalability implications if this is how much compute is required for an idle node.

Is this expected? Have I encountered a bug?

brenden@instance-5:~$ sudo docker exec -it lnd /bin/bash
root@instance-5:/lnd# lncli getinfo
{
    "identity_pubkey": "0327049d8d63f0c40193cdf3afc61817c8647808a4e482de0716fcef74e6d92ebf",
    "alias": "FederalReserve",
    "num_pending_channels": 0,
    "num_active_channels": 6,
    "num_peers": 14,
    "block_height": 514535,
    "block_hash": "0000000000000000000d5c6d4c9561d8a58c4052a58b9ea36400323ae5db6f55",
    "synced_to_chain": true,
    "testnet": false,
    "chains": [
        "bitcoin"
    ],
    "uris": [
        "0327049d8d63f0c40193cdf3afc61817c8647808a4e482de0716fcef74e6d92ebf@35.184.46.254:9735"
    ],
    "best_header_timestamp": "1521640825"
}
root@instance-5:/lnd#

Here's a portion of my lnd log: https://gist.github.com/brndnmtthws/303b320e0ea98578f3af5a89d5b6df61

The sample period for the screenshot above is 5s.

[halfik]

Cpu usage depends on how much information you node broadcast and recives. From what i understand each network change is broadacasted. So if anyone does a transaction, that changes a channel state and it is brodcasted, same if any nodes comes off or online and so on. Cpu usage is high when you just started lnd, later is should be way lower.

Each time i have to restart my pi my lnd takes up to 350% of cpu. But after it catches up it stays around 3-5% till a lot of network traffic comes in, then it jumps to like 150%.

[brndnmtthws]

The CPU usage stays above 200% on average even after a few days of running. The memory usage seems to grow without limit (it got as high as 20G before I restarted it yesterday).

[johnny-cash]

Hello,

The same thing happens to me with a raspberry, the consumption of resources is too high. Until yesterday I worked well, but since yesterday the consumption of CPU does not go down. Can it be caused by the ddos attack?

[halfik]

Hmm im using cache clear script. I had problems with cpu usage when bitcoind was syncing. Can you guys check if it helps: https://github.com/lizquilty/scripts/blob/master/cacheclear.sh

[brndnmtthws]

Hmm im using cache clear script. I had problems with cpu usage when bitcoind was syncing. Can you guys check if it helps: https://github.com/lizquilty/scripts/blob/master/cacheclear.sh

What's the point of dropping the disk caches?

[johnny-cash]

The problem has nothing to do with the disk cache. Bitcoind works correctly, the problem is from the lnd server.

[Roasbeef]

@halfik's description is false.

@brndnmtthws looking at your logs it appears that this IP 91.234.48.232 is repeatedly connecting, then disconnecting again. With the way the initial handshake works atm, there's a burst of activity post peer connection. With a new extension to the protocol that will be phased in by feature bits (which we'll make mandatory), this burst will go away.

Looking at the logs again, we see this:

Mar 21 14:01:29 instance-5 docker[2102]: 2018-03-21 14:01:29.986 [ERR] HSWC: ChannelPoint(cfb20ad3e33e6649ddc7a3e0079b13a4001954c30a111b68ff119f8c97db41fa:0): unable to accept new commitment: commitment transaction dips peer below chan reserve

The peer you're connected to may be an older peer, and doesn't actually adhere to the constraints w.r.t channel reserve. If you force close that channel (and maybe also use iptables to ban that peer) things should return to normal. We're working on a patchset to automatically force close channels from faulty nodes like these. It requires a bit of reworking in the contractcourt to only dispatch notifications on block confirmations, while ceasing to use a channel once broadcast.

Here's my node on mainnet for example:

14036 kek      20   0  710340 154184  13384 S   3.0  1.9 131:39.60 lnd_linux

As you can see it's hovering around 1.9% CPU, but once we apply this new protocol enchantment, it should go down even further unless were doing high-throughput payment forwarding.

[brndnmtthws]

@Roasbeef good to know. Other people might bump into this problem too, so it might be good to have some kind of workaround. It also sounds like it could be used as a DoS attack vector (starve a node of its resources by firing up a bunch of nodes and reconnecting over and over).

Aside from blocking that IP with iptables, any suggestions on how to mitigate this sort of problem? Any plan to implement a banlist of bad nodes like bitcoind et al?

[Roasbeef]

Yep, we have plans to implement a banlist, the scaffolding is already there, so what's left is: figuring out what actions constitute an increase in ban score, banning, and also writing the banlist persistently to disk.

We're also working on a workaround, but to do it properly requires a bit of additional testing and a change in the way we handle spend notifications in the contractcourt.

If it's OK with you @brndnmtthws, I'll close this issue, then create another issue that outlines our plan for DoS mitigation, and also persistent peer banning.

[brndnmtthws]

I blocked the IP mentioned above, but the CPU usage for lnd hasn't leveled off after 5 minutes or so:

The log is filled with errors that look like this: https://gist.github.com/brndnmtthws/30ba00e1a43d8f05882760ee9a06e728

If you look through the log, you'll see the same channel point (ex. 6860485751e2dd6d98b55b7f7a908d91d1502d0d287fc7ddfe5fb9f668936c28) is repeated over and over.

I'm not sure what this means, but it smells like a problem to me.

[Roasbeef]

You'll need to close out that channel (with a force close) since it has desyn'd. That was just an IP that I spotted in the logs, it might have been a different node.

[Roasbeef]

You'll get those log errors if you're connected to an old clightning implementation, as they seem to do sending out invalid channel announcements. From speaking with the devs, this is fixed in newer versions of the implementation. You may also want to ban/disconnect the node sending those announcements. Worth mentioning again, that once the new protocol features are finished, there will no longer be this initial burst on reconnect.

[brndnmtthws]

Got it. Thanks for the info. I'll keep an eye on it as things progress.

[brndnmtthws]

Not sure of the quality of this article, but it popped up in my Twitter feed and thought it might be relevant: https://www.trustnodes.com/2018/03/21/lightning-network-ddos-sends-20-nodes

[Roasbeef]

That article is mostly garbage; the author is uninformed. The attacks it details have kernel level defenses and are unrelated to implementations. That "attack" can be launched on any public facing misconfigured Internet service.

20% of nodes didn't go down. This is an issue with the way explorers are interpreted by users today. We'll only accept advertisements from nodes that actually have channels. However, if a node closes out all its channels, we don't yet remove them from our network view. This is why you at times see unconnected nodes on these explorers. When users updated to the new version of lnd, they didn't accept the announcements from nodes that no longer had channels open. To the uninformed this looks like many nodes "went down".