Node crashes with "Failed to write response"

[RiccardoM]

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):

tendermint: ""
abci: 0.17.0
blockprotocol: 11
p2pprotocol: 8

ABCI app (name for built-in, URL for self-written if it's publicly available):
Desmos v0.15.1

Environment:

• OS (e.g. from /etc/os-release):

  NAME="Ubuntu"
  VERSION="20.04.1 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.1 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal
  

• Install tools:
• Others:

What happened:
Yesterday, one of our chain nodes has stopped with error Failed to write response for no apparent reason. This is not the first time this happens, and we still have to identify why.

What you expected to happen:
The node should not crash

Have you tried the latest version:
No

How to reproduce it (as minimally and precisely as possible):
I have yet to know this

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):
https://pastebin.com/rbaw6FVH

Config (you can paste only the changes you've made):

pruning = "nothing"

[tessr]

Thanks for this report. One of the first things I notice is the the Tendermint version is missing 🤔 We'll look into that. Did you use tendermint version to get the version?

Also, I see that Desmos v0.15.1 is running Cosmos SDK v0.40. There's a new SDK release series out which fixes some halting bugs. The latest release is v0.41.3. I recommend you try upgrading first and if this is still happening we can try to take a closer look.

[RiccardoM]

Thanks for this report. One of the first things I notice is the the Tendermint version is missing We'll look into that. Did you use tendermint version to get the version?

I used desmos tendermint version to get the version

Also, I see that Desmos v0.15.1 is running Cosmos SDK v0.40. There's a new SDK release series out which fixes some halting bugs. The latest release is v0.41.3. I recommend you try upgrading first and if this is still happening we can try to take a closer look.

Thank, we'll surely update on our next chain upgrade.

[tac0turtle]

@RiccardoM
Hey,

if you'd like to get the Tendermint version in here:

tendermint: ""
abci: 0.17.0
blockprotocol: 11
p2pprotocol: 8

add:

VERSION := $(shell go list -m github.com/tendermint/tendermint | sed 's:.* ::')
LD_FLAGS = -X github.com/tendermint/tendermint/version.TMCoreSemVer=$(VERSION)

to your makefile.

[tessr]

Whoa, good tip, @marbar3778 - do we have that documented anywhere?

[tac0turtle]

Whoa, good tip, @marbar3778 - do we have that documented anywhere?

I think we only put it in the upgrading doc, but doesn't seem like anyone read it 😃

[tac0turtle]

do we have that documented anywhere?

added here: #6151

[melekes]

The node should not crash

I don't see any stacktrace. Are you sure the node has crashed and not simply hanged?

if crashed

Could you paste the stacktrace of the panic that lead to a crash? Usually it's the last line of the log / stdout.

if hanged

Do you have a goroutine list? Note you can get it by killing the frozen node with kill -6 <PID> or using tendermint debug kill <pid> </path/to/out.zip> — home=</path/to/app.d> https://docs.tendermint.com/master/tools/debugging/pro.html

[RiccardoM]

@melekes You right, it hangs. I'm now running tendermint debug kill but it also seems to be hanging 😅

[melekes]

kill -6 <PID> works all the time

[RiccardoM]

kill -6 <PID> works all the time

# tendermint debug kill 1858129 ~/debug.zip --home=~/.desmos
I[2021-02-22|10:05:50.012] getting node status...                       
I[2021-02-22|10:05:50.021] getting node network info...                 
I[2021-02-22|10:05:50.030] getting node consensus state...              
I[2021-02-22|10:05:50.050] copying node WAL...                          
ERROR: stat ~/.desmos/data/cs.wal/wal: no such file or directory

# ls ~/.desmos/data
application.db  blockstore.db  cs.wal  evidence.db  priv_validator_state.json  snapshots  state.db  tx_index.db

[melekes]

kill -6 <PID> works all the time

# tendermint debug kill 1858129 ~/debug.zip --home=~/.desmos
I[2021-02-22|10:05:50.012] getting node status...                       
I[2021-02-22|10:05:50.021] getting node network info...                 
I[2021-02-22|10:05:50.030] getting node consensus state...              
I[2021-02-22|10:05:50.050] copying node WAL...                          
ERROR: stat ~/.desmos/data/cs.wal/wal: no such file or directory

# ls ~/.desmos/data
application.db  blockstore.db  cs.wal  evidence.db  priv_validator_state.json  snapshots  state.db  tx_index.db

sorry, maybe I should've been more specific. I meant Linux kill command https://linux.die.net/man/1/kill

[RiccardoM]

sorry, maybe I should've been more specific. I meant Linux kill command https://linux.die.net/man//kill

Yeah, I've run the kill -6 command. Then I restarted the node and tried with tendermint debug kill but I got that error

[melekes]

Yeah, I've run the kill -6 command.

cool. so what was the stacktrace?

[RiccardoM]

cool. so what was the stacktrace?

I could not get any stacktrace. I just used that command to kill the service as the tendermint debug kill was not working. But I was not able to get any stracktrace from it. The node just stopped without any error and the only way to solve was to kill it or restart it.

[alexanderbez]

Since it hangs, we'll need to see the stacktrace/goroutine list. I fixed the debug kill command, but I'm not sure if that landed in a point release or not.

[tessr]

Let's figure out if the debug kill command was fixed or not. @alexanderbez can you point me to the commit/PR where you fixed it, and I can see if it was released? Otherwise we can backport and include it in 0.34.9.

[alexanderbez]

Here is the PR. I didn't add a backport label, so I don't think it exists in any release yet.

[melekes]

Closing as duplicate of #6184