kvserver: improve node liveness failure log, point to docs

[nvanbenschoten]

This commit improves the message logged on node liveness failures, which
is logged roughly every 4.5 seconds during liveness unavailability. The
improved message describes some of the common causes of liveness
unavailability (resource starvation and network connectivity problems)
and then links to our troubleshooting docs about the topic.

This was an action item coming out of a recent support incident postmortem.

[cockroach-teamcity]

This change is 

[jordanlewis]

, thanks for taking care of this!

@jseldess would you or someone from your team care to give this a quick review? What do you think of this strategy to write more verbose log messages (really, on-line documentation) during common troubleshooting scenarios?

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jordanlewis)

[nvanbenschoten]

@jseldess would you or someone from your team care to give this a quick review? What do you think of this strategy to write more verbose log messages (really, on-line documentation) during common troubleshooting scenarios?

Also, while here, could you confirm that we should be linking to https://www.cockroachlabs.com/docs/stable/... from Cockroach itself. Other alternatives would be to link to https://www.cockroachlabs.com/docs/<current version>/ or to link somewhere else that may have a lower chance of rotting.

[jseldess]

This very much LGTM! I love the idea of providing more guidance and links to docs in log messages as well as in error messages. Thanks for writing this, @nvanbenschoten, and for looping me in, @jordanlewis.

@nvanbenschoten, I think the link you're using is best and matches what I see done in other parts of the code base. The stable alias shouldn't rot. If the target url changes somehow, we should always be putting client or server-side redirects in place on our end.

Beyond the scope of this PR: I think texts like these should ultimately be centralized in a file or files somewhere, or even housed outside of the cockroach repo and fetched as a part of the build process. That would make it much easier for writers to know what and when to review log and error texts, and perhaps there'd even be a way to auto-assign docs reviewers in cases where those are touched. In an ideal world, this would apply to:

• CLI help
• SQL shell help
• Session variable, built-in function, cluster setting descriptions
• Error messages
• Log messages
• UI copy and tooltips
  I'm sure I'm missing others. Thoughts on something like this? Wonder if @mjibson has thoughts since he's worked on other docs efficiencies like SQL diagrams and is working on auto-generated API docs, from what I hear.

[nvanbenschoten]

TFTR!

If the target url changes somehow, we should always be putting client or server-side redirects in place on our end.

Got it, that's very helpful to know. And that's even true of section headers like #node-liveness-issues, right?

Thoughts on something like this?

Centralizing string assets like these seems like a natural progression. I think that's actually a common approach, especially when applications start thinking about internationalization. This would also make it much easier to auto-assign docs reviewers because we could match on the package of the files being changed.

bors r+

[jseldess]

Got it, that's very helpful to know. And that's even true of section headers like #node-liveness-issues, right?

Yep! I'm not sure how thoughtful we've been with header-level redirects, but I'll bring it up in the docs meeting.

[maddyblue]

I'm currently work on extracting out our admin ui HTTP endpoints into auto docs. If we want to put this stuff somewhere else it could follow a similar path. In general it's not too much work. Up to you. If users or operators of cockroach would find it useful, ask the dev interfaces team.

[craig]

Build failed:

• GitHub CI (Cockroach)

[lunevalex]

drive by kicking bors again, since it failed

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)