*: wrap ctx.Err() with a stack

[jordanlewis]

It's all to common to get an error like "context canceled" without any
helpful information about which context was canceled and when. This
commit strives to improve the situation by at least including the stack
trace of when the ctx.Err() was retrieved.

Not sure who to include on this. It's a straw-man PR, also.

Release note: None

[cockroach-teamcity]

This change is 

[maddyblue]

I like the idea, but it looks like the search and replace you did needs some more finesse.

[jordanlewis]

Thanks for the review, lol, I'll go audit these more carefully now.

[jordanlewis]

RFAL

[petermattis]

Generating stack traces is fairly expensive. I'm not sure if that will be a problem here, just pointing it out. I'm a bit reluctant to have this always on. Perhaps the wrapping should be done in a way that it is controlled by an environment variable. Just brainstorming here. I haven't thought through this enough.

I'd guess that you'd rather get a stack trace from the goroutine that cancelled the context. I think that might be possible if we vendor context (I'm pretty sure you can vendor stdlib packages).

[jordanlewis]

Would using the GetSmallTrace facility be sufficiently cheaper to help you get over your reluctance? The problem is that these errors are generally pesky and hard to track down. They come up in the wild, and when they do, you really want to see more context from the error.

For example, from @andreimatei:

btw, when I'm running TPCC on a 4-node roachprod cluster (2 or 3 workloads running at the same time), I see the client getting Error: error in newOrder: pq: context deadline exceeded pretty often. Don't know what's up with those :S

[petermattis]

Would using the GetSmallTrace facility be sufficiently cheaper to help you get over your reluctance? The problem is that these errors are generally pesky and hard to track down. They come up in the wild, and when they do, you really want to see more context from the error.

I haven't measured the perf difference between debug.Stack and GetSmallTrace. I'd be surprised if there is a significant difference, but it is worth looking at. Previous experience has shown that calling debug.Stack too frequently (e.g. in non error paths) is a significant perf hit. I realize this PR is only doing it in error paths, but I'm just being cautious.

[andreimatei]

In my opinion, the more important stack is the one of the routine doing the canceling than the canceled one. For the cancelee, I think a message identifying its location ("ctx canceled while waiting for foo") is sufficient, instead of the stack.

Btw, I hope you've searched the code for places that handle context.ErrCanceled explicitly. I know for example @vivekmenezes is just introducing one currently.

Reviewable status: complete! 0 of 0 LGTMs obtained

[jordanlewis]

This isn't as good as #29429.