tracing: enhance payloads API to allow for ephemeral payloads

[tbg]

Is your feature request related to a problem? Please describe.
Payloads, once emitted, are immutable. That means that payloads are generally emitted after something interesting happened. That "something interesting" may block for a long time; maybe forever! So it should ideally be observable as well.

At the time of writing, the best we've got is finding the open trace span, getting the goroutine ID, and looking at goroutine dumps (where hopefully the goroutine that created the span is also the one still blocking on the "something interesting").

This could be a lot more useful if there were a way to attach "ephemeral" payloads to the trace that convert to a full payload once finalized.

Describe the solution you'd like

From today:

{
  var item MyPayload
  time.Sleep(10*time.Second)
  item.Info1 = "foo"
  time.Sleep(10*time.Second)
  item.Info2 = "bar"
}
sp.LogStructured(item)

to some variation on:

item := sp.Ephemeral(&MyPayloadType{})

// Turns the item into an immutable payload of
// the span.
defer item.LogStructured()
{
  time.Sleep(10*time.Second)
  item.Do(func(inner interface{}) {
    inner.(*MyPayloadType).Info1 = "foo"
  }
  time.Sleep(10*time.Second)
  item.Do(func(inner tracing.Structured) {
    inner.(*MyPayloadType).Info2 = "foo"
  }
}

type Ephemeral struct {
  mu sync.Mutex
  wrapped Structured
}

func (e *Ephemeral) Do(f func(Structured) {
  e.mu.Lock()
  defer e.mu.Unlock()
  f(e.wrapped)
}

func (sp *Span) GetRecording(ephemeral bool) Recording {
  // If ephemeral == true, also return the ephemeral parts
  // as regular payloads (the caller agrees that it can
  // handle these). We could also wrap them in a payload
  // so that any caller not prepared to handle them would
  // ignore them?
}

Describe alternatives you've considered
There are probably better APIs than the above.

Additional context
This would definitely come in handy for contention events:

cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go

Lines 751 to 755 in 64aadfe

	h.ev = &roachpb.ContentionEvent{
	Key: s.key,
	TxnMeta: *s.txn,
	}
	h.tBegin = timeutil.Now()

But there has also been interest in making long-running backups more observable, for which this mechanism might be useful. For example, various backup spans could hang a progress report or latest error payload off the trace. The details here are unclear.

Jira issue: CRDB-3091

Epic CRDB-20779