tracing: fix incorrect field format when logging in JSON

[hawkw]

When configuring the tracing subscriber to emit JSON logs, we're
currently configuring it to use a JSON event formatter, but not a
JSON field formatter. This means that span and event fields are
formatted with the default field format, key=val, which is not valid
JSON. This results in parse errors when logging events in a span with
fields, and we format those fields as an error message instead. See
linkerd/linkerd2#5520.

Typically, the simplest way to configure a JSON fmt layer is to use
the .json() method. This sets both the event formatter and the field
formatter to emit JSON. However, since we are sharing some event
formatter configuration across the JSON and plaintext logging modes, the
proxy doesn't use this method; instead, it sets the JSON mode on the
event formatter which has already been configured with the shared
configuration, and sets it as the layer's event formatter without also
setting the field format, so we continue using the default field format.

This branch updates the proxy's tracing initialization code to also set
the field format to JSON when the JSON logging mode is enabled. I've
also added some comments to the JSON formatter configuration so we don't
mess this up in the future.

Before this change, trying to run the tests with the JSON log format
gets a panic (malformed fields cause the JSON formatter to panic in
debug mode, but it skips those fields in release mode so that it doesn't
crash the app). For example:

$ LINKERD2_PROXY_LOG=linkerd=debug LINKERD2_PROXY_LOG_FORMAT=json cargo test -p linkerd-app-integration -- transparency::inbound_http1 --test-threads=1 --nocapture
    Finished test [unoptimized] target(s) in 0.10s
     Running target/debug/deps/linkerd_app_integration-d798ce4657a07303

running 1 test
test tests::transparency::inbound_http1 ... thread 'main' panicked at 'span 'test_server' had malformed fields! this is a bug.
  error: expected value at line 1 column 1
  fields: FormattedFields { fields: "version=Http1 addr=127.0.0.1:58307 test=main" }', /home/eliza/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-subscriber-0.2.15/src/fmt/format/json.rs:162:49
stack backtrace:
    # ... snip ...

Now, the tests correctly output well-formed JSON:

$ LINKERD2_PROXY_LOG=debug LINKERD2_PROXY_LOG_FORMAT=json cargo test -p linkerd-app-integration -- transparency::inbound_http1 --test-threads=1 --nocapture
    Finished test [unoptimized] target(s) in 0.11s
     Running target/debug/deps/linkerd_app_integration-d798ce4657a07303

running 1 test
test tests::transparency::inbound_http1 ... {"timestamp":"Jan 14 09:20:34.499","level":"INFO","fields":{"message":"support server running"},"target":"linkerd_app_integration::server","spans":[{"addr":"127.0.0.1:55561","test":"main","version":"Http1","name":"test_server"}]}
{"timestamp":"Jan 14 09:20:34.499","level":"INFO","fields":{"message":"listening!"},"target":"linkerd_app_integration::server","spans":[{"addr":"127.0.0.1:55561","test":"main","version":"Http1","name":"test_server"}]}
Http1 server running; addr=127.0.0.1:55561
support destination service listening; addr=127.0.0.1:51487
{"timestamp":"Jan 14 09:20:34.501","level":"DEBUG","fields":{"message":"building app"},"target":"linkerd_app","spans":[{"test":"main:proxy","name":"proxy"}]}
{"timestamp":"Jan 14 09:20:34.502","level":"DEBUG","fields":{"message":"spawning daemon thread"},"target":"linkerd_app","spans":[{"test":"main:proxy","name":"proxy"}]}
proxy running; tap=, identity=None, inbound=127.0.0.1:33045 (SO_ORIGINAL_DST=10.1.2.3:55561), outbound=127.0.0.1:45721, metrics=127.0.0.1:45373
{"timestamp":"Jan 14 09:20:34.502","level":"INFO","fields":{"listen.addr":"127.0.0.1:45721","ingress_mode":false},"target":"linkerd_app","spans":[{"test":"main:proxy","name":"proxy"},{"name":"outbound"}]}
    # ... snip ...

[olix0r]

lgtm