slog-to-otel doesn't handle upgrade plan slog entries

[mhofman]

Describe the bug

The stacking of upgrade slog entries seem to cause the slog-to-otel logic to error

To Reproduce

See logs

Expected behavior

Slog traces with all spans and no errors

Platform Environment

agoric-upgrade-11 on emerynet

Screenshots

Error tracing slog: {
type: 'cosmic-swingset-upgrade-finish',
blockHeight: 1260068,
blockTime: 1693317619,
time: 1693318166.071951,
monotime: 306.1746670629978} (Error#1)
Error#1: Check failed
at Object.pop (packages/telemetry/src/slog-to-otel.js:448:11)
at slogSender (packages/telemetry/src/slog-to-otel.js:918:15)
at wrappedSlogSender (packages/telemetry/src/slog-to-otel.js:1072:7)
at slogSender (packages/telemetry/src/make-slog-sender.js:154:11)
at process.<anonymous> (packages/telemetry/src/slog-sender-pipe-entrypoint.js:117:15)
at process.emit (node:events:514:28)
Error tracing slog: {
type: 'cosmic-swingset-begin-block',
blockHeight: 1260068,
blockTime: 1693317619,
inboundQueueStats: {
cosmic_swingset_inbound_queue_length: 0,
cosmic_swingset_inbound_queue_add: 0,
cosmic_swingset_inbound_queue_remove: 0
},
time: 1693318166.078734,
monotime: 306.1814518990517
} (Error#2)
Error#2: Check failed
at Object.begin (packages/telemetry/src/kv-string-store.js:28:7)
at slogSender (packages/telemetry/src/slog-to-otel.js:930:30)
at wrappedSlogSender (packages/telemetry/src/slog-to-otel.js:1072:7)
at slogSender (packages/telemetry/src/make-slog-sender.js:154:11)
at process.<anonymous> (packages/telemetry/src/slog-sender-pipe-entrypoint.js:117:15)
at process.emit (node:events:514:28)

[mhofman]

Potential idea to avoid further regression of this kind: duplicate / move the check for unknown slog entries from the deployment / loadgen integration test, to the docker upgrade test.