Skip to content

Commit

Permalink
Merge #59492
Browse files Browse the repository at this point in the history 
59492: sql,util: add vtable for node-local inflight span registry r=knz a=angelapwen

This change adds a virtual table, `crdb_internal.node_inflight_trace_spans`
that exposes data on currently inflight spans in the current node.
The table includes information on the span IDs, parent span IDs, trace IDs,
start time, duration (measured by time of collection - start time of the
span), and the span's operation.

Part 1 of addressing #55733. Will follow up with a built-in that will
allow querying of a particular span ID to surface more detailed
recording information.


Release note (sql change): Adds a virtual table,
`crdb_internal.node_inflight_trace_spans`, which exposes span ID, parent span
ID, trace ID, start time, duration, and operation of node-local inflight spans.

Co-authored-by: angelapwen <angelaw@cockroachlabs.com>
  • Loading branch information
craig[bot] and angelapwen committed Feb 2, 2021
2 parents 9b25d4e + fb8c5d8 commit 3699ce6
Show file tree
Hide file tree
Showing 18 changed files with 244 additions and 152 deletions.
5 changes: 5 additions & 0 deletions pkg/cli/testdata/zip/partial1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/1/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/1/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/1/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/1/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/1/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/1/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/1/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/1/crdb_internal.node_runtime_info.txt
Expand Down Expand Up @@ -118,6 +119,9 @@ writing: debug/nodes/2/crdb_internal.leases.txt.err.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/2/crdb_internal.node_build_info.txt
writing: debug/nodes/2/crdb_internal.node_build_info.txt.err.txt
^- resulted in ...
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/2/crdb_internal.node_inflight_trace_spans.txt
writing: debug/nodes/2/crdb_internal.node_inflight_trace_spans.txt.err.txt
^- resulted in ...
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/2/crdb_internal.node_metrics.txt
writing: debug/nodes/2/crdb_internal.node_metrics.txt.err.txt
^- resulted in ...
Expand Down Expand Up @@ -171,6 +175,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/3/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/3/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/3/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/3/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/3/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/3/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/3/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/3/crdb_internal.node_runtime_info.txt
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/testdata/zip/partial1_excluded
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/1/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/1/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/1/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/1/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/1/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/1/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/1/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/1/crdb_internal.node_runtime_info.txt
Expand Down Expand Up @@ -105,6 +106,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/3/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/3/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/3/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/3/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/3/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/3/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/3/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/3/crdb_internal.node_runtime_info.txt
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/testdata/zip/partial2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/1/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/1/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/1/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/1/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/1/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/1/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/1/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/1/crdb_internal.node_runtime_info.txt
Expand Down Expand Up @@ -104,6 +105,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/3/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/3/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/3/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/3/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/3/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/3/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/3/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/3/crdb_internal.node_runtime_info.txt
Expand Down
1 change: 1 addition & 0 deletions pkg/cli/testdata/zip/testzip
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/1/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/1/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/1/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/1/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/1/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/1/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/1/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/1/crdb_internal.node_runtime_info.txt
Expand Down
1 change: 1 addition & 0 deletions pkg/cli/testdata/zip/unavailable
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ retrieving SQL data for crdb_internal.gossip_network... writing: debug/nodes/1/c
retrieving SQL data for crdb_internal.gossip_nodes... writing: debug/nodes/1/crdb_internal.gossip_nodes.txt
retrieving SQL data for crdb_internal.leases... writing: debug/nodes/1/crdb_internal.leases.txt
retrieving SQL data for crdb_internal.node_build_info... writing: debug/nodes/1/crdb_internal.node_build_info.txt
retrieving SQL data for crdb_internal.node_inflight_trace_spans... writing: debug/nodes/1/crdb_internal.node_inflight_trace_spans.txt
retrieving SQL data for crdb_internal.node_metrics... writing: debug/nodes/1/crdb_internal.node_metrics.txt
retrieving SQL data for crdb_internal.node_queries... writing: debug/nodes/1/crdb_internal.node_queries.txt
retrieving SQL data for crdb_internal.node_runtime_info... writing: debug/nodes/1/crdb_internal.node_runtime_info.txt
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/zip.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ var debugZipTablesPerNode = []string{
"crdb_internal.leases",

"crdb_internal.node_build_info",
"crdb_internal.node_inflight_trace_spans",
"crdb_internal.node_metrics",
"crdb_internal.node_queries",
"crdb_internal.node_runtime_info",
Expand All @@ -106,6 +107,7 @@ var debugZipTablesPerNode = []string{

// Override for the default SELECT * when dumping the table.
var customSelectClause = map[string]string{
"crdb.internal.node_inflight_trace_spans": "*, WHERE duration > 10*time.Second ORDER BY trace_id ASC, duration DESC",
"system.jobs": "*, to_hex(payload) AS hex_payload, to_hex(progress) AS hex_progress",
"system.descriptor": "*, to_hex(descriptor) AS hex_descriptor",
}
Expand Down
1 change: 1 addition & 0 deletions pkg/sql/catalog/catconstants/constants.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ const (
CrdbInternalGossipLivenessTableID
CrdbInternalGossipNetworkTableID
CrdbInternalIndexColumnsTableID
CrdbInternalInflightTraceSpanTableID
CrdbInternalJobsTableID
CrdbInternalKVNodeStatusTableID
CrdbInternalKVStoreStatusTableID
Expand Down
50 changes: 50 additions & 0 deletions pkg/sql/crdb_internal.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,13 @@ import (
"github.com/cockroachdb/cockroach/pkg/sql/sem/tree"
"github.com/cockroachdb/cockroach/pkg/sql/sessiondata"
"github.com/cockroachdb/cockroach/pkg/sql/types"
"github.com/cockroachdb/cockroach/pkg/util/duration"
"github.com/cockroachdb/cockroach/pkg/util/errorutil"
"github.com/cockroachdb/cockroach/pkg/util/json"
"github.com/cockroachdb/cockroach/pkg/util/log"
"github.com/cockroachdb/cockroach/pkg/util/protoutil"
"github.com/cockroachdb/cockroach/pkg/util/timeutil"
"github.com/cockroachdb/cockroach/pkg/util/tracing"
"github.com/cockroachdb/errors"
)

Expand Down Expand Up @@ -100,6 +102,7 @@ var crdbInternal = virtualSchema{
catconstants.CrdbInternalGossipLivenessTableID: crdbInternalGossipLivenessTable,
catconstants.CrdbInternalGossipNetworkTableID: crdbInternalGossipNetworkTable,
catconstants.CrdbInternalIndexColumnsTableID: crdbInternalIndexColumnsTable,
catconstants.CrdbInternalInflightTraceSpanTableID: crdbInternalInflightTraceSpanTable,
catconstants.CrdbInternalJobsTableID: crdbInternalJobsTable,
catconstants.CrdbInternalKVNodeStatusTableID: crdbInternalKVNodeStatusTable,
catconstants.CrdbInternalKVStoreStatusTableID: crdbInternalKVStoreStatusTable,
Expand Down Expand Up @@ -1131,6 +1134,53 @@ CREATE TABLE crdb_internal.session_trace (
},
}

// crdbInternalInflightTraceSpanTable exposes the node-local registry of in-flight spans.
var crdbInternalInflightTraceSpanTable = virtualSchemaTable{
comment: `in-flight spans (RAM; local node only)`,
schema: `
CREATE TABLE crdb_internal.node_inflight_trace_spans (
trace_id INT NOT NULL, -- The trace's ID.
parent_span_id INT NOT NULL, -- The span's parent ID.
span_id INT NOT NULL, -- The span's ID.
start_time TIMESTAMPTZ, -- The span's start time.
duration INTERVAL, -- The span's duration, measured by time of
-- collection - start time for all in-flight spans.
operation STRING NULL -- The span's operation.
)`,
populate: func(ctx context.Context, p *planner, _ *dbdesc.Immutable, addRow func(...tree.Datum) error) error {
return p.ExecCfg().Settings.Tracer.VisitSpans(func(span *tracing.Span) error {
for _, rec := range span.GetRecording() {
traceID := rec.TraceID
parentSpanID := rec.ParentSpanID
spanID := rec.SpanID

startTime, err := tree.MakeDTimestampTZ(rec.StartTime, time.Microsecond)
if err != nil {
return err
}

spanDuration := rec.Duration
operation := rec.Operation

if err := addRow(
tree.NewDInt(tree.DInt(traceID)),
tree.NewDInt(tree.DInt(parentSpanID)),
tree.NewDInt(tree.DInt(spanID)),
startTime,
tree.NewDInterval(
duration.MakeDuration(spanDuration.Nanoseconds(), 0, 0),
types.DefaultIntervalTypeMetadata,
),
tree.NewDString(operation),
); err != nil {
return err
}
}
return nil
})
},
}

// crdbInternalClusterSettingsTable exposes the list of current
// cluster settings.
//
Expand Down
6 changes: 6 additions & 0 deletions pkg/sql/logictest/testdata/logic_test/crdb_internal
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ crdb_internal kv_node_status table NULL NULL NULL
crdb_internal kv_store_status table NULL NULL NULL
crdb_internal leases table NULL NULL NULL
crdb_internal node_build_info table NULL NULL NULL
crdb_internal node_inflight_trace_spans table NULL NULL NULL
crdb_internal node_metrics table NULL NULL NULL
crdb_internal node_queries table NULL NULL NULL
crdb_internal node_runtime_info table NULL NULL NULL
Expand Down Expand Up @@ -249,6 +250,11 @@ SELECT * FROM crdb_internal.zones WHERE false
zone_id subzone_id target range_name database_name table_name index_name partition_name
raw_config_yaml raw_config_sql raw_config_protobuf full_config_yaml full_config_sql

query IIITTT colnames
SELECT * FROM crdb_internal.node_inflight_trace_spans WHERE span_id < 0
----
trace_id parent_span_id span_id start_time duration operation

query ITTTTTTTTTTTTI colnames
SELECT * FROM crdb_internal.ranges WHERE range_id < 0
----
Expand Down
6 changes: 6 additions & 0 deletions pkg/sql/logictest/testdata/logic_test/crdb_internal_tenant
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ crdb_internal kv_node_status table NULL NULL NULL
crdb_internal kv_store_status table NULL NULL NULL
crdb_internal leases table NULL NULL NULL
crdb_internal node_build_info table NULL NULL NULL
crdb_internal node_inflight_trace_spans table NULL NULL NULL
crdb_internal node_metrics table NULL NULL NULL
crdb_internal node_queries table NULL NULL NULL
crdb_internal node_runtime_info table NULL NULL NULL
Expand Down Expand Up @@ -260,6 +261,11 @@ SELECT * FROM crdb_internal.zones WHERE false
zone_id subzone_id target range_name database_name table_name index_name partition_name
raw_config_yaml raw_config_sql raw_config_protobuf full_config_yaml full_config_sql

query IIITTT colnames
SELECT * FROM crdb_internal.node_inflight_trace_spans WHERE span_id < 0
----
trace_id parent_span_id span_id start_time duration operation

statement error not fully contained in tenant keyspace
SELECT * FROM crdb_internal.ranges WHERE range_id < 0

Expand Down
1 change: 1 addition & 0 deletions pkg/sql/logictest/testdata/logic_test/grant_table
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ test crdb_internal kv_node_status public
test crdb_internal kv_store_status public SELECT
test crdb_internal leases public SELECT
test crdb_internal node_build_info public SELECT
test crdb_internal node_inflight_trace_spans public SELECT
test crdb_internal node_metrics public SELECT
test crdb_internal node_queries public SELECT
test crdb_internal node_runtime_info public SELECT
Expand Down
5 changes: 5 additions & 0 deletions pkg/sql/logictest/testdata/logic_test/information_schema
View file
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,7 @@ crdb_internal kv_node_status
crdb_internal kv_store_status
crdb_internal leases
crdb_internal node_build_info
crdb_internal node_inflight_trace_spans
crdb_internal node_metrics
crdb_internal node_queries
crdb_internal node_runtime_info
Expand Down Expand Up @@ -461,6 +462,7 @@ kv_node_status
kv_store_status
leases
node_build_info
node_inflight_trace_spans
node_metrics
node_queries
node_runtime_info
Expand Down Expand Up @@ -640,6 +642,7 @@ system crdb_internal kv_node_status SYSTEM
system crdb_internal kv_store_status SYSTEM VIEW NO 1
system crdb_internal leases SYSTEM VIEW NO 1
system crdb_internal node_build_info SYSTEM VIEW NO 1
system crdb_internal node_inflight_trace_spans SYSTEM VIEW NO 1
system crdb_internal node_metrics SYSTEM VIEW NO 1
system crdb_internal node_queries SYSTEM VIEW NO 1
system crdb_internal node_runtime_info SYSTEM VIEW NO 1
Expand Down Expand Up @@ -1880,6 +1883,7 @@ NULL public system crdb_internal kv_node_status
NULL public system crdb_internal kv_store_status SELECT NULL YES
NULL public system crdb_internal leases SELECT NULL YES
NULL public system crdb_internal node_build_info SELECT NULL YES
NULL public system crdb_internal node_inflight_trace_spans SELECT NULL YES
NULL public system crdb_internal node_metrics SELECT NULL YES
NULL public system crdb_internal node_queries SELECT NULL YES
NULL public system crdb_internal node_runtime_info SELECT NULL YES
Expand Down Expand Up @@ -2265,6 +2269,7 @@ NULL public system crdb_internal kv_node_status
NULL public system crdb_internal kv_store_status SELECT NULL YES
NULL public system crdb_internal leases SELECT NULL YES
NULL public system crdb_internal node_build_info SELECT NULL YES
NULL public system crdb_internal node_inflight_trace_spans SELECT NULL YES
NULL public system crdb_internal node_metrics SELECT NULL YES
NULL public system crdb_internal node_queries SELECT NULL YES
NULL public system crdb_internal node_runtime_info SELECT NULL YES
Expand Down
Loading

0 comments on commit 3699ce6

Please sign in to comment.