Skip to content

Commit

Permalink
sql: use time-based sampling for logical plans
Browse files Browse the repository at this point in the history
Previously, the rate at which we sampled logical plans was once every 1000th
time that fingerprint was executed. This diff changes the sampling rate from
count-based (once every 1000th execution) to a time-based cluster setting
(the default is set to once every 5 minutes).

This solves the following:
- For fingerprints with high execution counts, this prevent us from sampling
  too often and creating unnecessary overhead.
- For fingerprints with lower execution counts, this allows us to refresh
  plans more frequently, which could be helpful if a user wants to verify
  that a configuration change has improved a fingerprint's logical plan and
  execution time.
- Because stats are stored per node, having a timestamp additionally means
  that the UI can now determine and surface the plan that's the most
  recent across all nodes.
- This also solves an edge case where statementStatistics might exist
  without a sample, which can happen if user has set
  `sql.metrics.statement_details.plan_collection.enabled` to false, or if the
  stats were collected before this feature was introduced.

Release note: None
  • Loading branch information
celiala committed Jan 29, 2019
1 parent 7c3e69c commit 3b64a74
Show file tree
Hide file tree
Showing 5 changed files with 130 additions and 73 deletions.
3 changes: 2 additions & 1 deletion docs/generated/settings/settings.html
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@
<tr><td><code>sql.distsql.temp_storage.workmem</code></td><td>byte size</td><td><code>64 MiB</code></td><td>maximum amount of memory in bytes a processor can use before falling back to temp storage</td></tr>
<tr><td><code>sql.metrics.statement_details.dump_to_logs</code></td><td>boolean</td><td><code>false</code></td><td>dump collected statement statistics to node logs when periodically cleared</td></tr>
<tr><td><code>sql.metrics.statement_details.enabled</code></td><td>boolean</td><td><code>true</code></td><td>collect per-statement query statistics</td></tr>
<tr><td><code>sql.metrics.statement_details.sample_logical_plans</code></td><td>boolean</td><td><code>true</code></td><td>periodically save a logical plan for each fingerprint</td></tr>
<tr><td><code>sql.metrics.statement_details.plan_collection.enabled</code></td><td>boolean</td><td><code>true</code></td><td>periodically save a logical plan for each fingerprint</td></tr>
<tr><td><code>sql.metrics.statement_details.plan_collection.period</code></td><td>duration</td><td><code>5m0s</code></td><td>the time until a new logical plan is collected</td></tr>
<tr><td><code>sql.metrics.statement_details.threshold</code></td><td>duration</td><td><code>0s</code></td><td>minimum execution time to cause statistics to be collected</td></tr>
<tr><td><code>sql.parallel_scans.enabled</code></td><td>boolean</td><td><code>true</code></td><td>parallelizes scanning different ranges when the maximum result size can be deduced</td></tr>
<tr><td><code>sql.query_cache.enabled</code></td><td>boolean</td><td><code>false</code></td><td>enable the query cache</td></tr>
Expand Down
174 changes: 112 additions & 62 deletions pkg/roachpb/app_stats.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions pkg/roachpb/app_stats.proto
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ package cockroach.sql;
option go_package = "roachpb";

import "gogoproto/gogo.proto";
import "google/protobuf/timestamp.proto";

message StatementStatistics {
// Count is the total number of times this statement was executed
Expand Down Expand Up @@ -88,6 +89,9 @@ message SensitiveInfo {

// MostRecentPlanDescription is a serialized representation of the logical plan most recently captured for this query.
optional ExplainTreePlanNode most_recent_plan_description = 2 [(gogoproto.nullable) = false];

// Timestamp is the time at which the logical plan was last sampled.
optional google.protobuf.Timestamp most_recent_plan_timestamp = 3 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
}

message NumericStat {
Expand Down
Loading

0 comments on commit 3b64a74

Please sign in to comment.