Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cribl Thruput Summary Dashboard & Reports #39

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Cribl Thruput Summary Dashboard & Reports #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions criblvision-for-splunk/criblvision/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,13 @@ This app ships with a number of disabled alerts that can be used to alert when i

</table>

### Reports - Summary Indexing

This app contains the ability to summarize Cribl's internal metrics on data throughput in order to provide a historical reference for metrics that is readily accessible and performant. These searches summarize total bytes and events by input and output which allows for a look at per-component look at data thruput. Turn these searches on to enable use of the Cribl Thruput Summary dashboard:

* Summary Gen - Cribl Thruput - Inputs
* Summary Gen - Cribl Thruput - Outputs

### The CriblVision for Splunk Pack

The CriblVision for Splunk Pack is a companion to this Splunk app. It **is not** a requirement to use the Pack to receive value from this app. The Pack is only required if you would like to take advantage of the Collector Jobs that the Pack provides templates for.
Expand Down
179 changes: 179 additions & 0 deletions criblvision-for-splunk/criblvision/default/data/ui/views/cribl_thruput_summary.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
<form version="1.1" theme="dark">
<label>Cribl Thruput Summary</label>
<description>Cribl Thruput Stats by Input and Output. Note that to enable the summaries for this dashboard, you must enable the two Summary Indexing searches in this app that start with "Summary Gen - Cribl Thruput"</description>
<init>
<set token="timechart_span"></set>
</init>
<fieldset submitButton="false">
<input type="time" searchWhenChanged="true" token="global_time_tok">
<label>Time Range</label>
<default>
<earliest>-30d@d</earliest>
<latest>@d</latest>
</default>
</input>
<input type="dropdown" token="mstats_span">
<label>Time Span</label>
<choice value="auto">auto</choice>
<choice value="5s@s">5s</choice>
<choice value="10s@s">10s</choice>
<choice value="30s@s">30s</choice>
<choice value="1m@m">1m</choice>
<choice value="5m@m">5m</choice>
<choice value="10m@m">10m</choice>
<choice value="30m@m">30m</choice>
<choice value="1h@h">1h</choice>
<choice value="1d@d">1d</choice>
<choice value="1w@w">1w</choice>
<choice value="1mon@mon">1mon</choice>
<default>auto</default>
<change>
<eval token="timechart_span">if($value$ == "auto", "", "span=".$value$)</eval>
<eval token="mstats_span">"span=".$value$</eval>
</change>
<prefix>span=</prefix>
<initialValue>auto</initialValue>
</input>
</fieldset>
<row>
<panel>
<input type="multiselect" token="output_token" searchWhenChanged="true">
<label>Outputs</label>
<choice value="*">All</choice>
<valuePrefix>"</valuePrefix>
<valueSuffix>"</valueSuffix>
<delimiter>, </delimiter>
<default></default>
<initialValue></initialValue>
<fieldForLabel>field</fieldForLabel>
<fieldForValue>field</fieldForValue>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=output
| stats values(out_bytes_*) AS *
| fieldsummary
| table field</query>
<earliest>-48h@h</earliest>
<latest>now</latest>
</search>
</input>
<chart>
<title>Cribl Total Output Events</title>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=output
| timechart max(out_events_*) AS *
| table _time, $output_token$</query>
<earliest>$global_time_tok.earliest$</earliest>
<latest>$global_time_tok.latest$</latest>
</search>
<option name="charting.axisY.abbreviation">auto</option>
<option name="charting.axisY.minimumNumber">0</option>
<option name="charting.axisY2.abbreviation">auto</option>
<option name="charting.axisY2.enabled">1</option>
<option name="charting.chart">area</option>
<option name="charting.chart.nullValueMode">connect</option>
<option name="charting.chart.overlayFields">splunk_diff</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.drilldown">all</option>
<option name="charting.gridLinesX.showMajorLines">1</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>
<option name="charting.legend.mode">seriesCompare</option>
<option name="displayview">analytics_workspace</option>
<option name="refresh.display">progressbar</option>
</chart>
<chart>
<title>Cribl Total Output Bytes</title>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=output
| timechart max(out_bytes_*) AS *
| table _time, $output_token$</query>
<earliest>$global_time_tok.earliest$</earliest>
<latest>$global_time_tok.latest$</latest>
</search>
<option name="charting.axisLabelsY.majorUnit">auto</option>
<option name="charting.axisY.abbreviation">auto</option>
<option name="charting.axisY2.abbreviation">auto</option>
<option name="charting.axisY2.enabled">1</option>
<option name="charting.chart">area</option>
<option name="charting.chart.nullValueMode">connect</option>
<option name="charting.chart.overlayFields">splunk_diff</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.drilldown">all</option>
<option name="charting.gridLinesX.showMajorLines">1</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>
<option name="charting.legend.mode">seriesCompare</option>
<option name="displayview">analytics_workspace</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Inputs</title>
<input type="multiselect" token="input_token" searchWhenChanged="true">
<label>Inputs</label>
<valuePrefix>"</valuePrefix>
<valueSuffix>"</valueSuffix>
<delimiter>, </delimiter>
<default></default>
<initialValue></initialValue>
<fieldForLabel>field</fieldForLabel>
<fieldForValue>field</fieldForValue>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=input
| stats values(in_bytes_*) AS *
| fieldsummary
| table field</query>
<earliest>-48h</earliest>
<latest>now</latest>
</search>
</input>
<chart>
<title>Cribl Total Input Events</title>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=input
| timechart max(in_events_*) AS *
| table _time, $input_token$</query>
<earliest>$global_time_tok.earliest$</earliest>
<latest>$global_time_tok.latest$</latest>
</search>
<option name="charting.axisLabelsY.majorUnit">auto</option>
<option name="charting.axisY.abbreviation">auto</option>
<option name="charting.chart">area</option>
<option name="charting.chart.nullValueMode">connect</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.drilldown">all</option>
<option name="charting.gridLinesX.showMajorLines">1</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>
<option name="charting.legend.mode">standard</option>
<option name="displayview">analytics_workspace</option>
<option name="refresh.display">progressbar</option>

</chart>
<chart>
<title>Cribl Total Input Bytes</title>
<search>
<query>`set_cribl_internal_log_index` source=cribl_thruput_summary component=input
| timechart max(in_bytes_*) AS *
| table _time, $input_token$</query>
<earliest>$global_time_tok.earliest$</earliest>
<latest>$global_time_tok.latest$</latest>
</search>
<option name="charting.axisLabelsY.majorUnit">auto</option>
<option name="charting.axisY.abbreviation">auto</option>
<option name="charting.chart">area</option>
<option name="charting.chart.nullValueMode">connect</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.drilldown">all</option>
<option name="charting.gridLinesX.showMajorLines">1</option>
<option name="charting.legend.mode">standard</option>
<option name="displayview">analytics_workspace</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
</form>
50 changes: 49 additions & 1 deletion criblvision-for-splunk/criblvision/default/savedsearches.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,4 +429,52 @@ request.ui_dispatch_app = criblvision
request.ui_dispatch_view = search
search = `set_cribl_internal_log_index` `set_cribl_log_sourcetype` message="restarting worker process"\
| stats count AS worker_process_restarts BY host instance_type worker_group\
| where worker_process_restarts > `set_alert_threshold_worker_process_restarts`
| where worker_process_restarts > `set_alert_threshold_worker_process_restarts`

### Summary Indexing Searches ###

[Summary Gen - Cribl Thruput - Inputs]
alert.track = 0
cron_schedule = 0 1 * * *
description = Generates a summary index of cribl thruput statistics for the inputs of cribl.
disabled = 1
dispatch.earliest_time = -1d@d
dispatch.latest_time = @d
enableSched = 1
request.ui_dispatch_app = criblvision
request.ui_dispatch_view = search
search = | mstats sum(`set_cribl_metrics_prefix(total.in_bytes)`) sum(`set_cribl_metrics_prefix(total.in_events)`) prestats=true WHERE `set_cribl_metrics_index` span=auto BY input\
| timechart span=1d limit=0 sum(`set_cribl_metrics_prefix(total.in_bytes)`) AS in_bytes sum(`set_cribl_metrics_prefix(total.in_events)`) AS in_events useother=false BY input\
| fields - _span*\
| rename *:* AS *_*\
| rename *:* AS *_*\
| rename *:* AS *_*\
| rename "* *" AS **\
| foreach in_bytes_kinesis* \
[eval in_bytes_kinesis=0, in_bytes_kinesis=sum(in_bytes_kinesis, <<FIELD>>)]\
| foreach in_events_kinesis* \
[eval in_events_kinesis=0, in_events_kinesis=sum(in_events_kinesis, <<FIELD>>)]\
| fields - in_*_kinesis_*\
| fillnull value=0\
| eval component="input"\
| collect `set_cribl_internal_log_index` source="cribl_thruput_summary"

[Summary Gen - Cribl Thruput - Outputs]
alert.track = 0
cron_schedule = 0 1 * * *
description = Generates a summary index of cribl thruput statistics for the outputs of cribl.
disabled = 1
dispatch.earliest_time = -1d@d
dispatch.latest_time = @d
enableSched = 1
request.ui_dispatch_app = criblvision
request.ui_dispatch_view = search
search = | mstats sum(`set_cribl_metrics_prefix(total.out_bytes)`) sum(`set_cribl_metrics_prefix(total.out_events)`) prestats=true WHERE `set_cribl_metrics_index` span=auto BY output\
| timechart span=1d limit=0 sum(`set_cribl_metrics_prefix(total.out_bytes)`) AS out_bytes sum(`set_cribl_metrics_prefix(total.out_events)`) AS out_events useother=false BY output\
| fields - _span*\
| rename *:* AS *_*\
| rename *:* AS *_*\
| rename "* *" AS **\
| fillnull value=0\
| eval component="output"\
| collect `set_cribl_internal_log_index` source="cribl_thruput_summary"\