Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/dashboard #227

Merged
merged 49 commits into from
Oct 27, 2023
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7c8e5ca
update dashboard workflow
ruthenian8 Sep 5, 2023
bbd2b3d
run on workflow_dispatch
ruthenian8 Sep 6, 2023
6712898
trigger on PR to dev for debugging
ruthenian8 Sep 7, 2023
941a581
update build push action
ruthenian8 Sep 7, 2023
5d7c753
update context && file
ruthenian8 Sep 7, 2023
eaf772a
remove trigger on PR
ruthenian8 Sep 7, 2023
27170ea
add sample data provider
ruthenian8 Sep 12, 2023
0531c49
Update dashboard, finalize data provider
ruthenian8 Sep 14, 2023
97b67c3
Add extractors for requests and responses as default instrumentors
ruthenian8 Sep 15, 2023
c180182
add 'get_last_request', 'get_last_response'; add charts for requests …
ruthenian8 Sep 15, 2023
92318cd
Update sections in superset_guide; update extractor functions tutor
ruthenian8 Sep 18, 2023
e45c298
doc build fix: Change functions to func names in __all__
ruthenian8 Sep 18, 2023
ac00230
Apply suggestions by RLKRO from code review
ruthenian8 Sep 20, 2023
1d5a92a
intermediary update; update data provider tutor; update charts;
ruthenian8 Sep 20, 2023
ce2f1d7
Apply suggestions from code review by RLKRO
ruthenian8 Sep 20, 2023
ee86703
Upload annotations.png via GUI
ruthenian8 Sep 20, 2023
721593e
Update requests & responses charts; update superset_guide
ruthenian8 Sep 20, 2023
fd05a11
expand the Superset guide with chart creation instructions;Add links …
ruthenian8 Sep 21, 2023
675e71b
update graph plot; update dataset to get flow lag; update data provider
ruthenian8 Sep 21, 2023
6af9a9f
update guide; move sample data collector to /utils
ruthenian8 Sep 22, 2023
532597e
update links to avoid sphinx warnings
ruthenian8 Sep 22, 2023
ee61909
merge dev
ruthenian8 Sep 26, 2023
6ace191
update dashboard description; update tutorials
ruthenian8 Sep 26, 2023
c6f1259
update superset guide
ruthenian8 Sep 26, 2023
6646577
Merge branch 'dev' into fix/dashboard
ruthenian8 Sep 26, 2023
17225e1
update label extractor; add infile parameter to main; remove link to …
ruthenian8 Sep 26, 2023
d355290
update tests
ruthenian8 Sep 26, 2023
eddc688
update format
ruthenian8 Sep 26, 2023
3c15511
minor doc improvements
RLKRo Oct 5, 2023
457a50c
update dashboard datasource config
ruthenian8 Oct 6, 2023
a470da3
add dff_stats dataset; link charts to dff_stats dataset
ruthenian8 Oct 10, 2023
49d2e8a
update tests and documentation to reflect config changes
ruthenian8 Oct 10, 2023
ed2db47
update formatting
ruthenian8 Oct 10, 2023
d9265ca
add the test configuration file
ruthenian8 Oct 10, 2023
0809a48
merge remote dev; resolve conflicts
ruthenian8 Oct 18, 2023
af92f36
move get_current_label to after_handler in sample_data_provider.py
RLKRo Oct 19, 2023
9ae731e
update sql with if statements
ruthenian8 Oct 20, 2023
cd0fc8e
specify healthcheck for clickhouse and wait for healthcheck with otlp…
ruthenian8 Oct 20, 2023
c53ab31
add dashboard-metadata docker image; adjust to run on port 5433
ruthenian8 Oct 20, 2023
32ce2ec
Update .github/workflows/update_dashboard.yml
ruthenian8 Oct 24, 2023
9bfaeac
Update dff/stats/cli.py
ruthenian8 Oct 24, 2023
d73b8d4
remove query statements from dashboard files; remove filter values fr…
ruthenian8 Oct 24, 2023
e8e725d
wait for pg database; sleep to avoid process conflict
ruthenian8 Oct 25, 2023
ec7cf06
Apply suggestions by @RLKRo
ruthenian8 Oct 26, 2023
fb3cf72
Update config files; Update sql expressions in CLI; update healthchec…
ruthenian8 Oct 26, 2023
fa5d550
convert request_id to int in final_nodes
RLKRo Oct 26, 2023
3fad9d6
remove sql in final_nodes
RLKRo Oct 26, 2023
464a2bd
update history filter to numerical range type & change datasource to …
RLKRo Oct 26, 2023
c329025
remove utils include from manifest
RLKRo Oct 26, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions .github/workflows/update_dashboard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:
paths:
- 'dff/utils/docker/dockerfile_stats'
- 'dff/utils/docker/entrypoint_stats.sh'
ruthenian8 marked this conversation as resolved.
Show resolved Hide resolved
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -33,13 +34,16 @@ jobs:
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository }}/superset_df_dashboard
flavor: |
latest=auto
tags: |
type=ref,event=branch
type=semver,event={{version}}
type=ref,event=tag
type=raw,value=latest,enable={{is_default_branch}}

- name: Build and upload image
uses: docker/build-push-action@v4
uses: docker/build-push-action@v4
with:
context: dff/utils/docker
file: dff/utils/docker/dockerfile_stats
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
ruthenian8 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
slice_name: Current topic slot [bar chart]
description: null
certified_by: null
certification_details: null
viz_type: dist_bar
params:
datasource: 2__table
viz_type: dist_bar
slice_id: 6
granularity_sqla: start_time
time_range: No filter
metrics:
- count
adhoc_filters:
- clause: WHERE
comparator: get_slots
datasourceWarning: false
expressionType: SIMPLE
filterOptionName: filter_sz14lhn7d1d_c0zqn5dpgk
isExtra: false
isNew: false
operator: ==
operatorId: EQUALS
sqlExpression: null
subject: data_key
- clause: WHERE
comparator: null
datasourceWarning: false
expressionType: SQL
filterOptionName: filter_945dhn41x2m_sci7gkxy7o
isExtra: false
isNew: false
operator: null
sqlExpression: JSON_VALUE(data, '$.current_topic') <> ''
subject: null
groupby:
- request_id
columns:
- label: My column
sqlExpression: JSON_VALUE(data, '$.current_topic')
expressionType: SQL
row_limit: 10000
order_desc: true
color_scheme: echarts4Colors
show_legend: true
rich_tooltip: true
bar_stacked: true
order_bars: false
y_axis_format: SMART_NUMBER
y_axis_label: Topic counts
y_axis_bounds:
- null
- null
x_axis_label: Dialog turn
bottom_margin: auto
x_ticks_layout: auto
extra_form_data: {}
dashboards:
- 1
query_context: '{"datasource":{"id":2,"type":"table"},"force":false,"queries":[{"time_range":"No
filter","granularity":"start_time","filters":[{"col":"data_key","op":"==","val":"get_slots"}],"extras":{"having":"","where":"(JSON_VALUE(data,
''$.current_topic'') <> '''')"},"applied_time_extras":{},"columns":["request_id",{"label":"My
column","sqlExpression":"JSON_VALUE(data, ''$.current_topic'')","expressionType":"SQL"}],"metrics":["count"],"annotation_layers":[],"row_limit":10000,"series_limit":0,"order_desc":true,"url_params":{},"custom_params":{},"custom_form_data":{}}],"form_data":{"datasource":"2__table","viz_type":"dist_bar","slice_id":6,"granularity_sqla":"start_time","time_range":"No
filter","metrics":["count"],"adhoc_filters":[{"clause":"WHERE","comparator":"get_slots","datasourceWarning":false,"expressionType":"SIMPLE","filterOptionName":"filter_sz14lhn7d1d_c0zqn5dpgk","isExtra":false,"isNew":false,"operator":"==","operatorId":"EQUALS","sqlExpression":null,"subject":"data_key"},{"clause":"WHERE","comparator":null,"datasourceWarning":false,"expressionType":"SQL","filterOptionName":"filter_945dhn41x2m_sci7gkxy7o","isExtra":false,"isNew":false,"operator":null,"sqlExpression":"JSON_VALUE(data,
''$.current_topic'') <> ''''","subject":null}],"groupby":["request_id"],"columns":[{"label":"My
column","sqlExpression":"JSON_VALUE(data, ''$.current_topic'')","expressionType":"SQL"}],"row_limit":10000,"order_desc":true,"color_scheme":"echarts4Colors","show_legend":true,"rich_tooltip":true,"bar_stacked":true,"order_bars":false,"y_axis_format":"SMART_NUMBER","y_axis_label":"Topic
counts","y_axis_bounds":[null,null],"x_axis_label":"Dialog turn","bottom_margin":"auto","x_ticks_layout":"auto","extra_form_data":{},"dashboards":[1],"force":false,"result_format":"json","result_type":"full"},"result_format":"json","result_type":"full"}'
cache_timeout: null
uuid: a70c05d0-770b-4068-a55d-934283f5b1bb
version: 1.0.0
dataset_uuid: fda98ab8-f550-45f1-9ded-0113f3e67260
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
slice_name: Current topic [time series bar chart]
description: null
certified_by: null
certification_details: null
viz_type: echarts_timeseries_bar
params:
datasource: 2__table
viz_type: echarts_timeseries_bar
slice_id: 15
granularity_sqla: start_time
time_grain_sqla: PT1M
time_range: No filter
metrics:
- count
groupby:
- expressionType: SQL
label: context_id
sqlExpression: JSON_VALUE(data, '$.current_topic')
adhoc_filters:
- clause: WHERE
comparator: get_slots
datasourceWarning: false
expressionType: SIMPLE
filterOptionName: filter_8tft5fr07ea_urtdezftgn
isExtra: false
isNew: false
operator: ==
operatorId: EQUALS
sqlExpression: null
subject: data_key
- clause: WHERE
comparator: null
datasourceWarning: false
expressionType: SQL
filterOptionName: filter_4ffdpny1zzm_vmlo11yw7i
isExtra: false
isNew: false
operator: null
sqlExpression: JSON_VALUE(data, '$.current_topic') <> ''
subject: null
order_desc: true
row_limit: 10000
truncate_metric: true
show_empty_columns: true
comparison_type: values
annotation_layers: []
forecastPeriods: 10
forecastInterval: 0.8
orientation: vertical
x_axis_title: Time axis
x_axis_title_margin: 30
y_axis_title: Value counts over time
y_axis_title_margin: 30
y_axis_title_position: Left
color_scheme: echarts4Colors
show_value: false
stack: true
only_total: true
show_legend: true
legendType: scroll
legendOrientation: top
x_axis_time_format: smart_date
y_axis_format: SMART_NUMBER
y_axis_bounds:
- null
- null
rich_tooltip: true
tooltipSortByMetric: true
tooltipTimeFormat: smart_date
extra_form_data: {}
dashboards:
- 1
query_context: '{"datasource":{"id":2,"type":"table"},"force":false,"queries":[{"time_range":"No
filter","granularity":"start_time","filters":[{"col":"data_key","op":"==","val":"get_slots"}],"extras":{"time_grain_sqla":"PT1M","having":"","where":"(JSON_VALUE(data,
''$.current_topic'') <> '''')"},"applied_time_extras":{},"columns":[{"expressionType":"SQL","label":"context_id","sqlExpression":"JSON_VALUE(data,
''$.current_topic'')"}],"metrics":["count"],"orderby":[["count",false]],"annotation_layers":[],"row_limit":10000,"series_columns":[{"expressionType":"SQL","label":"context_id","sqlExpression":"JSON_VALUE(data,
''$.current_topic'')"}],"series_limit":0,"order_desc":true,"url_params":{},"custom_params":{},"custom_form_data":{},"is_timeseries":true,"time_offsets":[],"post_processing":[{"operation":"pivot","options":{"index":["__timestamp"],"columns":["context_id"],"aggregates":{"count":{"operator":"mean"}},"drop_missing_columns":false}},{"operation":"rename","options":{"columns":{"count":null},"level":0,"inplace":true}},{"operation":"flatten"}]}],"form_data":{"datasource":"2__table","viz_type":"echarts_timeseries_bar","slice_id":15,"granularity_sqla":"start_time","time_grain_sqla":"PT1M","time_range":"No
filter","metrics":["count"],"groupby":[{"expressionType":"SQL","label":"context_id","sqlExpression":"JSON_VALUE(data,
''$.current_topic'')"}],"adhoc_filters":[{"clause":"WHERE","comparator":"get_slots","datasourceWarning":false,"expressionType":"SIMPLE","filterOptionName":"filter_8tft5fr07ea_urtdezftgn","isExtra":false,"isNew":false,"operator":"==","operatorId":"EQUALS","sqlExpression":null,"subject":"data_key"},{"clause":"WHERE","comparator":null,"datasourceWarning":false,"expressionType":"SQL","filterOptionName":"filter_4ffdpny1zzm_vmlo11yw7i","isExtra":false,"isNew":false,"operator":null,"sqlExpression":"JSON_VALUE(data,
''$.current_topic'') <> ''''","subject":null}],"order_desc":true,"row_limit":10000,"truncate_metric":true,"show_empty_columns":true,"comparison_type":"values","annotation_layers":[],"forecastPeriods":10,"forecastInterval":0.8,"orientation":"vertical","x_axis_title":"Time
axis","x_axis_title_margin":30,"y_axis_title":"Value counts over time","y_axis_title_margin":30,"y_axis_title_position":"Left","color_scheme":"echarts4Colors","show_value":false,"stack":true,"only_total":true,"show_legend":true,"legendType":"scroll","legendOrientation":"top","x_axis_time_format":"smart_date","y_axis_format":"SMART_NUMBER","y_axis_bounds":[null,null],"rich_tooltip":true,"tooltipSortByMetric":true,"tooltipTimeFormat":"smart_date","extra_form_data":{},"dashboards":[1],"force":false,"result_format":"json","result_type":"full"},"result_format":"json","result_type":"full"}'
cache_timeout: null
uuid: f8215b4d-cdaf-489a-90b2-040da840ab35
version: 1.0.0
dataset_uuid: fda98ab8-f550-45f1-9ded-0113f3e67260

This file was deleted.

Loading