Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/dashboard #227

Merged
merged 49 commits into from
Oct 27, 2023
Merged
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7c8e5ca
update dashboard workflow
ruthenian8 Sep 5, 2023
bbd2b3d
run on workflow_dispatch
ruthenian8 Sep 6, 2023
6712898
trigger on PR to dev for debugging
ruthenian8 Sep 7, 2023
941a581
update build push action
ruthenian8 Sep 7, 2023
5d7c753
update context && file
ruthenian8 Sep 7, 2023
eaf772a
remove trigger on PR
ruthenian8 Sep 7, 2023
27170ea
add sample data provider
ruthenian8 Sep 12, 2023
0531c49
Update dashboard, finalize data provider
ruthenian8 Sep 14, 2023
97b67c3
Add extractors for requests and responses as default instrumentors
ruthenian8 Sep 15, 2023
c180182
add 'get_last_request', 'get_last_response'; add charts for requests …
ruthenian8 Sep 15, 2023
92318cd
Update sections in superset_guide; update extractor functions tutor
ruthenian8 Sep 18, 2023
e45c298
doc build fix: Change functions to func names in __all__
ruthenian8 Sep 18, 2023
ac00230
Apply suggestions by RLKRO from code review
ruthenian8 Sep 20, 2023
1d5a92a
intermediary update; update data provider tutor; update charts;
ruthenian8 Sep 20, 2023
ce2f1d7
Apply suggestions from code review by RLKRO
ruthenian8 Sep 20, 2023
ee86703
Upload annotations.png via GUI
ruthenian8 Sep 20, 2023
721593e
Update requests & responses charts; update superset_guide
ruthenian8 Sep 20, 2023
fd05a11
expand the Superset guide with chart creation instructions;Add links …
ruthenian8 Sep 21, 2023
675e71b
update graph plot; update dataset to get flow lag; update data provider
ruthenian8 Sep 21, 2023
6af9a9f
update guide; move sample data collector to /utils
ruthenian8 Sep 22, 2023
532597e
update links to avoid sphinx warnings
ruthenian8 Sep 22, 2023
ee61909
merge dev
ruthenian8 Sep 26, 2023
6ace191
update dashboard description; update tutorials
ruthenian8 Sep 26, 2023
c6f1259
update superset guide
ruthenian8 Sep 26, 2023
6646577
Merge branch 'dev' into fix/dashboard
ruthenian8 Sep 26, 2023
17225e1
update label extractor; add infile parameter to main; remove link to …
ruthenian8 Sep 26, 2023
d355290
update tests
ruthenian8 Sep 26, 2023
eddc688
update format
ruthenian8 Sep 26, 2023
3c15511
minor doc improvements
RLKRo Oct 5, 2023
457a50c
update dashboard datasource config
ruthenian8 Oct 6, 2023
a470da3
add dff_stats dataset; link charts to dff_stats dataset
ruthenian8 Oct 10, 2023
49d2e8a
update tests and documentation to reflect config changes
ruthenian8 Oct 10, 2023
ed2db47
update formatting
ruthenian8 Oct 10, 2023
d9265ca
add the test configuration file
ruthenian8 Oct 10, 2023
0809a48
merge remote dev; resolve conflicts
ruthenian8 Oct 18, 2023
af92f36
move get_current_label to after_handler in sample_data_provider.py
RLKRo Oct 19, 2023
9ae731e
update sql with if statements
ruthenian8 Oct 20, 2023
cd0fc8e
specify healthcheck for clickhouse and wait for healthcheck with otlp…
ruthenian8 Oct 20, 2023
c53ab31
add dashboard-metadata docker image; adjust to run on port 5433
ruthenian8 Oct 20, 2023
32ce2ec
Update .github/workflows/update_dashboard.yml
ruthenian8 Oct 24, 2023
9bfaeac
Update dff/stats/cli.py
ruthenian8 Oct 24, 2023
d73b8d4
remove query statements from dashboard files; remove filter values fr…
ruthenian8 Oct 24, 2023
e8e725d
wait for pg database; sleep to avoid process conflict
ruthenian8 Oct 25, 2023
ec7cf06
Apply suggestions by @RLKRo
ruthenian8 Oct 26, 2023
fb3cf72
Update config files; Update sql expressions in CLI; update healthchec…
ruthenian8 Oct 26, 2023
fa5d550
convert request_id to int in final_nodes
RLKRo Oct 26, 2023
3fad9d6
remove sql in final_nodes
RLKRo Oct 26, 2023
464a2bd
update history filter to numerical range type & change datasource to …
RLKRo Oct 26, 2023
c329025
remove utils include from manifest
RLKRo Oct 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
specify healthcheck for clickhouse and wait for healthcheck with otlp…
…; make a variable for superset metadata db; adjust dockerfile to use configuration overrides
ruthenian8 committed Oct 20, 2023

Verified

This commit was signed with the committer’s verified signature.
sreuland shawn
commit cd0fc8e2ada91f9ab6de607670e93e69ac77a93c
3 changes: 2 additions & 1 deletion .env_file
Original file line number Diff line number Diff line change
@@ -19,4 +19,5 @@ CLICKHOUSE_USER=username
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
CLICKHOUSE_PASSWORD=pass
SUPERSET_USERNAME=superset
SUPERSET_PASSWORD=superset
SUPERSET_PASSWORD=superset
SUPERSET_METADATA_DB=test
3 changes: 2 additions & 1 deletion dff/utils/docker/dockerfile_stats
Original file line number Diff line number Diff line change
@@ -2,6 +2,7 @@ FROM apache/superset:2.1.0rc1
USER root
RUN cd /app && pip install .[clickhouse]
COPY entrypoint_stats.sh /app/docker/
COPY superset_config_docker.py /app/pythonpath/
COPY --chown=superset superset_config_docker.py /app/pythonpath/
ENV SUPERSET_CONFIG_PATH /app/pythonpath/superset_config_docker.py
USER superset
ENTRYPOINT ["/bin/bash","/app/docker/entrypoint_stats.sh"]
2 changes: 1 addition & 1 deletion dff/utils/docker/superset_config_docker.py
ruthenian8 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -25,6 +25,6 @@
import os

SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://{0}:{1}@psql:5432/{2}".format(
os.getenv("POSTGRES_USERNAME"), os.getenv("POSTGRES_PASSWORD"), os.getenv("POSTGRES_DB")
os.getenv("POSTGRES_USERNAME"), os.getenv("POSTGRES_PASSWORD"), os.getenv("SUPERSET_METADATA_DB")
)
SQLALCHEMY_ECHO = True
14 changes: 11 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -52,8 +52,10 @@ services:
dockerfile: dockerfile_stats
image: ghcr.io/deeppavlov/superset_df_dashboard:latest
ruthenian8 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should build a local image in tests instead of using a remote image.
(as mentioned here)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be addressed by passing the --build flag to the docker-compose up command

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently, it was already building instead of pulling an image in previous commits:
image

I don't know why tests didn't fail due to authentication errors.
Maybe because context storages were also launched it gave enough time to whatever dashboard awaits for.

depends_on:
- 'psql'
- 'clickhouse'
psql:
condition: service_started
clickhouse:
condition: service_started
profiles:
- stats
ports:
@@ -70,6 +72,11 @@ services:
- '9000:9000'
volumes:
- ch-data:/var/lib/clickhouse/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's weird that only clickhouse has its own volume.
Why not give a volume to all db containers?

Restarting all containers still resets all the metadata stored in dashboard-metadata.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that adding volumes to all containers lies outside the scope of this PR, so for now I will only add volumes for clickhouse and dashboard-metadata

healthcheck:
test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
interval: 2s
timeout: 2s
retries: 5
RLKRo marked this conversation as resolved.
Show resolved Hide resolved
otelcol:
image: otel/opentelemetry-collector-contrib:latest
profiles:
@@ -78,7 +85,8 @@ services:
restart: unless-stopped
command: [ "--config=/etc/otelcol-config.yml", "--config=/etc/otelcol-config-extras.yml" ]
depends_on:
- 'clickhouse'
clickhouse:
condition: service_healthy
volumes:
- ./dff/utils/otel/otelcol-config.yml:/etc/otelcol-config.yml:ro
- ./dff/utils/otel/otelcol-config-extras.yml:/etc/otelcol-config-extras.yml:ro
8 changes: 3 additions & 5 deletions docs/source/user_guides/superset_guide.rst
Original file line number Diff line number Diff line change
@@ -174,12 +174,10 @@ a separate menu for data filters will also be available. If you need to use the
from the `data` column, you will need to find the `custom_sql` option when adding the column
and put in the extraction expression, as shown in the examples above.

**Persisting the chart configuration**
**Exporting the chart configuration**

If you define your own charts, it's important to save their configuration to the file system of your
host machine, so that no information is lost when the Superset container or the Docker process is restarted.

The most convenient way to do that is to export the dashboard configuration as a whole. Navigate to the
The configuration of a Superset dashboard can be easily exported and then reused
in other Superset instances. This can be done using the GUI: navigate to the
`Dashboards` section of the Superset application, locate your dashboard (named `DFF statistics` per default).
Then press the `export` button on the right and save the zip file to any convenient location.