best practice of workflow

[HaveF]

Hi, Romain,

I love you repo! ❤️ very useful!

I'm curious about your workflow. Could you please give me an overview?
When I'm debugging, I find myself manually closing the server with ctrl+c every time I adjust the metadata.yml file. Then I have to restart it and refresh the page to see if the configuration is appropriate. If it's not, I have to repeat the previous steps.
I'm not sure if there's a way to automate this entire process, such as automatically refreshing the page. I'm wondering how you handle it.
Thanks again!

[HaveF]

In general, when using vega lite, the default format is JSON. Do we have a way to use the Vega-lite specification directly? If the specification is complex, converting it to YAML can feel a bit cumbersome -- while many times you can find tools to convert from YAML to JSON, there will be a lot of manual YAML <<-->> JSON conversion process if you consider that debugging is required

[rclement]

Thanks @HaveF for your appreciation of datasette-dashboards and for the good points related to the usage workflow.
I took the liberty of converting this issue into a discussion as this seems more appropriate.

Here are some pointers:

I'm curious about your workflow. Could you please give me an overview?
When I'm debugging, I find myself manually closing the server with ctrl+c every time I adjust the metadata.yml file. Then I have to restart it and refresh the page to see if the configuration is appropriate. If it's not, I have to repeat the previous steps.
I'm not sure if there's a way to automate this entire process, such as automatically refreshing the page. I'm wondering how you handle it.

First thing you can do is to start the Datasette server with live-reload enabled, e.g. datasette --reload mydata.db --metadata metadata.yml. This way you will be able to update your metadata.yml / metadata.json and datasette will reload itself.

Also, here are some thoughts to improve the workflow:

• Make a chart and dashboard builder like a BI tool (Metabase, PBI, Tableau, etc.): this would require a great amount of work to make this happen
• Edit the metadata file directly from Datasette (cf. Updating metadata.json on Datasette for MacOS simonw/datasette#1791): this would allow to even use the plugin with standalone Datasette versions (Desktop, Lite), but there is an ongoing discussion about the use of Settings vs Metadata for plugins, and this should be sorted out for Datasette 1.0 so we must wait for this to happen

In general, when using vega lite, the default format is JSON. Do we have a way to use the Vega-lite specification directly? If the specification is complex, converting it to YAML can feel a bit cumbersome -- while many times you can find tools to convert from YAML to JSON, there will be a lot of manual YAML <<-->> JSON conversion process if you consider that debugging is required

Actually, you can do 2 things right now:

• From a Datasette dashboard chart use Vega or Vega-Lite, click on the top-right action button and click on "Open in Vega Editor": from there you can experiment more easily with your graph (this even works on localhost as the Vega Editor is making client-side calls using XHR)
• You can actually embed JSON into a YAML document, for instance:

...
          library: vega-lite
          display:
            {
              "encoding": {
                "x": {"field": "year", "timeUnit": "year", "type": "temporal"},
                "y": {"field": "count", "type": "quantitative"}
              },
              "mark": {"tooltip": true, "type": "line"}
            }
...

[HaveF]

Hi, Romain @rclement

Thanks for your detailed explanation.

          display:
            {
              "encoding": {
                "x": {"field": "year", "timeUnit": "year", "type": "temporal"},
                "y": {"field": "count", "type": "quantitative"}
              },
              "mark": {"tooltip": true, "type": "line"}
            }

It's great! It is exactly what I want.

datasette --reload mydata.db --metadata metadata.yml

It's a secret option... I also specifically checked for the relevant command line arguments, but I didn't find them.

╰─$ datasette --version
datasette, version 0.64.3
╰─$ datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect    Generate JSON summary of provided database files
  install    Install plugins and packages from PyPI into the same...
  package    Package SQLite files into a Datasette Docker container
  plugins    List currently installed plugins
  publish    Publish specified SQLite database files to the internet...
  uninstall  Uninstall plugins and Python packages from the Datasette...

Oh, it become a sub option of serve. It make sense.

[rclement]

Indeed the --reload option is part of the serve command, good catch!

[HaveF]

Make a chart and dashboard builder like a BI tool (Metabase, PBI, Tableau, etc.): this would require a great amount of work to make this happen

Btw, have you ever heard of https://github.com/Kanaries/pygwalker? I suppose it would works. As far as I know, it based on vega/vega lite ecosystem, it is like voyager, but it is a python package. I guess it can be embedded in your package with less work.

[rclement]

I came across Graphic Walker / Pygwalker a while ago, I thought of experimenting with it to integrate a query builder but never found the time to do it. From my understanding, Graphic Walker is more appropriate as this is just the client-side frontend to be plugged-in to whatever backend, while Pygwalker is specifically designed to work with Pandas Dataframes within a Notebook environment.

Good idea nevertheless!

[HaveF]

Btw, voyager can accept url, like your vega lite's csv url. I like it, you should try it directly! :D

http://vega.github.io/voyager/

[rclement]

Good to know! However, this builder is specific to Vega while Datasette Dashboards uses Vega / Vega-Lite only for some charts. Actually the end goal would be not to expose Vega / Vega-Lite directly (only for advanced users) and provide abstracted charts such as Line, Bar, Chroropleth, Pin map, etc.

[HaveF]

When I use your package today(Thank you again! ❤️ ), I don't know if I use layout right...Here is my workflow:

1. I create 2*2 grid layout first

[analysis-note, offers-day]
[analysis-note, offers-source]

2. Then I want to adjust it to 2*4

[analysis-note, offers-day, offers-day, offers-count]
[analysis-note, offers-source, offers-day, offers-count]

3. I had another idea to add a note-1 panel, but I wanted to be narrower, Ok, adjust it to 2*6

[analysis-note, offers-day, offers-day, offers-count, offers-count, note-1]
[analysis-note, offers-source, offers-day, offers-count, offers-count, note-1]

I don't know if I've made it clear, but I always have some new ideas when it comes to layout. During the adjustment, I kept copying the field to make it wider or narrower, and I always felt that my manual adjustment method was problematic, and sometimes because of copying the wrong field, I had to double-check what was wrong. Don't know if you have any better way?

Of course, the layout should be set at the beginning of the design of the dashboard, but it is inevitable to adjust it in use.

[rclement]

Datasette Dashboards layout system is already using grid-template-areas:

• What you specify in your layout dashboard attribute is directly translated to the grid-template-areas
• See dashboard_view.html:

  datasette-dashboards/datasette_dashboards/templates/dashboard_view.html

  Line 23 in 54a41a2

  grid-template-areas: {% for row in dashboard.layout %}"{% for col in row %}{{ col }} {% endfor %}" {% endfor %};

However what I did not know and I just tried is using the CSS . placeholder to specify blank spots in the layout and it works out-of-the-box:

plugins:
  datasette-dashboards:
    job-offers-stats:
      title: Job offers statistics
      layout:
        - [analysis-note, offers-day, offers-day, offers-count]
        - [analysis-note, offers-source, offers-day-source, offers-region]
        - [job-titles, offers-map, ., latest-offers]

I will add a note in the README / documentation about this neat feature!

[rclement]

I opened a PR #149 with updated layout examples in the README

[HaveF]

I probably mainly want to emphasize that adjusting the layout is not so convenient and intuitive. But it doesn't seem to be too important.
Also, congratulations on discovering the '.' operator! This placeholder is helpful for the layout layout in advance.

[rclement]

Thanks for your feedback. Indeed the current layout syntax might not be the greatest but it is easy, easily translatable and compatible with native CSS Grids.

Regarding your initial use-case:

I don't know if I've made it clear, but I always have some new ideas when it comes to layout. During the adjustment, I kept copying the field to make it wider or narrower, and I always felt that my manual adjustment method was problematic, and sometimes because of copying the wrong field, I had to double-check what was wrong. Don't know if you have any better way?

Of course, the layout should be set at the beginning of the design of the dashboard, but it is inevitable to adjust it in use.

Using the . placeholder you can easily experiment with complex and varying grids without having to extend previous charts, so that's a workflow enhancement in a sense

[HaveF]

I completely agree with your opinion and taste. Thank you and your plugins again ❤️

[janmechtel]

Thanks for this discussion guys, it helped me a lot, I really like the dashboard plugin + vega.

In the datasette plugin directory I also came across this: https://datasette.io/plugins/datasette-nteract-data-explorer

Might be helpful for you to know

[HaveF]

Thanks. It looks nice.
But I guess it is a bit hard to reuse the explorer result in the datasette-dashboards