Datawrapper loader example #1585
Conversation
There was a problem hiding this comment.
The example demonstrates the technique, but the resulting chart is not really showing the best of what datawrapper offers—why should a user invest so much effort into creating a basic bar chart? It might be worth looking for a more visually striking example, where going through datawrapper really makes sense (and shines).
A different way of using Datawrapper inside of Framework might be to use the datawrapper UX to create a chart, then export to an iframe. Maybe we just need a comment about this.
Another “tip” that we could add is that this approach creates a new chart in DataWrapper each time the data loader runs. Although it is unavoidable, it can be surprising.
|
|
||
| <div class="note"> | ||
|
|
||
| To run this data loader, you will need to create an API token in Datawrapper and set it as an environment variable named `DATAWRAPPER_ACCESS_TOKEN`. |
There was a problem hiding this comment.
It would be nice to have detailed instructions to create the venv. In particular since this repo expects the venv to live in .venv (judging from the .gitignore file).
I tried this:
python -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txtbut it failed with:
error: externally-managed-environment
× This environment is externally managed
To make this work I had to run the last command with the --break-system-packages option 😅:
python -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt --break-system-packages(Maybe it's just me—I appear to be doomed when I need to install python stuff.)
There was a problem hiding this comment.
I'm agnostic on Python package managers, so I'm open to handling this however you want.
My routine is to use pipenv. So, in my world, I run something like this:
pipenv install datawrapper pandas
pipenv run npm run devI put the requirements file in my proposals because I saw it that way in other examples and presumed it was your preference. It's not the canonical way to run pipenv, but it's supported, so here's how I did things in this case.
pipenv shell
pipenv install -r requirements.txtI don't have any experience with venv but there must be a way to make it work without resorting to hacks or unusual options. Let me know how you'd like to proceed.
There was a problem hiding this comment.
wow, I didn't know about pipenv; maybe it can save me from my personal python hell!
|
|
||
| <div class="note"> | ||
|
|
||
| To run this data loader, you will need to create an API token in Datawrapper and set it as an environment variable named `DATAWRAPPER_ACCESS_TOKEN`. |
There was a problem hiding this comment.
For the DATAWRAPPER_ACCESS_TOKEN can we specify where to create it (https://app.datawrapper.de/account/api-tokens) and the scopes we have to create? I initially tried with a narrow scope (Chart | read write) and the data loader created the chart, but couldn't read it back. It worked when I enabled more reading capabilities (Folder | read write; Theme | read; User | read; Visualization | read), but I'm not sure which of those were strictly necessary.
It could also be useful to explain how to set it inside the venv.
There was a problem hiding this comment.
I'm open to whatever environment management technique you want to recommend. I typically use a .env file with the standard pipenv conventions that is excluded from the git repository. In this case, for simplicity's sake I just did something like:
DATAWRAPPER_ACCESS_TOKEN=xxxyyzzz pipenv run npm run dev
|
To help reviewing I have deployed the project to https://observablehq.observablehq.cloud/framework-example-loader-datawrapper/ |
Co-authored-by: Philippe Rivière <fil@rezo.net>
|
I believe I addressed all of your requests. I'm open to whatever approach you want to take regarding the Python environment. |
Co-authored-by: Philippe Rivière <fil@rezo.net>
|
I'm sorry I edited my main comment just after I sent it, but you might have read it by email? Also you have removed the "fixed" data/chart.html that I added. We need it so people can see what to expect without actually having to connect to the API (and so we can also rebuild all the examples when necessary). |
|
I did miss that, sorry. I'm open to another example. I was thinking, "Let's keep it super simple" and give the user a bare-bones starting point. I'm happy to change the chart to something else. What appeals to you? A dressed-up stacked bar, a range plot, a dozen donuts? |
|
If you ask me, I like "Germany is the third-oldest country in the world". But it's only an opinion :) |
|
Should we make a note that the datawrapper iframe does not appear to support dark mode? |
|
Alrighty. I've updated it with the dot range plot. |
|
Ready for your review again, @Fil |
|
I'm a bit reluctant to publish it as is, because there are a few things that ought to be better: dark mode I've found a way to enable dark="auto" in the chart's preferences (editing the published chart), but I don't know how to pass this "render" option in the API call that creates the chart. Maybe it's not available in the python API? responsive Datawrapper has this concept of responsive charts, where the height depends on the (reactive) width and some javascript updates the iframe height. This example instead uses a fixed height, which ends up hiding some information (in our example, the footer) in a mobile view. EDIT: I've created a helper function to solve both problems, though. See below. tons of new charts It's kind of a bummer that it creates a new chart each time it runs. This makes it quite damaging for Ideally what I'd like to do is have a unique code in the script; if a chart with that code already exists, we reuse it, otherwise we populate it. If the chart depends on the data, then the code can be computed as a hash of the data state. I have not implemented that (yet), it's more on the python side.
|
|
I've redeployed to https://observablehq.observablehq.cloud/framework-example-loader-datawrapper/ Note that this helper function should be able to accommodate several charts on the page — maybe that's something we could add to the example too. |
the case is a browser in dark mode and a page that only has a light theme: dark is false, and the chart should not be left with the default behavior set up in Datawrapper
We could search for a chart that has that id set in a given hidden field, but I don't see if it's a possibility with the current API. |
|
Thanks for this detailed feedback. I'm going to try to address your concerns piecemeal. The most recent commit is my first effort. I believe I have configured the Python chart configuration to activate Datawrapper's dark mode, which is off by default. I'll loop back on your other items soon. |
|
I've added the cached chart.txt file and renamed the example as "Datawrapper API." I'm not sure I know what you mean by "the simpler version," but I took a thwack at what I think you mean. A web component that does not rely on the data loader is at the top. It is then followed by the Python code, which is then embedded with a variation on the snippet at the top. Is that what you mean? |
|
Yes, exactly! |
|
Reading https://blog.datawrapper.de/web-component-embedding/ I found out how to solve the dark mode issue, by adding data-dark="true" or "false". I'll fix in a moment (also doing some light copy-editing). |
|
hmmm, I was trying to embed another chart and I'm getting errors on nki3d and K2J9t (from https://blog.datawrapper.de/pigeon-internet-speed/), looks like in these instances the script tries to get the data from my own server…? |
|
Huh. Now you're past my knowledge. The chart isn't so old that it predates web components. Maybe we found another bug?
…On Fri, Aug 23, 2024, at 10:37 AM, Philippe Rivière wrote:
hmmm, I was trying to embed another chart and I'm getting errors on nki3d and K2J9t (from https://blog.datawrapper.de/pigeon-internet-speed/), looks like in these instances the script tries to get the data from my own server…?
—
Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACOCJSULNKQQP65NVECYTZS5CMJAVCNFSM6AAAAABMVKF276VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXGIZDQOJWGA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
I don't know what I was doing wrong, but K2J9t and nki3d now work as expected. I've added a different chart at the top for some diversity, polished the prose a bit, and added a helper function that handles dark mode in sync with Framework (bypassing the bug we've noticed). I think it's close to done! |
|
Looks good. I'm ready to ship it when you are. Where is that |
|
It's a reactive variable available in all pages. See https://observablehq.com/framework/lib/generators#dark (I linked to this page in the "tip") |
|
Very cool
…On Fri, Aug 23, 2024, at 11:49 AM, Philippe Rivière wrote:
It's a reactive variable available in all pages. See https://observablehq.com/framework/lib/generators#dark (I linked to this page in the "tip")
—
Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACOCLIXAT4LRSIUWPI2VLZS5KX5AVCNFSM6AAAAABMVKF276VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXGM2TMNJZGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
I was ready to ship but as I was giving it another look I again saw the error where the first chart is broken as it tries to load the dataset from https://observablehq.observablehq.cloud/framework-example-datawrapper-api/dataset.csv instead of from datawrapper. But when I reloaded, it worked. I don't understand if it's a bug in the dw script, or in something we do… |
|
Out of curiousity, I cloned the chart. The data appear to be "locally" hosted rather than an external URL or Google Sheet or anything. 
I wonder if that's their bug, which if you screenshot I can report to their support channel. In the meantime, I wonder if we can get something workable shipped if we use an example with an externally hosted CSV file, where there's a static URL in the configuration. You can see one of those here: https://academy.datawrapper.de/article/346-how-to-create-a-live-updating-chart-or-table |
|
Here's my screenshot. The weirdest part of this is that it's not happening all the time, maybe once out of 10. 
|
|
Gotcha. I see the bug too sometimes. I've sent your screenshot and a bug report with links to the Datawrapper support team. Do you get the bug with the externally hosted data example on COVID I linked? |
|
No I don't see it on the covid example — it's loading https://static.dwcdn.net/data-feed/coronavirus/confirmed-per-day-selected-countries-rolling.csv?b&v=1724685720000 |
|
Gotcha. I think that gives a little weight to my "the bug is with local data uploads" theory. I've got that filed in with the DW support team now. You okay with release as is with the external fix? I suspect the problem will be fixed upstream shortly. |
|
I don't want someone to be excited to test this out and then get disappointed. Either we wait for the upstream fix (my preference), or we add a warning that there is an issue and explain how to go around it. |
|
I respect that for sure. Let's see if I hear anything from Datawrapper soon. |
|
Almost on cue, I heard this from the Datawrapper team:
If that's correct and you can verify the republish has fixed things, I'd vote for merging. It would mean that this is only a bug in the edge case where you are trying to publish multiple version of years-old charts, which seems unusual. |
|
It's hard to prove a negative, but I've tested a dozen times and it's not coming back—and more importantly we can trust the Datawrapper team :) So yeah, let's 🥳 |
|
Great. If you need anything else from me, just shout. I'll promote on social media once it's live. |
This proposal adds an example that generates and then embeds a chart created by Datawrapper.
In addition to the Python dependencies, it requires a Datawrapper API key available in the Python environment as
DATAWRAPPER_ACCESS_TOKEN.