Dashboard showing up empty after hitting "publish"

[sunahsuh]

As reported by @davismtl -- he created this dashboard this afternoon and after he was done and hit publish, the graphs disappeared: https://sql.telemetry.mozilla.org/dashboard/fxa-login-signin-kpis

I ran this query to check and the visualizations seem to be linked correctly, at least at the DB level: https://sql.telemetry.mozilla.org/queries/1976/source

But the api call on the dashboard page appears to be empty:
https://sql.telemetry.mozilla.org/api/dashboards/fxa-login-signin-kpis
Response:

{"is_archived": false, "updated_at": "2016-12-21T23:27:26.711851+00:00", "layout": "[[1106, 1107], [1115, 1116], [1113], [1114, 1110], [1117, 1118], [1111, 1108]]", "is_draft": false, "id": 142, "can_edit": true, "user_id": 15, "name": "FxA: Overall (adavis)", "created_at": "2016-12-21T18:03:43.936011+00:00", "slug": "fxa-login-signin-kpis", "version": 30, "widgets": [[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []], "dashboard_filters_enabled": false}

[davismtl]

If it can help, I'm nearly certain the last chart I added before hitting publish was: https://sql.telemetry.mozilla.org/queries/1974/source#3528

This chart was in a "Draft" state at the time of publishing the dashboard. It was probably the only chart in a draft state at the time of publishing.

[davismtl]

Hey @sunahsuh , couple of questions:

• How long do we think it will take to fix the dashboard?
• If it will take a while, can you let me know what query or chart numbers were in it and in what order and perhaps I can remake it in the meantime.

Thanks for your help

[sunahsuh]

Hey Alex, I'm not sure re: a fix -- the original idea was that someone on Data Tools that knows the codebase would investigate further.
I was able to re-create this list of chart urls, at least, from the IDs that are still stored in the DB: https://sql.telemetry.mozilla.org/queries/2010/source

[davismtl]

Thank you! This is super helpful!

[washort]

I'm having a look at this, going to see if it can be reproduced in a development environment.

[davismtl]

@rfk just experienced the same thing. Maybe he can add details here.

[rfk]

Not sure I can add any more than a "+1", but it did seem to happen under the same set of circumstances as @davismtl described - adding draft charts to a draft dashboard, and publishing the dashboard without marking all the charts as published.

[washort]

My attempts to reproduce this so far haven't worked, publishing a draft dashboard with draft queries appears to work most of the time. Here's one, for example, with one of your draft queries in it: https://sql.telemetry.mozilla.org/dashboard/dashboard-bug-test?p_days=30

I'll keep digging.

[washort]

@robotblake identified the immediate problem: the layout field in the dashboard data structure was stored as a JSON string instead of a JSON list of widget ids. So the bug is probably in the dashboard-editing JS. A workaround for now would be to just fix up any existing layout fields malformed in this way while we track down the origin of this.

[washort]

We're working on getting a newer version of redash deployed that should fix this.

[washort]

This Sentry error appears to be the same issue: https://sentry.prod.mozaws.net/operations/pipeline-sql-prod/issues/371388/

[rafrombrc]

These are the impacted dashboards:

https://sql.telemetry.mozilla.org/dashboard/fxa-password-reset
https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-demo-test
https://sql.telemetry.mozilla.org/dashboard/fxa-login-signin-kpis
https://sql.telemetry.mozilla.org/dashboard/user-preferences
https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-demo---deduped-combined-frecency_2
https://sql.telemetry.mozilla.org/dashboard/china-global-vs-china-build-android-

They seem to all be archived, but they should be fixed now anyway.

[madalincm]

I have verified all the above dashboards and all are looking good. Closing issue.