fix(ingest): ingest deleted looker dashboards

[karoliskascenas]

all_dashboards() does not return deleted dashboards (even says so in the comments).

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[frsann]

Can you make that addition optional at least? We have no use of deleted dashboards in our datahub. Or are you more interested in marking previously ingested dashboards as deleted, so that they disappear in datahub?

[karoliskascenas]

Or are you more interested in marking previously ingested dashboards as deleted, so that they disappear in datahub?

That's exactly what I am trying to achieve.

[frsann]

But as a side effect you will be "ingesting" all deleted ones as well. They wont show up in the UI, but they will be stored in the database. Maybe not a huge problem, but worth thinking about.

[karoliskascenas]

Judging by the code that adds a status aspect this was always the intended functionality - it just never had a chance to work.

[frsann]

Sure, but it might be good to add a note about this in the docs ("be very conservative with the allow-pattern"). We are at least closing in on 5 digit dashboard IDs (I assume they are strictly increasing) as users create and delete dashboards as the see fit, and I certainly don't want to re-ingest it on a, say, daily basis.

[shirshanka]

LGTM!

[shirshanka]

@frsann makes a good point, it may not be a great idea to query looker for all dashboards (deleted + not-deleted) also.

Didn't review this too carefully, we'll have to think about a different way to accomplish deletes IMO

[karoliskascenas]

Would making the ingestion of deleted dashboards optional work for you?

[shirshanka]

LGTM!