✨ Add anomalist to owidbot

[Marigold]

• Add anomalist to owidbot, this will allow us to run it automatically after every commit on staging servers (it only runs new datasets)
• Save dataset.sourceChecksum to anomalies table. This means we can skip computing anomalies unless a dataset changes (this is especially useful for owidbot and running anomalist on staging servers)
• Function _load_datasets_new_ids to get new datasets (if it works, we can use it in anomalist app)

[owidbot]

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-owidbot-anomalist

chart-diff: ✅

No charts for review.

Edited: 2024-10-18 12:30:07 UTC
Execution time: 3.69 seconds

[pabloarosado]

Looks good! Your way to detect new datasets is very simple and it makes sense. Two comments (probably better to discuss on slack):

1. If this is meant to happen under the hood, then it would make sense to infer the variable mapping if indicator upgrader has not yet been executed, which is what I implemented on ✨ anomalist: Improve automatic detection of new datasets #3429
2. It's not clear to me what should happen now if the user opens Anomalist. We need to adapt the app so that anomalies are automatically detected from database, or is that already working somehow? I suppose that loading anomalies for the new datasets from the start is the best option. And, if the user wants to check anomalies on other datasets, that should still be possible (but it will take a while to recalculate anomalies).

[Marigold]

1. Yep, I'm waiting for you to merge your PRs, then I'll rebase and update my code to use your new features

We need to adapt the app so that anomalies are automatically detected from database, or is that already working somehow?

It's not working yet. You're right that it should select new datasets by default and load anomalies without needing to click on the button. I'll add it either in this PR or the next one.