Regularly check for dataset updates by providers

[Marigold]

Problem

Providers release updates to their datasets, but we are not notified of them. Knowing which datasets have newer versions and could be updated would be helpful information on the ETL dashboard.

Solutions

Check data from the snapshot URL

Many snapshots contain a URL for the data. We could fetch the data from the URL, compare it to the data in our existing snapshot, and notify us if they differ.

Ask ChatGPT to check for updates

Our snapshot metadata file contains all the information about the provider, dataset, and URLs. We could provide this information to ChatGPT and ask it to use web search to check for updates (e.g., by checking the provider's changelog). This might not work perfectly but is simple enough to try.

[pabloarosado]

Hi @Marigold, thanks for writing this up, and sorry, I didn't ignore it! I just thought it would be good to create a more detailed issue to properly shape the project. If you want, you can add your proposal in #3339 , and feel free to close this issue. Thanks!

[Marigold]

Closing in favour of #3339