-
Notifications
You must be signed in to change notification settings - Fork 971
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
updated FR parser to enable historical backfill #5240
Conversation
✅ Deploy Preview for phenomenal-syrniki-c5ceaa ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note this is not a review of the code itself but the implementation of the new data source. In #4853 there was some concerns about validating the data source and making sure all the data was valid before we proceeded. Is this still the case?
If not might it not be better to proceed with #4853 and split France up in the process when switching the parsers (and aggregating it again together with corsica)?
Hi @VIKTORVAV99, I don't know how much we want to prioritize splitting FR into regions. We realised we couldn't backfill and that the data can be significantly modified and thought that this was more important to the Sales team. As you can see on the OpenData page, the data we collect is provisional. |
Perhaps we should increase the priority of https://github.com/electricitymaps/electricitymaps/pull/3230, since it would allow us to fetch estimated data (separate from validated data)? Seems like yet another scenario where it would be beneficial. I'll do a full code review this afternoon as well 👍🏻 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some naming suggestions but looks good to me!
Co-authored-by: Robin TROESCH <38283096+unitrium@users.noreply.github.com>
Co-authored-by: Robin TROESCH <38283096+unitrium@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything looks good to me with the exception of the type issue. If you can take care of that it would be greatly appreciated.
Issue
We noticed that we couldn't refetch data that was older than 9 months using the current dataset. We know that RTE publishes final data at Y+1 and we want to be able to update our dataset with the most up-to-date data.
![image](https://user-images.githubusercontent.com/107848894/227958778-4e9a17b7-63d4-4927-b35f-b3d131c7e4e4.png)
Description
the aim here is to enable historical refetch. the historical data is available in the dataset
eco2mix-national-cons-def
Preview
2023-03-26
![image](https://user-images.githubusercontent.com/107848894/228005526-71221f58-e725-43ae-8881-abdf21a79849.png)
2021-12-31
Double check
poetry run test_parser "zone_key"
pnpx prettier --write .
andpoetry run format
to format my changes.