Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Let VersionTracker optionally ignore archive steps #2447

Closed
wants to merge 63 commits into from
Closed

Let VersionTracker optionally ignore archive steps #2447

wants to merge 63 commits into from

Conversation

pabloarosado
Copy link
Contributor

In StepUpdater, the property steps_df should only contain active steps (since they are the only ones to be considered for update, and the only ones relevant for the ETL dashboard). Indeed, column step of steps_df (which is the main index) only contains active seps. However, other columns (e.g. direct_usages) do contain archive steps. This issue was mentioned in #2365

This raises an IndexError in the ETL dashboard, if, in the Operations list, you click on "Add direct usages" for steps that are used by archive steps.

This PR fixes that. Now VersionTracker (which is loaded by StepUpdater) can be initialized with the argument ignore_archive=True.

@pabloarosado
Adapt create new snapshots script
22e5dec
@pabloarosado
Fix create_new_snapshots
418274d
@pabloarosado
Add snapshots and meadow steps, and remove wcad
7508ad4
@pabloarosado
Remove faostat_gn and fix issues
79e1784
@pabloarosado
Fix meadow faostat_sdgb
3d0eaf2
@pabloarosado
Update docs
ac6e691
@pabloarosado
Add garden steps and minor improvements on meadow
17de3c2
@pabloarosado
Fix issues with item code changes in faostat_fs
273de4f
@pabloarosado
Add new flag, and increase number of characters for item codes of fao…
6277a02 
…stat_sdgb
@pabloarosado
Remove unused excluded country, fix new element mismatch between fbsh…
660ed7f 
… and fbs, and replace assertions with log errors
@pabloarosado
Fix issue with categorical columns in script that updates custom meta…
a9c5fc4 
…data
@pabloarosado
Update custom metadata files
c4169f8
@pabloarosado
Add element amendments for fbsc
c7dd40a
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
196e9db
@pabloarosado
Fix various issues
8a39331
@pabloarosado
Remove anomaly fix on cocoa beans, which is no longer in the data
9ff00b5
@pabloarosado
Fix issues with region aggregates in fbsc
eb45fab
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
27ffd25
@pabloarosado
Explicitly use regions, population and income groups datasets
d4606b8
@pabloarosado
Fix anomaly of spinach area harvested in China
2966376
@pabloarosado
Fix errors on missing units
91f8f43
@pabloarosado
Minor fixes
649303d
@pabloarosado
Fix missing metadata in latest faostat_qv, big refactor
d906ba8
@pabloarosado
Fix changed item in qcl and global food explorer
a109206
@pabloarosado
Add grapher steps (WIP)
e4cc4d7
@pabloarosado
Use tables instead of dataframes
3da102e
@pabloarosado
Use tables instead of dataframes in fbsc and qcl steps
4aad908
@pabloarosado
Use tables instead of dataframes in garden metadata step
38d5a4a
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
eb45770
@pabloarosado
Improve metadata handling of meadow steps
ef0a19a
@pabloarosado
Ensure origins are propagated
632a27e
@pabloarosado
Fix changed country names
5963b81
@pabloarosado
Avoid repetitive warning on missing units
5b63f9e
@pabloarosado
Check discrepant item name definitions and improve warning messages a…
c8f00c5 
…nd documentation
@pabloarosado
Improve warning messages
5aee362
@pabloarosado
Inspect remaining warnings and improve warning handling
7ac08af
@pabloarosado
Add garden and grapher step of additional variables
acd456a
@pabloarosado
Adapt garden and grapher steps of additional variables
5a54af1
@pabloarosado
Add multi_merge function to owid catalog
06ae673
@pabloarosado
Update metadata of additional variables dataset
751594c
@pabloarosado
Improve metadata of additional variables dataset
5de6499
@pabloarosado
Improve function that finds dataset id from name
1b5febd
@pabloarosado
Improve script to submit faostat chart revisions
95c11e4
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
6c28550
@pabloarosado
Fix errors in script that creates chart revisions
c98ef24
@pabloarosado
Delete unused faostat script
cfc3a5e
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
4f4ecb1
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
a817da2
@pabloarosado
Fix bad format of description_key
ed19a50
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
195d7cb
@pabloarosado
Improve snapshot descriptions
c029fd3
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
c19fb04
@pabloarosado
Update custom metadata files and documentation
f0924be
@pabloarosado
Improve metadata
6289859
@pabloarosado
Merge branch 'master' of github.com:owid/etl into update-faostat-data
8ea47e2
@pabloarosado
Fix issue in StepUpdater where archive steps are not really ignored
c389835
@pabloarosado pabloarosado self-assigned this Mar 25, 2024
@pabloarosado pabloarosado changed the title Version tracker optionally ignore archive steps Let VersionTracker optionally ignore archive steps Mar 25, 2024
@pabloarosado pabloarosado deleted the version-tracker-optionally-ignore-archive-steps branch March 25, 2024 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@pabloarosado pabloarosado

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pabloarosado @Marigold