Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Gap #174

Merged
merged 57 commits into from
Nov 25, 2024
Merged
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
823612a
✨ Add data gap utils
i-be-snek Oct 17, 2024
3ccf223
🚧 Add data gap script
i-be-snek Oct 30, 2024
166314f
🔊 Update logs
i-be-snek Oct 30, 2024
012e458
🚨 Fix mypy typing warnings
i-be-snek Oct 30, 2024
1c6c6a5
➕ Add dev deps for typing hints with vcode mypy extention
i-be-snek Oct 30, 2024
23c9518
🚨 Fix more mypy lint warnings
i-be-snek Oct 30, 2024
38f0455
✨ Fill YEAR downstream l1->l2/l3
i-be-snek Oct 31, 2024
f80be3b
🚧 Add data gap filler for areas
i-be-snek Nov 1, 2024
53bbe88
✨ Create geojson de-duplicated db in insertion step
i-be-snek Nov 1, 2024
8d70d06
🗃️ Convert to str before insertion
i-be-snek Nov 1, 2024
2e35c8c
📝 Update docs on inserting a full run
i-be-snek Nov 1, 2024
ac3ea98
✏️ Fix typo in docs
i-be-snek Nov 1, 2024
313fd53
🔊 Update logs and comments
i-be-snek Nov 3, 2024
1e27b85
🦺 Improve validation of currencies (None)
i-be-snek Nov 13, 2024
a1e2865
🚧 Fill l3->l2 impact gap
i-be-snek Nov 13, 2024
1391147
🚧 Append l3 records not found in l2 to l2
i-be-snek Nov 13, 2024
b22c620
🚧 Validate impact values (l3<=l2)
i-be-snek Nov 14, 2024
4784ee5
🐛 Fix repeated list of areas not in l2
i-be-snek Nov 14, 2024
da74a09
🔊 Update log message for clairty
i-be-snek Nov 14, 2024
c6ff795
♻️ Refactor util code -m add logs and improve flow + add an exception…
i-be-snek Nov 14, 2024
226e235
💡 Remove rogue print statement
i-be-snek Nov 14, 2024
31d516d
🚧 Fill area gap for l2/l3 areas missing in l1
i-be-snek Nov 14, 2024
b73e457
🚧 Add various debug fixes
i-be-snek Nov 14, 2024
570d17d
🐛 Fix countring Nones as `0` values
i-be-snek Nov 15, 2024
97572a4
🚧 Fill empty l1 records with l2 aggregates
i-be-snek Nov 15, 2024
ae0bdb8
🚧 Fill l2->l1 when l1 is None or < l2
i-be-snek Nov 16, 2024
ba842e0
🚧 Store output files
i-be-snek Nov 16, 2024
09629bb
📝 Add docs on data gap filling
i-be-snek Nov 17, 2024
6b332c3
📦️ Add deduplicated files for full run
i-be-snek Nov 17, 2024
c147765
🐛 Fix mystery error when loading parquet from a dir
i-be-snek Nov 17, 2024
40419c6
🐛 Add try/except block for show stopper
i-be-snek Nov 17, 2024
0886f2a
🩹 Fix using a subset of impact categories instead of all
i-be-snek Nov 18, 2024
7101e63
🐛 Convert bytes to str if present
i-be-snek Nov 18, 2024
ac4ac95
♻️ Remove bytes at the start
i-be-snek Nov 18, 2024
e76955f
♻️ Chunk at 25 records for each file
i-be-snek Nov 18, 2024
c3f6d1e
📦️ Add deduplicated files with data gap filled
i-be-snek Nov 18, 2024
755cd98
🐛 Fix L1 not filled with L2 agg
i-be-snek Nov 19, 2024
5701407
🔨 Refactor dev script
i-be-snek Nov 19, 2024
e2f45c0
🐛 Fix L1 NaNs not filled with aggregate L2 values
i-be-snek Nov 20, 2024
21e7a0f
🔊 Add logs for year/date gap filling
i-be-snek Nov 20, 2024
7ef4e5c
♻️ Avoid replacing missing years with None
i-be-snek Nov 20, 2024
2399457
🏷️ Fix types
i-be-snek Nov 21, 2024
612fc91
✨ Handle duplicated records
i-be-snek Nov 21, 2024
0baeea4
🐛 Fix duplicate aggregaes
i-be-snek Nov 22, 2024
0ad3004
📦️ Add data gap fixed files
i-be-snek Nov 22, 2024
b9d8834
🐛 Fix bytes to str conversion
i-be-snek Nov 23, 2024
66e59b7
🗃️ Remove non-climate related events
i-be-snek Nov 23, 2024
c406f05
♻️ Use preset vars
i-be-snek Nov 23, 2024
453ce56
🐛 Fix nan event_ids
i-be-snek Nov 24, 2024
1a6b302
🔊 Demote log info->debug for year filling
i-be-snek Nov 24, 2024
99c919a
🐛 Fix l3->l2 fill bug
i-be-snek Nov 24, 2024
aa311b4
📦️ Add compiled de-duplicated files with data gap filled (again)
i-be-snek Nov 24, 2024
197a290
🐛 Fix not null validation in db insert
i-be-snek Nov 24, 2024
e695429
🗃️ Add impactdb v.1.2 (NOT NULL validation bug fix)
i-be-snek Nov 24, 2024
7f9c74b
🗃️ Data-gap filled database (v1.0)
i-be-snek Nov 24, 2024
b2f843a
🚚 Rename v.1 -> v1.0
i-be-snek Nov 24, 2024
ab1eac2
🦺 Fix dtypes for validation
i-be-snek Nov 24, 2024

Sorry, this diff is taking too long to generate.

It may be too large to display on GitHub.