Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Climate change and health - precipitation and healthcare provision #1485

Draft
wants to merge 211 commits into
base: master
Choose a base branch
from
+20,473 −0
Draft

Climate change and health - precipitation and healthcare provision #1485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
211 commits
Select commit Hold shift + click to select a range
0c0ca8c
Grid_malawi.py: Loads shap files and creates a graph of Malawi. Have …
RachelMurray-Watson Oct 9, 2024
32134f6
Initial plot of heaviest precipitation days from CCKP. Will abandon d…
RachelMurray-Watson Oct 9, 2024
caa77da
Plot of precipitation over all of malawi
RachelMurray-Watson Oct 9, 2024
23ca94e
Attempt to subdivide by region
RachelMurray-Watson Oct 9, 2024
7fd413a
Found intersection between grid and the admin areas, so now can say w…
RachelMurray-Watson Oct 9, 2024
97e404f
Script for retrieving data from each CMIP6 projection model and write…
RachelMurray-Watson Oct 10, 2024
0cbfbb3
Functions and process for unzipping file directories and extracting n…
RachelMurray-Watson Oct 10, 2024
ccb5178
isort
RachelMurray-Watson Oct 10, 2024
61a1ec2
Downloaded facility-level reporting data from DHIS2.
RachelMurray-Watson Oct 10, 2024
52f243a
Added file path to write facilities doc
RachelMurray-Watson Oct 10, 2024
a25aefa
renamed variable for clarity
RachelMurray-Watson Oct 10, 2024
fda157e
# So, because of the fact one district may overlap with many grids, t…
RachelMurray-Watson Oct 10, 2024
1a0c615
Updated comments
RachelMurray-Watson Oct 10, 2024
e98f1fb
Wrote to shapefile
RachelMurray-Watson Oct 10, 2024
94c569d
So, can't use the shap file for Malawi to get my grids for the health…
RachelMurray-Watson Oct 11, 2024
db73168
Fixed aesthetics of overlap
RachelMurray-Watson Oct 11, 2024
a67f31e
Changed to daily data retrieval (historical can be monthly)
RachelMurray-Watson Oct 11, 2024
8921204
Changed to daily data retrieval (historical can be monthly)
RachelMurray-Watson Oct 11, 2024
61c8c59
Typo
RachelMurray-Watson Oct 11, 2024
29af34f
Added actual days for dates
RachelMurray-Watson Oct 11, 2024
2eb0ce7
Creating dictionaries of data by model and grid
RachelMurray-Watson Oct 11, 2024
590a39c
Added csv
RachelMurray-Watson Oct 11, 2024
4a10441
Fixed setting directory
RachelMurray-Watson Oct 11, 2024
7563316
Changed some comments for clarity
RachelMurray-Watson Oct 11, 2024
9542bdd
Extracted more variables. Converted from seconds to days.
RachelMurray-Watson Oct 11, 2024
3b02dbd
API request for accessing ERA5 reanalysis data.
RachelMurray-Watson Oct 11, 2024
c5e9e21
isort
RachelMurray-Watson Oct 11, 2024
8e5ad0f
Re-organized data by month, averaged across all reporting types for t…
RachelMurray-Watson Oct 11, 2024
7bef52a
Saving shap file of malawi grid
RachelMurray-Watson Oct 11, 2024
02f2b20
Saving shap file of malawi grid, removed print
RachelMurray-Watson Oct 11, 2024
22f46b0
Imported malawi shap file and did an initial regrid to get correct re…
RachelMurray-Watson Oct 11, 2024
d6c3d62
Edited months to make sure no mental health data inlcuded (only relav…
RachelMurray-Watson Oct 11, 2024
e773e6a
Added grid index to the facilities list for referral later
RachelMurray-Watson Oct 14, 2024
270a294
Changed numerical grid reference so that it is the major grid, not th…
RachelMurray-Watson Oct 14, 2024
42c985c
Added linear regression but need to investigate as uneven numbers of …
RachelMurray-Watson Oct 14, 2024
25bd34c
Dropped division by metric as unneeded
RachelMurray-Watson Oct 14, 2024
ecca012
Dropped October-December 2024 from the reporting data as do not have …
RachelMurray-Watson Oct 14, 2024
c18c129
removed imports
RachelMurray-Watson Oct 14, 2024
0785843
File for examination of trend between weather and reporting
RachelMurray-Watson Oct 14, 2024
0c077a9
Added labels to graph
RachelMurray-Watson Oct 14, 2024
a25a0fc
Comments for clarity
RachelMurray-Watson Oct 14, 2024
81caf49
Added year as a fixed effect - as reporting goes up over time anyway
RachelMurray-Watson Oct 14, 2024
c53faf2
Included year and month as fixed effects
RachelMurray-Watson Oct 14, 2024
3d88f3b
isort
RachelMurray-Watson Oct 14, 2024
a8ed9c6
Two versions of linear regression - one by facility (year and month a…
RachelMurray-Watson Oct 14, 2024
e5d468b
One-hot encoded the facilities variable for fixed effect
RachelMurray-Watson Oct 14, 2024
7106542
Changed to statsmodel to get summary data
RachelMurray-Watson Oct 14, 2024
0e5e6fc
Added binary predictor of above/below average precipitation for that …
RachelMurray-Watson Oct 14, 2024
ae6a5bc
Added whether in top 80 percentile... But too few observations
RachelMurray-Watson Oct 14, 2024
25be631
Retrieval script for daily (sampled hourly) analysis data. Returns ma…
RachelMurray-Watson Oct 15, 2024
e74ff7c
Request was too large must loop over years
RachelMurray-Watson Oct 15, 2024
5e37b42
Couldn't access the newest dataset (https://cds.climate.copernicus.eu…
RachelMurray-Watson Oct 15, 2024
e51d318
Whitespace
RachelMurray-Watson Oct 15, 2024
0273979
Updated fpr ssp2_4_5
RachelMurray-Watson Oct 15, 2024
dfd6dbf
changed back to the max daily precip (even though it is still not wor…
RachelMurray-Watson Oct 16, 2024
441af24
removed double request for days
RachelMurray-Watson Oct 16, 2024
b86b88e
rename for specificity
RachelMurray-Watson Oct 16, 2024
c5267f3
New file that has finer-grid facilities. Have broad mapping to grids;…
RachelMurray-Watson Oct 16, 2024
ed47a96
Edited to correspond directly to the malawi grid
RachelMurray-Watson Oct 16, 2024
ae81b8a
removed accidental filtering of data
RachelMurray-Watson Oct 16, 2024
2c35964
Updated to include the 2011 data
RachelMurray-Watson Oct 16, 2024
7ca170f
Tidied
RachelMurray-Watson Oct 16, 2024
1ce484f
Added new data for finer grain on facilities
RachelMurray-Watson Oct 16, 2024
618a192
See nothing by DHO
RachelMurray-Watson Oct 16, 2024
a62075d
added district level, but dropped central hospitals
RachelMurray-Watson Oct 16, 2024
2c9c439
ssp245
RachelMurray-Watson Oct 16, 2024
3de67c7
Added accessing of historical grids
RachelMurray-Watson Oct 16, 2024
12eb3c0
Updated to use data from new file that has lat/long for each facility…
RachelMurray-Watson Oct 16, 2024
46193b3
Back to central facilities (not more accurate)
RachelMurray-Watson Oct 16, 2024
aba7ece
Added extra variables, e.g. if rain was above/below a certain thresho…
RachelMurray-Watson Oct 17, 2024
cad17be
tidied
RachelMurray-Watson Oct 17, 2024
89562b2
First stab at aggregating across all of CMIP data. However, not every…
RachelMurray-Watson Oct 17, 2024
2bdfe15
File that collects data from "smaller" facilities (all District/Rural…
RachelMurray-Watson Oct 17, 2024
aa025cd
allowed writing csv
RachelMurray-Watson Oct 17, 2024
3e7c078
Edited to allow for small facilities.
RachelMurray-Watson Oct 17, 2024
fc21fad
notes
RachelMurray-Watson Oct 17, 2024
7b7b561
notes
RachelMurray-Watson Oct 18, 2024
f540d13
Removed the actually facility as a predictor, as it introduced a lot …
RachelMurray-Watson Oct 18, 2024
c636da0
tidied
RachelMurray-Watson Oct 18, 2024
17b7728
Changed to beta regression model - no avail
RachelMurray-Watson Oct 18, 2024
4bccfd2
Kept at 1000mm level
RachelMurray-Watson Oct 18, 2024
bfeed74
Linear model for rain and reporting
RachelMurray-Watson Oct 21, 2024
9df09d2
Accidentally was comparing latitude and longitude
RachelMurray-Watson Oct 21, 2024
bd9a521
Have correct data now... but association is gone
RachelMurray-Watson Oct 21, 2024
19077e8
Added beta model check
RachelMurray-Watson Oct 21, 2024
b57ed49
plotting
RachelMurray-Watson Oct 22, 2024
d0a6bb7
Plots raw reanalysis data against reporting
RachelMurray-Watson Oct 22, 2024
524c9f7
Inlcuded plot of y_pred
RachelMurray-Watson Oct 23, 2024
2a8be78
Plot of reporting vs precip
RachelMurray-Watson Oct 23, 2024
7236ac2
isort
RachelMurray-Watson Oct 24, 2024
7955ebf
file to calculate the 5- day average from the historical ERA5 reanaly…
RachelMurray-Watson Oct 24, 2024
d148786
file to calculate the 5- day average from the historical ERA5 reanaly…
RachelMurray-Watson Oct 25, 2024
83c5414
renamed
RachelMurray-Watson Oct 25, 2024
606be18
added association to facilities based on grid squares
RachelMurray-Watson Oct 25, 2024
1c02b96
changed to remove oct 2024
RachelMurray-Watson Oct 25, 2024
6e54dab
changed so that a seperate file is created if it is using the ANC data
RachelMurray-Watson Oct 25, 2024
1230993
Reduced window to max weather for that month
RachelMurray-Watson Oct 25, 2024
feaa71a
Added check to see if it was ANC data, so then changed how files were…
RachelMurray-Watson Oct 25, 2024
50fc2aa
Added check to see if it was ANC data, so then changed how files were…
RachelMurray-Watson Oct 25, 2024
6c45d2b
Added new weather data writing to account for the fact there are diff…
RachelMurray-Watson Oct 25, 2024
7bef017
Allowed for toggle between ANC/reporting and daily max/monthly
RachelMurray-Watson Oct 25, 2024
0cb6086
made more flexible for going between model scenarios
RachelMurray-Watson Nov 6, 2024
603e96c
Changed to allow for ssp245
RachelMurray-Watson Nov 7, 2024
e0383fa
Changed how data were aggregated (do not need to search over model na…
RachelMurray-Watson Nov 7, 2024
986aa24
Deleted as now part of "process_CMIP6_data"
RachelMurray-Watson Nov 7, 2024
371a078
Redid so no longer need to list the models, only need to look in dire…
RachelMurray-Watson Nov 7, 2024
15f1cac
Added averaging over time points
RachelMurray-Watson Nov 8, 2024
14925e2
isort
RachelMurray-Watson Nov 8, 2024
4098fa7
realized I averaged over all timepoints
RachelMurray-Watson Nov 8, 2024
38e1c1c
Realized for daily precipitation, doing an incorrect conversion. The …
RachelMurray-Watson Nov 8, 2024
c494264
Realized for daily precipitation, doing an incorrect conversion. The …
RachelMurray-Watson Nov 8, 2024
6e235d4
Added plot of projected rain by grid point
RachelMurray-Watson Nov 8, 2024
dead8bf
Changed 86400 as the mulitplier to depend on what data source is bein…
RachelMurray-Watson Nov 8, 2024
00f5317
Changed to collect daily totals, not daily maximums
RachelMurray-Watson Nov 11, 2024
a83bf90
Added in median and 25/75 percentiles
RachelMurray-Watson Nov 11, 2024
1980632
added in multiplier variable
RachelMurray-Watson Nov 11, 2024
96ee2ef
intorduced mask level variable
RachelMurray-Watson Nov 11, 2024
d7b20b4
intorduced mask level variable
RachelMurray-Watson Nov 11, 2024
53dd04a
Plot for median and IQR (needs improving for legibility)
RachelMurray-Watson Nov 11, 2024
eaf3ec5
removed multiplier from reporting (?)
RachelMurray-Watson Nov 11, 2024
1e5d464
File for processing the ERA5 historical reanalysis data for daily tot…
RachelMurray-Watson Nov 12, 2024
5597d4d
Added in daily total data
RachelMurray-Watson Nov 12, 2024
40a10ca
Added in changes for five day and/or cumulative sum
RachelMurray-Watson Nov 12, 2024
f71b7d7
Did all time points
RachelMurray-Watson Nov 12, 2024
a317df1
Changed to allow for fact that data was hourly, so need to add up ove…
RachelMurray-Watson Nov 12, 2024
8c2d12b
Removed unnecessary repeat
RachelMurray-Watson Nov 12, 2024
1af6a22
Added in code to load 5 day cumulative
RachelMurray-Watson Nov 12, 2024
fc5b4b5
Data to put CMIP6 models on the Malawi grid
RachelMurray-Watson Nov 12, 2024
54fbd7a
Plot individual grids
RachelMurray-Watson Nov 12, 2024
94abde8
Added model to generate new y data set. Residuals of model without th…
RachelMurray-Watson Nov 12, 2024
a24a8e4
Reordered code for clarity.
RachelMurray-Watson Nov 12, 2024
41904f2
Added in if else if not doing 5 day
RachelMurray-Watson Nov 13, 2024
a6410d9
Added in function for stepwise selection
RachelMurray-Watson Nov 13, 2024
aef2d35
changed file name
RachelMurray-Watson Nov 13, 2024
37d0f92
exclude some outliers
RachelMurray-Watson Nov 13, 2024
0a764da
have somehow lost significant relationship...
RachelMurray-Watson Nov 13, 2024
f45f5a6
Checked "all" model and find the R2 is better
RachelMurray-Watson Nov 13, 2024
b5955f7
File for using the picked models and predicting the shortfall in ANC …
RachelMurray-Watson Nov 14, 2024
f462a07
Used ANC data and daily totals for 5-day cumulative.
RachelMurray-Watson Nov 14, 2024
c3a993c
Added loop to create monthly and 5-day cumulatve data. Needs tidying.
RachelMurray-Watson Nov 14, 2024
1a15b5a
Soft reset
RachelMurray-Watson Jan 3, 2025
674bf96
Added in facilities to map
RachelMurray-Watson Jan 3, 2025
20331e4
Added calculation of average precipitation
RachelMurray-Watson Jan 3, 2025
6b10869
Coloured dots of clinics by average annual precipitation
RachelMurray-Watson Jan 3, 2025
dba70c8
removed facilities
RachelMurray-Watson Jan 3, 2025
9654cc3
removed facilities legend
RachelMurray-Watson Jan 3, 2025
baf9038
isort
RachelMurray-Watson Jan 3, 2025
8018c8f
Put era5 and CMIP6 on same graph
RachelMurray-Watson Jan 6, 2025
7bbf91a
Added loop for going over each scenario of the CMIP6 and then the med…
RachelMurray-Watson Jan 6, 2025
0e7f0c1
tidied function
RachelMurray-Watson Jan 6, 2025
0d7402a
Saved model prediction coefficients
RachelMurray-Watson Jan 6, 2025
8b83ff9
added p value and coefficient names to saving
RachelMurray-Watson Jan 6, 2025
3112713
file for accessing CIL downscaled data
RachelMurray-Watson Jan 7, 2025
9278314
Did piecewise downloading of downscaled data (as kept getting nan)
RachelMurray-Watson Jan 7, 2025
39af21e
Changed params
RachelMurray-Watson Jan 7, 2025
ac9d1d1
Removed the scaling
RachelMurray-Watson Jan 7, 2025
fd5c98b
Rename
RachelMurray-Watson Jan 7, 2025
1d398ee
Run kdball
RachelMurray-Watson Jan 7, 2025
ffa6846
Have quicker processing of downscaled data
RachelMurray-Watson Jan 8, 2025
3fed2c4
Changed naming
RachelMurray-Watson Jan 8, 2025
75777bc
Changed data to access the files where data was missing
RachelMurray-Watson Jan 9, 2025
c545685
Changed data to access the files where data was missing
RachelMurray-Watson Jan 9, 2025
572a54a
Undid hash out
RachelMurray-Watson Jan 9, 2025
26b613f
Large files
RachelMurray-Watson Jan 9, 2025
4591b07
Stops saving large files
RachelMurray-Watson Jan 9, 2025
20171b8
Added district
RachelMurray-Watson Jan 9, 2025
d5f2e30
Found Blantyre was mispelled as "Blanytyre"
RachelMurray-Watson Jan 9, 2025
48422c7
Added district
RachelMurray-Watson Jan 9, 2025
ae22229
"Contents are identical"
RachelMurray-Watson Jan 9, 2025
a65f099
Added map showing projected decrease (%) by district of ANC cases (fo…
RachelMurray-Watson Jan 9, 2025
0016678
Added map showing projected decrease (%) by district of ANC cases (fo…
RachelMurray-Watson Jan 9, 2025
3c687ac
Python file for calculating and plotting the projected disruptions in…
RachelMurray-Watson Jan 9, 2025
7d49812
Added plot of them all
RachelMurray-Watson Jan 9, 2025
87fdc85
Readded scaling
RachelMurray-Watson Jan 10, 2025
34110a1
Rmoved if log y as y will always be logged
RachelMurray-Watson Jan 10, 2025
26c2521
Rmoved use residuals as changed that analysis
RachelMurray-Watson Jan 10, 2025
2b2d028
The standard scaler introduces very weird results (esp. for the Dedza…
RachelMurray-Watson Jan 10, 2025
3096b0f
Removing the facilities reduces R^2 but also increases significance o…
RachelMurray-Watson Jan 10, 2025
91dacec
Tried PCA... Made worse
RachelMurray-Watson Jan 10, 2025
706f404
Tried PCA... Made worse, but all predictors did have significant coef…
RachelMurray-Watson Jan 10, 2025
d3e4aae
Actually, it was the imputer that made it work
RachelMurray-Watson Jan 10, 2025
0e5c0c7
Not even, it was a mistake...
RachelMurray-Watson Jan 10, 2025
0c85041
Removed PCA
RachelMurray-Watson Jan 10, 2025
e6f38f9
Testing statistically significant results
RachelMurray-Watson Jan 10, 2025
345e545
Tidy
RachelMurray-Watson Jan 10, 2025
940967f
Tidy
RachelMurray-Watson Jan 10, 2025
f9b5ff3
Correlation matrix - removing ftype and district reduced multicolline…
RachelMurray-Watson Jan 13, 2025
d9d0959
Correlation matrix - removing ftype and district reduced multicolline…
RachelMurray-Watson Jan 13, 2025
07efa88
Reordered naming of predictors for saving results
RachelMurray-Watson Jan 13, 2025
2881fac
Identical
RachelMurray-Watson Jan 13, 2025
4ad5234
Fixed up graphs and edited to make 5-year groupings
RachelMurray-Watson Jan 13, 2025
4a1567c
Undid hashing out
RachelMurray-Watson Jan 14, 2025
86878eb
Tidied
RachelMurray-Watson Jan 14, 2025
1fd422d
Added calculation based on the % in each district.
RachelMurray-Watson Jan 15, 2025
765a14e
Added in ssp126
RachelMurray-Watson Jan 15, 2025
bb9bb8a
Unneccessary
RachelMurray-Watson Jan 15, 2025
fbd064a
added 126
RachelMurray-Watson Jan 15, 2025
e39a404
Was using %, not proption... daft
RachelMurray-Watson Jan 15, 2025
f7a7044
Identical
RachelMurray-Watson Jan 15, 2025
36cb131
Changed median to mean
RachelMurray-Watson Jan 15, 2025
58b4f7b
Added conversion factor of births to pregnancies (1.4)
RachelMurray-Watson Jan 16, 2025
bba3ea4
Added ssp126
RachelMurray-Watson Jan 16, 2025
e25ea80
Changed median to mean
RachelMurray-Watson Jan 16, 2025
1605f46
Ensured that the cumulative numbers were also pregnancies.
RachelMurray-Watson Jan 16, 2025
c580b59
tidied
RachelMurray-Watson Jan 16, 2025
e4ff762
Added map of historical weather
RachelMurray-Watson Jan 16, 2025
a582531
Updated to aggregate over all years, and ensure that only the negativ…
RachelMurray-Watson Jan 17, 2025
2888408
Updated to aggregate over all years, and ensure that only the negativ…
RachelMurray-Watson Jan 17, 2025
f4b48ec
Added in calculation of cases disrupted due to precip in top 10%
RachelMurray-Watson Jan 17, 2025
ab553d2
graph aesthetics
RachelMurray-Watson Jan 17, 2025
f107e03
typo
RachelMurray-Watson Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
*.xlsx filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
resources/** filter=lfs diff=lfs merge=lfs -text
best_model_monthly_total_precip.pkl filter=lfs diff=lfs merge=lfs -text
6 changes: 6 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,3 +129,9 @@ docs/parameters.rst
docs/reference/modules.rst
docs/reference/tlo*.rst
docs/resources/**/*.rst

# Climate data
src/scripts/climate_change/assessing_CMIP6_data_min_med_max.ipynb
best_model_weather_monthly_total_precip.pkl
best_model_ANC_prediction_monthly_total_precip.pkl
best_model_monthly_total_precip.pkl
3 changes: 3 additions & 0 deletions resources/~$ResourceFile_Lifestyle_Enhanced.xlsx
View file
Open in desktop
Git LFS file not shown
4 changes: 4 additions & 0 deletions src/scripts/climate_change/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
src/scripts/climate_change/assessing_CMIP6_data_min_med_max.ipynb
best_model_weather_monthly_total_precip.pkl
best_model_ANC_prediction_monthly_total_precip.pkl
best_model_monthly_total_precip.pkl
Loading
Loading