🔨 engineering: switch from threads to asyncio in grapher upserts

[Marigold]

Switch from threads to asyncio when upserting to MySQL. This simplifies the code a bit and might make it slightly faster, but at the cost of introducing asyncio to our codebase. Whether this is worth it depends on how faster and convenient grapher step becomes.

@pabloarosado if you run into a situation where grapher upserts are annoyingly long, can you ping me and I can revisit the performance on that concrete example? (By the way, we've recently merged an improvement that upserts only indicators that changed, not the entire dataset. This could already speed things up)

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs

Login: ssh owid@staging-site-asyncio-upserts

chart-diff: ✅

No charts for review.

data-diff: ❌ Found differences

= Dataset garden/bgs/2024-07-09/world_mineral_statistics
  = Table world_mineral_statistics
    ~ Column production (changed data)
        ~ Changed values: 13 / 91105 (0.01%)
           country  year commodity sub_commodity  production -  production +
          Colombia  1972  Chromium          Mine           NaN         200.0
          Colombia  1974  Chromium          Mine           NaN       12000.0
          Colombia  1976  Chromium          Mine           NaN        5000.0
             World  1973  Chromium          Mine     6779334.0     6791334.0
             World  1974  Chromium          Mine     7452837.0     7464837.0
  = Table world_mineral_statistics_flat
    ~ Column production_chromium_mine_tonnes (changed data)
        ~ Changed values: 13 / 8121 (0.16%)
           country  year  production_chromium_mine_tonnes -  production_chromium_mine_tonnes +
          Colombia  1972                               <NA>                                200
          Colombia  1974                               <NA>                              12000
          Colombia  1976                               <NA>                               5000
             World  1973                            6779334                            6791334
             World  1974                            7452837                            7464837
= Dataset garden/cancer/2024-08-30/gco_alcohol
  = Table gco_alcohol
    ~ Column value (changed data)
        ~ Changed values: 2192 / 11040 (19.86%)
               country  year    sex     cancer indicator value - value +
                Brazil  2020 female     larynx       asr    0.05    <0.1
          Burkina Faso  2020 female    pharynx       asr    0.05    <0.1
              Colombia  2020 female oesophagus       asr    0.05    <0.1
             Mauritius  2020 female     larynx       asr    0.05    <0.1
                  Oman  2020   male     breast       asr    0.05    <0.1
= Dataset garden/iea/2024-07-04/critical_minerals
- - Table demand_by_scenario
-   - Column demand_graphite__battery_grade__grid_battery_storage
-   - Column demand_cobalt_grid_battery_storage
-   - Column demand_copper_grid_battery_storage
-   - Column demand_lithium_grid_battery_storage
-   - Column demand_manganese_grid_battery_storage
-   - Column demand_nickel_grid_battery_storage
-   - Column demand_silicon_grid_battery_storage
-   - Column demand_vanadium_grid_battery_storage
-   - Column demand_graphite__battery_grade__electric_vehicles
-   - Column demand_cobalt_electric_vehicles
-   - Column demand_copper_electric_vehicles
-   - Column demand_lithium_electric_vehicles
-   - Column demand_manganese_electric_vehicles
-   - Column demand_nickel_electric_vehicles
-   - Column demand_silicon_electric_vehicles
-   - Column demand_cobalt_hydrogen_technologies
-   - Column demand_copper_hydrogen_technologies
-   - Column demand_iridium_hydrogen_technologies
-   - Column demand_nickel_hydrogen_technologies
-   - Column demand_pgms__other_than_iridium__hydrogen_technologies
-   - Column demand_yttrium_hydrogen_technologies
-   - Column demand_zirconium_hydrogen_technologies
-   - Column demand_arsenic_solar_pv
-   - Column demand_cadmium_solar_pv
-   - Column demand_copper_solar_pv
-   - Column demand_gallium_solar_pv
-   - Column demand_germanium_solar_pv
-   - Column demand_indium_solar_pv
-   - Column demand_lead_solar_pv
-   - Column demand_molybdenum_solar_pv
-   - Column demand_nickel_solar_pv
-   - Column demand_selenium_solar_pv
-   - Column demand_silicon_solar_pv
-   - Column demand_silver_solar_pv
-   - Column demand_tellurium_solar_pv
-   - Column demand_tin_solar_pv
-   - Column demand_zinc_solar_pv
-   - Column demand_boron_wind
-   - Column demand_chromium_wind
-   - Column demand_copper_wind
-   - Column demand_manganese_wind
-   - Column demand_molybdenum_wind
-   - Column demand_nickel_wind
-   - Column demand_zinc_wind
-   - Column demand_magnet_rare_earth_elements_electric_vehicles
-   - Column demand_magnet_rare_earth_elements_wind
-   - Column demand_graphite__other_than_battery_grade__grid_battery_storage
-   - Column demand_graphite__other_than_battery_grade__electric_vehicles
  ~ Table demand_for_key_minerals (changed metadata)
+     +   - country
-     -   - process
    ~ Column case (changed data)
-       - Removed values: 194 / 2582 (7.51%)
                                              case
          Wider adoption of perovskite solar cells
                  Wider use of silicon-rich anodes
                  Wider use of silicon-rich anodes
                  Wider use of silicon-rich anodes
                  Wider use of silicon-rich anodes
        ~ Changed values: 1178 / 2582 (45.62%)
                                            case -                                   case +
                 Comeback of high Cd-Te technology   Constrained rare earth elements supply
            Faster uptake of solid state batteries                     High material prices
                              High material prices                      Lower battery sizes
                Wider adoption of Ga-As technology Wider adoption of perovskite solar cells
          Wider adoption of perovskite solar cells         Wider use of silicon-rich anodes
    ~ Column country (changed data)
-       - Removed values: 194 / 2582 (7.51%)
           country
               NaN
               NaN
               NaN
               NaN
               NaN
        ~ Changed values: 2582 / 2582 (100.00%)
           country - country +
                 NaN     World
                 NaN     World
                 NaN     World
                 NaN     World
                 NaN     World
    ~ Column demand (changed metadata, changed data)
-       -     - Metals & Minerals
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 194 / 2582 (7.51%)
                 demand
             893.510986
          685564.000000
          371682.000000
          916856.000000
           40277.394531
        ~ Changed values: 2579 / 2582 (99.88%)
              demand -     demand +
          3.813240e+05 1.699650e+06
          3.115778e+04 1.734660e+02
          1.744870e+06 3.448460e+02
          1.690210e+02 4.927660e+02
          6.790010e+02 6.645610e+06
    ~ Column demand_as_a_share_of_global_demand (changed metadata, changed data)
+       + {}
-       - origins:
-       -   - producer: International Energy Agency
-       -     title: Critical Minerals Dataset
-       -     description: |-
-       -       In its first release, this dataset provides demand projection results for 37 key metals and minerals that are critical to the clean energy sector under different scenarios and technology cases, including 12 alternative cases modeling the impact of different technologies or consumer behaviours.
-       -       This edition is also updated with long-term supply projections for mining and refining of several key minerals.
-       -     citation_full: |-
-       -       International Energy Agency (2024), Critical Minerals Data Explorer, IEA, Paris https://www.iea.org/data-and-statistics/data-tools/critical-minerals-data-explorer
-       -     attribution_short: IEA
-       -     url_main: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset
-       -     date_accessed: '2024-07-04'
-       -     date_published: '2024-05-17'
-       -     license:
-       -       name: CC BY 4.0
-       -       url: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset#license
-       - processing_level: major
-       - presentation:
-       -   attribution_short: IEA
-       -   topic_tags:
-       -     - Metals & Minerals
-       -     - Energy
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 194 / 2582 (7.51%)
           demand_as_a_share_of_global_demand
                                          NaN
                                          NaN
                                          NaN
                                          NaN
                                          NaN
        ~ Changed values: 458 / 2582 (17.74%)
           demand_as_a_share_of_global_demand -  demand_as_a_share_of_global_demand +
                                      12.730516                                   NaN
                                      46.021126                                   NaN
                                       1.291378                                   NaN
                                       4.691339                                   NaN
                                      41.831516                                   NaN
    ~ Column demand_as_a_share_of_global_supply (changed metadata, changed data)
+       + {}
-       - origins:
-       -   - producer: International Energy Agency
-       -     title: Critical Minerals Dataset
-       -     description: |-
-       -       In its first release, this dataset provides demand projection results for 37 key metals and minerals that are critical to the clean energy sector under different scenarios and technology cases, including 12 alternative cases modeling the impact of different technologies or consumer behaviours.
-       -       This edition is also updated with long-term supply projections for mining and refining of several key minerals.
-       -     citation_full: |-
-       -       International Energy Agency (2024), Critical Minerals Data Explorer, IEA, Paris https://www.iea.org/data-and-statistics/data-tools/critical-minerals-data-explorer
-       -     attribution_short: IEA
-       -     url_main: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset
-       -     date_accessed: '2024-07-04'
-       -     date_published: '2024-05-17'
-       -     license:
-       -       name: CC BY 4.0
-       -       url: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset#license
-       - processing_level: major
-       - presentation:
-       -   attribution_short: IEA
-       -   topic_tags:
-       -     - Metals & Minerals
-       -     - Energy
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 194 / 2582 (7.51%)
           demand_as_a_share_of_global_supply
                                          NaN
                                          NaN
                                          NaN
                                          NaN
                                          NaN
        ~ Changed values: 296 / 2582 (11.46%)
           demand_as_a_share_of_global_supply -  demand_as_a_share_of_global_supply +
                                       1.503444                                   NaN
                                       0.021848                                   NaN
                                      91.520851                                   NaN
                                       0.019531                                   NaN
                                      16.063564                                   NaN
    ~ Column mineral (changed data)
-       - Removed values: 194 / 2582 (7.51%)
                             mineral
                              Nickel
            Graphite (battery-grade)
                           Manganese
                           Manganese
          Magnet rare earth elements
        ~ Changed values: 2489 / 2582 (96.40%)
          mineral -                  mineral +
             Copper                        Tin
             Silver Magnet rare earth elements
            Lithium Magnet rare earth elements
               Lead                     Indium
               Zinc                  Manganese
    ~ Column process (changed data)
-       - Removed values: 194 / 2582 (7.51%)
           process
          Refinery
          Refinery
          Refinery
          Refinery
          Refinery
        ~ Changed values: 2582 / 2582 (100.00%)
          process -  process +
           Refinery        NaN
           Refinery        NaN
           Refinery        NaN
           Refinery        NaN
           Refinery        NaN
    ~ Column scenario (changed data)
-       - Removed values: 194 / 2582 (7.51%)
                   scenario
           Net zero by 2050
          Announced pledges
          Announced pledges
          Announced pledges
            Stated policies
        ~ Changed values: 1868 / 2582 (72.35%)
                 scenario -        scenario +
            Stated policies Announced pledges
          Announced pledges   Stated policies
           Net zero by 2050 Announced pledges
           Net zero by 2050   Stated policies
            Stated policies  Net zero by 2050
    ~ Column technology (changed data)
-       - Removed values: 194 / 2582 (7.51%)
                 technology
                   Solar PV
          Electric vehicles
          Electric vehicles
          Electric vehicles
          Electric vehicles
        ~ Changed values: 1553 / 2582 (60.15%)
                  technology -      technology +
                      Solar PV Electric vehicles
          Grid battery storage Electric vehicles
                    Other uses          Solar PV
                      Solar PV Electric vehicles
                      Solar PV Electric vehicles
    ~ Column year (changed data)
-       - Removed values: 194 / 2582 (7.51%)
           year
           2050
           2023
           2030
           2035
           2050
        ~ Changed values: 2036 / 2582 (78.85%)
           year -  year +
             2030  2035.0
             2040  2035.0
             2030  2050.0
             2035  2030.0
             2040  2035.0
- - Table supply_by_country
-   - Column supply_lithium_refinery_base_case_all_scenarios
-   - Column supply_lithium_mine_base_case_all_scenarios
-   - Column supply_cobalt_mine_base_case_all_scenarios
-   - Column supply_magnet_rare_earth_elements_mine_base_case_all_scenarios
-   - Column supply_magnet_rare_earth_elements_refinery_base_case_all_scenarios
-   - Column supply_nickel_mine_base_case_all_scenarios
-   - Column supply_nickel_refinery_base_case_all_scenarios
-   - Column supply_graphite_mine_base_case_all_scenarios
-   - Column supply_cobalt_refinery_base_case_all_scenarios
-   - Column supply_graphite_refinery_base_case_all_scenarios
-   - Column supply_copper_mine_base_case_all_scenarios
-   - Column supply_copper_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_demand_lithium_refinery_base_case_announced_pledges
-   - Column supply_as_a_share_of_global_demand_lithium_refinery_base_case_net_zero_by_2050
-   - Column supply_as_a_share_of_global_demand_lithium_refinery_base_case_stated_policies
-   - Column supply_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_announced_pledges
-   - Column supply_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_net_zero_by_2050
-   - Column supply_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_stated_policies
-   - Column supply_as_a_share_of_global_demand_nickel_refinery_base_case_announced_pledges
-   - Column supply_as_a_share_of_global_demand_nickel_refinery_base_case_net_zero_by_2050
-   - Column supply_as_a_share_of_global_demand_nickel_refinery_base_case_stated_policies
-   - Column supply_as_a_share_of_global_demand_cobalt_refinery_base_case_announced_pledges
-   - Column supply_as_a_share_of_global_demand_cobalt_refinery_base_case_net_zero_by_2050
-   - Column supply_as_a_share_of_global_demand_cobalt_refinery_base_case_stated_policies
-   - Column supply_as_a_share_of_global_demand_copper_refinery_base_case_announced_pledges
-   - Column supply_as_a_share_of_global_demand_copper_refinery_base_case_net_zero_by_2050
-   - Column supply_as_a_share_of_global_demand_copper_refinery_base_case_stated_policies
-   - Column supply_as_a_share_of_global_supply_lithium_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_lithium_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_cobalt_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_magnet_rare_earth_elements_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_magnet_rare_earth_elements_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_nickel_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_nickel_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_graphite_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_cobalt_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_graphite_refinery_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_copper_mine_base_case_all_scenarios
-   - Column supply_as_a_share_of_global_supply_copper_refinery_base_case_all_scenarios
- - Table demand_by_technology
-   - Column demand_graphite__battery_grade__refinery_base_case_announced_pledges
-   - Column demand_graphite__battery_grade__refinery_base_case_net_zero_by_2050
-   - Column demand_graphite__battery_grade__refinery_base_case_stated_policies
-   - Column demand_cobalt_refinery_base_case_announced_pledges
-   - Column demand_cobalt_refinery_base_case_net_zero_by_2050
-   - Column demand_cobalt_refinery_base_case_stated_policies
-   - Column demand_copper_refinery_base_case_announced_pledges
-   - Column demand_copper_refinery_base_case_net_zero_by_2050
-   - Column demand_copper_refinery_base_case_stated_policies
-   - Column demand_lithium_refinery_base_case_announced_pledges
-   - Column demand_lithium_refinery_base_case_net_zero_by_2050
-   - Column demand_lithium_refinery_base_case_stated_policies
-   - Column demand_manganese_refinery_base_case_announced_pledges
-   - Column demand_manganese_refinery_base_case_net_zero_by_2050
-   - Column demand_manganese_refinery_base_case_stated_policies
-   - Column demand_nickel_refinery_base_case_announced_pledges
-   - Column demand_nickel_refinery_base_case_net_zero_by_2050
-   - Column demand_nickel_refinery_base_case_stated_policies
-   - Column demand_silicon_refinery_base_case_announced_pledges
-   - Column demand_silicon_refinery_base_case_net_zero_by_2050
-   - Column demand_silicon_refinery_base_case_stated_policies
-   - Column demand_vanadium_refinery_base_case_announced_pledges
-   - Column demand_vanadium_refinery_base_case_net_zero_by_2050
-   - Column demand_vanadium_refinery_base_case_stated_policies
-   - Column demand_copper_refinery_wider_direct_current__dc__technology_development_announced_pledges
-   - Column demand_copper_refinery_wider_direct_current__dc__technology_development_net_zero_by_2050
-   - Column demand_copper_refinery_wider_direct_current__dc__technology_development_stated_policies
-   - Column demand_graphite__battery_grade__refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_graphite__battery_grade__refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_graphite__battery_grade__refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_cobalt_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_cobalt_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_cobalt_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_copper_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_copper_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_copper_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_lithium_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_lithium_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_lithium_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_manganese_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_manganese_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_manganese_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_nickel_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_nickel_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_nickel_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_silicon_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_silicon_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_silicon_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_graphite__battery_grade__refinery_high_material_prices_announced_pledges
-   - Column demand_graphite__battery_grade__refinery_high_material_prices_net_zero_by_2050
-   - Column demand_graphite__battery_grade__refinery_high_material_prices_stated_policies
-   - Column demand_cobalt_refinery_high_material_prices_announced_pledges
-   - Column demand_cobalt_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_cobalt_refinery_high_material_prices_stated_policies
-   - Column demand_copper_refinery_high_material_prices_announced_pledges
-   - Column demand_copper_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_copper_refinery_high_material_prices_stated_policies
-   - Column demand_lithium_refinery_high_material_prices_announced_pledges
-   - Column demand_lithium_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_lithium_refinery_high_material_prices_stated_policies
-   - Column demand_manganese_refinery_high_material_prices_announced_pledges
-   - Column demand_manganese_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_manganese_refinery_high_material_prices_stated_policies
-   - Column demand_nickel_refinery_high_material_prices_announced_pledges
-   - Column demand_nickel_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_nickel_refinery_high_material_prices_stated_policies
-   - Column demand_silicon_refinery_high_material_prices_announced_pledges
-   - Column demand_silicon_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_silicon_refinery_high_material_prices_stated_policies
-   - Column demand_graphite__battery_grade__refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_cobalt_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_copper_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_lithium_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_manganese_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_nickel_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_silicon_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_graphite__battery_grade__refinery_lower_battery_sizes_announced_pledges
-   - Column demand_graphite__battery_grade__refinery_lower_battery_sizes_stated_policies
-   - Column demand_cobalt_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_cobalt_refinery_lower_battery_sizes_stated_policies
-   - Column demand_copper_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_copper_refinery_lower_battery_sizes_stated_policies
-   - Column demand_lithium_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_lithium_refinery_lower_battery_sizes_stated_policies
-   - Column demand_manganese_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_manganese_refinery_lower_battery_sizes_stated_policies
-   - Column demand_nickel_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_nickel_refinery_lower_battery_sizes_stated_policies
-   - Column demand_silicon_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_silicon_refinery_lower_battery_sizes_stated_policies
-   - Column demand_graphite__battery_grade__refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_graphite__battery_grade__refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_graphite__battery_grade__refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_cobalt_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_cobalt_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_cobalt_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_copper_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_copper_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_copper_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_lithium_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_lithium_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_lithium_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_manganese_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_manganese_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_manganese_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_nickel_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_nickel_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_nickel_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_silicon_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_silicon_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_silicon_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_iridium_refinery_base_case_announced_pledges
-   - Column demand_iridium_refinery_base_case_net_zero_by_2050
-   - Column demand_iridium_refinery_base_case_stated_policies
-   - Column demand_pgms__other_than_iridium__refinery_base_case_announced_pledges
-   - Column demand_pgms__other_than_iridium__refinery_base_case_net_zero_by_2050
-   - Column demand_pgms__other_than_iridium__refinery_base_case_stated_policies
-   - Column demand_yttrium_refinery_base_case_announced_pledges
-   - Column demand_yttrium_refinery_base_case_net_zero_by_2050
-   - Column demand_yttrium_refinery_base_case_stated_policies
-   - Column demand_zirconium_refinery_base_case_announced_pledges
-   - Column demand_zirconium_refinery_base_case_net_zero_by_2050
-   - Column demand_zirconium_refinery_base_case_stated_policies
-   - Column demand_arsenic_refinery_base_case_announced_pledges
-   - Column demand_arsenic_refinery_base_case_net_zero_by_2050
-   - Column demand_arsenic_refinery_base_case_stated_policies
-   - Column demand_cadmium_refinery_base_case_announced_pledges
-   - Column demand_cadmium_refinery_base_case_net_zero_by_2050
-   - Column demand_cadmium_refinery_base_case_stated_policies
-   - Column demand_gallium_refinery_base_case_announced_pledges
-   - Column demand_gallium_refinery_base_case_net_zero_by_2050
-   - Column demand_gallium_refinery_base_case_stated_policies
-   - Column demand_germanium_refinery_base_case_announced_pledges
-   - Column demand_germanium_refinery_base_case_net_zero_by_2050
-   - Column demand_germanium_refinery_base_case_stated_policies
-   - Column demand_indium_refinery_base_case_announced_pledges
-   - Column demand_indium_refinery_base_case_net_zero_by_2050
-   - Column demand_indium_refinery_base_case_stated_policies
-   - Column demand_lead_refinery_base_case_announced_pledges
-   - Column demand_lead_refinery_base_case_net_zero_by_2050
-   - Column demand_lead_refinery_base_case_stated_policies
-   - Column demand_molybdenum_refinery_base_case_announced_pledges
-   - Column demand_molybdenum_refinery_base_case_net_zero_by_2050
-   - Column demand_molybdenum_refinery_base_case_stated_policies
-   - Column demand_selenium_refinery_base_case_announced_pledges
-   - Column demand_selenium_refinery_base_case_net_zero_by_2050
-   - Column demand_selenium_refinery_base_case_stated_policies
-   - Column demand_silver_refinery_base_case_announced_pledges
-   - Column demand_silver_refinery_base_case_net_zero_by_2050
-   - Column demand_silver_refinery_base_case_stated_policies
-   - Column demand_tellurium_refinery_base_case_announced_pledges
-   - Column demand_tellurium_refinery_base_case_net_zero_by_2050
-   - Column demand_tellurium_refinery_base_case_stated_policies
-   - Column demand_tin_refinery_base_case_announced_pledges
-   - Column demand_tin_refinery_base_case_net_zero_by_2050
-   - Column demand_tin_refinery_base_case_stated_policies
-   - Column demand_zinc_refinery_base_case_announced_pledges
-   - Column demand_zinc_refinery_base_case_net_zero_by_2050
-   - Column demand_zinc_refinery_base_case_stated_policies
-   - Column demand_arsenic_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_arsenic_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_arsenic_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_cadmium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_cadmium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_cadmium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_copper_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_copper_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_copper_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_gallium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_gallium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_gallium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_germanium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_germanium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_germanium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_indium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_indium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_indium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_lead_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_lead_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_lead_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_molybdenum_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_molybdenum_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_molybdenum_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_nickel_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_nickel_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_nickel_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_selenium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_selenium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_selenium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_silicon_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_silicon_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_silicon_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_silver_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_silver_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_silver_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_tellurium_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_tellurium_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_tellurium_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_tin_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_tin_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_tin_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_zinc_refinery_comeback_of_high_cd_te_technology_announced_pledges
-   - Column demand_zinc_refinery_comeback_of_high_cd_te_technology_net_zero_by_2050
-   - Column demand_zinc_refinery_comeback_of_high_cd_te_technology_stated_policies
-   - Column demand_arsenic_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_arsenic_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_arsenic_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_cadmium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_cadmium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_cadmium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_copper_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_copper_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_copper_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_gallium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_gallium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_gallium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_germanium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_germanium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_germanium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_indium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_indium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_indium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_lead_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_lead_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_lead_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_molybdenum_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_molybdenum_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_molybdenum_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_nickel_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_nickel_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_nickel_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_selenium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_selenium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_selenium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_silicon_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_silicon_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_silicon_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_silver_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_silver_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_silver_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_tellurium_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_tellurium_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_tellurium_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_tin_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_tin_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_tin_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_zinc_refinery_wider_adoption_of_ga_as_technology_announced_pledges
-   - Column demand_zinc_refinery_wider_adoption_of_ga_as_technology_net_zero_by_2050
-   - Column demand_zinc_refinery_wider_adoption_of_ga_as_technology_stated_policies
-   - Column demand_arsenic_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_arsenic_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_arsenic_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_cadmium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_cadmium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_cadmium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_copper_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_copper_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_copper_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_gallium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_gallium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_gallium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_germanium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_germanium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_germanium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_indium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_indium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_indium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_lead_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_lead_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_lead_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_molybdenum_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_molybdenum_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_molybdenum_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_nickel_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_nickel_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_nickel_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_selenium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_selenium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_selenium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_silicon_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_silicon_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_silicon_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_silver_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_silver_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_silver_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_tellurium_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_tellurium_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_tellurium_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_tin_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_tin_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_tin_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_zinc_refinery_wider_adoption_of_perovskite_solar_cells_announced_pledges
-   - Column demand_zinc_refinery_wider_adoption_of_perovskite_solar_cells_net_zero_by_2050
-   - Column demand_zinc_refinery_wider_adoption_of_perovskite_solar_cells_stated_policies
-   - Column demand_boron_refinery_base_case_announced_pledges
-   - Column demand_boron_refinery_base_case_net_zero_by_2050
-   - Column demand_boron_refinery_base_case_stated_policies
-   - Column demand_chromium_refinery_base_case_announced_pledges
-   - Column demand_chromium_refinery_base_case_net_zero_by_2050
-   - Column demand_chromium_refinery_base_case_stated_policies
-   - Column demand_boron_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_boron_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_boron_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_chromium_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_chromium_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_chromium_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_copper_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_copper_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_copper_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_manganese_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_manganese_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_manganese_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_molybdenum_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_molybdenum_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_molybdenum_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_nickel_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_nickel_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_nickel_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_zinc_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_zinc_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_zinc_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_base_case_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_base_case_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_base_case_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_faster_uptake_of_solid_state_batteries_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_faster_uptake_of_solid_state_batteries_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_faster_uptake_of_solid_state_batteries_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_high_material_prices_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_high_material_prices_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_high_material_prices_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_limited_battery_size_reduction_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_limited_battery_size_reduction_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_limited_battery_size_reduction_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_lower_battery_sizes_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_lower_battery_sizes_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_lower_battery_sizes_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_wider_use_of_silicon_rich_anodes_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_wider_use_of_silicon_rich_anodes_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_wider_use_of_silicon_rich_anodes_stated_policies
-   - Column demand_magnet_rare_earth_elements_refinery_constrained_rare_earth_elements_supply_announced_pledges
-   - Column demand_magnet_rare_earth_elements_refinery_constrained_rare_earth_elements_supply_net_zero_by_2050
-   - Column demand_magnet_rare_earth_elements_refinery_constrained_rare_earth_elements_supply_stated_policies
-   - Column demand_graphite__other_than_battery_grade__refinery_base_case_announced_pledges
-   - Column demand_graphite__other_than_battery_grade__refinery_base_case_net_zero_by_2050
-   - Column demand_graphite__other_than_battery_grade__refinery_base_case_stated_policies
-   - Column demand_graphite__natural_and_synthetic__refinery_base_case_announced_pledges
-   - Column demand_graphite__natural_and_synthetic__refinery_base_case_net_zero_by_2050
-   - Column demand_graphite__natural_and_synthetic__refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_cobalt_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_cobalt_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_cobalt_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_copper_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_copper_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_copper_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_lithium_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_lithium_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_lithium_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_nickel_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_nickel_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_nickel_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_magnet_rare_earth_elements_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_demand_graphite__natural_and_synthetic__refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_demand_graphite__natural_and_synthetic__refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_demand_graphite__natural_and_synthetic__refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_supply_cobalt_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_supply_cobalt_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_supply_cobalt_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_supply_copper_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_supply_copper_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_supply_copper_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_supply_lithium_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_supply_lithium_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_supply_lithium_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_supply_nickel_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_supply_nickel_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_supply_nickel_refinery_base_case_stated_policies
-   - Column demand_as_a_share_of_global_supply_magnet_rare_earth_elements_refinery_base_case_announced_pledges
-   - Column demand_as_a_share_of_global_supply_magnet_rare_earth_elements_refinery_base_case_net_zero_by_2050
-   - Column demand_as_a_share_of_global_supply_magnet_rare_earth_elements_refinery_base_case_stated_policies
  ~ Table supply_for_key_minerals (changed metadata)
-     -   - scenario
    ~ Column case (changed data)
-       - Removed values: 272 / 340 (80.00%)
               case
          Base case
          Base case
          Base case
          Base case
          Base case
    ~ Column country (changed data)
-       - Removed values: 272 / 340 (80.00%)
          country
           Russia
            World
            World
            World
            World
        ~ Changed values: 312 / 340 (91.76%)
                             country -     country +
                                Canada         China
                                 China     Indonesia
          Democratic Republic of Congo Rest of world
                               Finland Rest of world
                                 India        Russia
    ~ Column mineral (changed metadata, changed data)
+       + {}
-       - origins:
-       -   - producer: International Energy Agency
-       -     title: Critical Minerals Dataset
-       -     description: |-
-       -       In its first release, this dataset provides demand projection results for 37 key metals and minerals that are critical to the clean energy sector under different scenarios and technology cases, including 12 alternative cases modeling the impact of different technologies or consumer behaviours.
-       -       This edition is also updated with long-term supply projections for mining and refining of several key minerals.
-       -     citation_full: |-
-       -       International Energy Agency (2024), Critical Minerals Data Explorer, IEA, Paris https://www.iea.org/data-and-statistics/data-tools/critical-minerals-data-explorer
-       -     attribution_short: IEA
-       -     url_main: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset
-       -     date_accessed: '2024-07-04'
-       -     date_published: '2024-05-17'
-       -     license:
-       -       name: CC BY 4.0
-       -       url: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset#license

-       - Removed values: 272 / 340 (80.00%)
                             mineral
                              Copper
                              Nickel
          Magnet rare earth elements
                             Lithium
                             Lithium
        ~ Changed values: 281 / 340 (82.65%)
                           mineral - mineral +
          Magnet rare earth elements   Lithium
                              Nickel   Lithium
                             Lithium    Copper
                              Nickel  Graphite
                              Cobalt    Copper
    ~ Column process (changed data)
-       - Removed values: 272 / 340 (80.00%)
           process
          Refinery
          Refinery
          Refinery
          Refinery
          Refinery
        ~ Changed values: 340 / 340 (100.00%)
          process -        process +
           Refinery           Mining
           Refinery         Refining
           Refinery Mining (natural)
           Refinery           Mining
           Refinery         Refining
    ~ Column scenario (changed data)
-       - Removed values: 272 / 340 (80.00%)
                   scenario
           Net zero by 2050
            Stated policies
          Announced pledges
            Stated policies
           Net zero by 2050
        ~ Changed values: 340 / 340 (100.00%)
                 scenario -  scenario +
              All scenarios         NaN
           Net zero by 2050         NaN
            Stated policies         NaN
           Net zero by 2050         NaN
          Announced pledges         NaN
    ~ Column supply (changed metadata, changed data)
-       -     - Metals & Minerals
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 272 / 340 (80.00%)
             supply
          1059000.0
          3932000.0
           105600.0
           369700.0
           369700.0
        ~ Changed values: 338 / 340 (99.41%)
            supply -  supply +
            135900.0   34000.0
          14788000.0  127900.0
          14788000.0  675300.0
           1413700.0   38600.0
              4000.0 3090800.0
    ~ Column supply_as_a_share_of_global_demand (changed metadata, changed data)
+       + {}
-       - origins:
-       -   - producer: International Energy Agency
-       -     title: Critical Minerals Dataset
-       -     description: |-
-       -       In its first release, this dataset provides demand projection results for 37 key metals and minerals that are critical to the clean energy sector under different scenarios and technology cases, including 12 alternative cases modeling the impact of different technologies or consumer behaviours.
-       -       This edition is also updated with long-term supply projections for mining and refining of several key minerals.
-       -     citation_full: |-
-       -       International Energy Agency (2024), Critical Minerals Data Explorer, IEA, Paris https://www.iea.org/data-and-statistics/data-tools/critical-minerals-data-explorer
-       -     attribution_short: IEA
-       -     url_main: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset
-       -     date_accessed: '2024-07-04'
-       -     date_published: '2024-05-17'
-       -     license:
-       -       name: CC BY 4.0
-       -       url: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset#license
-       - processing_level: major
-       - presentation:
-       -   attribution_short: IEA
-       -   topic_tags:
-       -     - Metals & Minerals
-       -     - Energy
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 272 / 340 (80.00%)
           supply_as_a_share_of_global_demand
                                     4.086436
                                   126.654900
                                    79.113815
                                    50.240082
                                    25.838745
        ~ Changed values: 252 / 340 (74.12%)
           supply_as_a_share_of_global_demand -  supply_as_a_share_of_global_demand +
                                      11.839811                                   NaN
                                      44.557209                                   NaN
                                      50.699120                                   NaN
                                       8.753393                                   NaN
                                       5.642195                                   NaN
    ~ Column supply_as_a_share_of_global_supply (changed metadata, changed data)
+       + {}
-       - origins:
-       -   - producer: International Energy Agency
-       -     title: Critical Minerals Dataset
-       -     description: |-
-       -       In its first release, this dataset provides demand projection results for 37 key metals and minerals that are critical to the clean energy sector under different scenarios and technology cases, including 12 alternative cases modeling the impact of different technologies or consumer behaviours.
-       -       This edition is also updated with long-term supply projections for mining and refining of several key minerals.
-       -     citation_full: |-
-       -       International Energy Agency (2024), Critical Minerals Data Explorer, IEA, Paris https://www.iea.org/data-and-statistics/data-tools/critical-minerals-data-explorer
-       -     attribution_short: IEA
-       -     url_main: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset
-       -     date_accessed: '2024-07-04'
-       -     date_published: '2024-05-17'
-       -     license:
-       -       name: CC BY 4.0
-       -       url: https://www.iea.org/data-and-statistics/data-product/critical-minerals-dataset#license
-       - processing_level: major
-       - presentation:
-       -   attribution_short: IEA
-       -   topic_tags:
-       -     - Metals & Minerals
-       -     - Energy
-       - description_processing: |-
-       -   - We have integrated data from multiple sheets within the IEA's Critical Minerals Dataset to provide a comprehensive overview.
-       -   - Where applicable, we have merged supply and demand data, enabling the calculation of relative shares for a more detailed analysis.

-       - Removed values: 272 / 340 (80.00%)
           supply_as_a_share_of_global_supply
                                     4.021081
                                   100.000000
                                   100.000000
                                   100.000000
                                   100.000000
        ~ Changed values: 340 / 340 (100.00%)
           supply_as_a_share_of_global_supply -  supply_as_a_share_of_global_supply +
                                      91.717911                                   NaN
                                       8.400358                                   NaN
                                      35.953712                                   NaN
                                       2.844359                                   NaN
                                       5.691785                                   NaN
    ~ Column year (changed data)
-       - Removed values: 272 / 340 (80.00%)
           year
           2023
           2023
           2030
           2035
           2040
        ~ Changed values: 255 / 340 (75.00%)
           year -  year +
             2030  2023.0
             2023  2030.0
             2040  2023.0
             2040  2030.0
             2040  2035.0
= Dataset garden/igh/2024-07-05/better_data_homelessness
  = Table better_data_homelessness
    ~ Column igh_framework_category_simplified (changed metadata)
-       -   - We only consider the data from the source that is recent.
    ~ Column people_experiencing_homelessness (changed data)
        ~ Changed values: 129 / 129 (100.00%)
                                   country  year  people_experiencing_homelessness - people_experiencing_homelessness +
                                   Bahrain  2015                                   0                                  0
                                     Congo  2018                             1500000                            1500000
                                   Georgia  2019                                1531                               1531
                                      Iran  2022                               20000                              20000
          Saint Vincent and the Grenadines  2012                                  85                                 85
    ~ Column total_homeless_data_scorecard (changed metadata)
-       -   - We only consider the data from the source that is recent.
-       -   title_public: Completeness of homelessness data
        ?                 ^^^^ ^ ^^^^^
+       +   title_public: Quality of homelessness data
        ?                 ^^^ ^ ^
= Dataset garden/minerals/2024-07-15/minerals
  = Table minerals
    ~ Column production_chromium_mine_tonnes (changed data)
        ~ Changed values: 7 / 8380 (0.08%)
           country  year  production_chromium_mine_tonnes -  production_chromium_mine_tonnes +
          Colombia  1970                               <NA>                                170
          Colombia  1971                               <NA>                                180
          Colombia  1972                               <NA>                                200
          Colombia  1973                               <NA>                              12000
          Colombia  1976                               <NA>                               5000
= Dataset garden/moatsos/2023-10-09/moatsos_historical_poverty
  = Table moatsos_historical_poverty
    ~ Column headcount_190_smooth (changed metadata)
+       +   title_public: $1.90 a day - Number in poverty
-       -   title_public: Number of people living in extreme poverty
-       -   title_variant: Long-run estimates
    ~ Column headcount_cbn (changed metadata)
+       +   title_public: Cost of Basic Needs - Number in poverty
-       -   title_public: Number of people living in extreme poverty
-       -   title_variant: Long-run estimates - Cost of basic needs
    ~ Column headcount_ratio_190_smooth (changed metadata)
-       -   title_public: Share of population living in extreme poverty
        ?                                     ---  ------------
+       +   title_public: $1.90 a day - Share of population in poverty
        ?                 ++++++++++++++
-       -   title_variant: Long-run estimates
    ~ Column headcount_ratio_cbn (changed metadata)
+       +   title_public: Cost of Basic Needs - Share of population in poverty
-       -   title_public: Share of population living in extreme poverty
-       -   title_variant: Long-run estimates - Cost of basic needs
= Dataset garden/wb/2024-03-27/world_bank_pip
  = Table consumption_2017_avg_shortfall_60_median
  = Table income_consumption_2017_decile2_share
  = Table income_2017_dec

...diff too long, truncated...

Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2024-11-20 10:38:01 UTC
Execution time: 3.98 seconds