Add pre-commit (#14)

worldbank · Feb 26, 2024 · 0baf500 · 0baf500
1 parent 5e13132
commit 0baf500
Show file tree

Hide file tree

Showing 45 changed files with 4,523 additions and 2,438 deletions.
diff --git a/.gitignore b/.gitignore
@@ -129,6 +129,6 @@ dmypy.json
 .pyre/
 
 # Project-specific
-_build/ 
+_build/
 docs/api/
-src/GOSTurban/_version.py
+src/GOSTurban/_version.py
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,39 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+      - id: check-ast
+      - id: check-json
+      - id: detect-aws-credentials
+        args: [--allow-missing-credentials]
+      - id: detect-private-key
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.39.0
+    hooks:
+      - id: markdownlint
+        name: Markdownlint
+        files: \.(md|mdown|markdown)$
+        args: [
+            "--disable=MD013", # line-length
+            "--disable=MD033", # no-inline-html
+          ]
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.6
+    hooks:
+      - id: codespell
+        name: codespell
+        description: Checks for common misspellings in text files
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.2.2
+    hooks:
+      - id: ruff
+        types_or: [python, pyi, jupyter]
+        args: [--fix]
+      - id: ruff-format
+        types_or: [python, pyi, jupyter]
diff --git a/build_commands.txt b/build_commands.txt
@@ -1,5 +1,5 @@
 # Commit to github
-git status 
+git status
 git add -A
 git commit -m "FUBAR"
 git push

diff --git a/docs/novel_context.md b/docs/novel_context.md
@@ -4,10 +4,10 @@
 
 The European Commission developed a globally consistent, people-centric definition of urban areas. The basic approach is to apply a threshold to population grids on both the minimum population density, and then on the minimum total population of the resulting settlements. While the team at the EC continues to advance and iterate on their methodology, we rely on the original definitions of urban they produced:
 
-| Urban area | Min Pop Density | Min Settlement Pop |  
+| Urban area | Min Pop Density | Min Settlement Pop |
 | --- | --- | --- |
-| Urban areas | 300 people/km2 | 5000 people |  
-| High density urban areas | 1500 people/km2 | 50000 people |  
+| Urban areas | 300 people/km2 | 5000 people |
+| High density urban areas | 1500 people/km2 | 50000 people |
 
 ## [Bellefon (2021)](https://www.sciencedirect.com/science/article/pii/S0094119019301032)
 
@@ -21,6 +21,6 @@ This method eschews the absolute density thresholds of the EC methodology and in
 
 | Urban area | Definition |
 | --- | --- |
-| Urban areas | contiguous pixels for which the density is above the 95th percentile of the counterfactual |  
+| Urban areas | contiguous pixels for which the density is above the 95th percentile of the counterfactual |
 | Cores | contiguous pixels within urban areas that are above the 95th percentile of the counterfactual within the urban core |
-| Cities | urban areas that have a core |  
+| Cities | urban areas that have a core |
diff --git a/docs/urbanization_comparison.md b/docs/urbanization_comparison.md
@@ -4,18 +4,18 @@ Map comparison is a complicated process, as there are many tools and approaches.
 
 ## Degree of Urbanization
 
-| Urban area | Min Pop Density | Min Settlement Pop |  
+| Urban area | Min Pop Density | Min Settlement Pop |
 | --- | --- | --- |
-| Urban areas | 300 people/km2 | 5000 people |  
-| High density urban areas | 1500 people/km2 | 50000 people |  
+| Urban areas | 300 people/km2 | 5000 people |
+| High density urban areas | 1500 people/km2 | 50000 people |
 
 ## Dartboard
 
 | Urban area | Definition |
 | --- | --- |
-| Urban areas | contiguous pixels for which the density is above the 95th percentile of the counterfactual |  
+| Urban areas | contiguous pixels for which the density is above the 95th percentile of the counterfactual |
 | Cores | contiguous pixels within urban areas that are above the 95th percentile of the counterfactual within the urban core |
-| Cities | urban areas that have a core |  
+| Cities | urban areas that have a core |
 
 Any attempt to compare these methods has to start with a question of what do we compare: the DoU method produces two layers, and the DB method produces three. After investigating the data it is clear that the DoU urban area is equivalent to the DB urban areas. However, the comparison of the DoU high density to the DB could be to either the Cores or the cities. In the figures below you can see the comparison to both, however, let's look at the nature of the comparison:
 

diff --git a/notebooks/Implementations/JUP_SURGP_GLO_B_D__LEI_Evaluation.ipynb b/notebooks/Implementations/JUP_SURGP_GLO_B_D__LEI_Evaluation.ipynb
@@ -28,22 +28,21 @@
     }
    ],
    "source": [
-    "import os, sys, logging\n",
+    "import os\n",
+    "import sys\n",
     "\n",
-    "import geojson, rasterio\n",
+    "import rasterio\n",
     "import rasterio.features\n",
     "\n",
     "import pandas as pd\n",
     "import numpy as np\n",
     "\n",
-    "from shapely.geometry import shape, GeometryCollection\n",
-    "from shapely.wkt import loads\n",
     "from matplotlib import pyplot\n",
-    "from rasterio.plot import show, show_hist\n",
+    "from rasterio.plot import show\n",
     "\n",
-    "#Import GOST urban functions\n",
+    "# Import GOST urban functions\n",
     "sys.path.append(\"../../\")\n",
-    "from src.LEI import *\n"
+    "from src.LEI import *"
    ]
   },
   {
@@ -54,16 +53,25 @@
    },
    "outputs": [],
    "source": [
-    "LEI_folder = '/home/wb411133/data/Projects/LEI'\n",
+    "LEI_folder = \"/home/wb411133/data/Projects/LEI\"\n",
     "results = {}\n",
     "GHSL_files = []\n",
     "for root, dirs, files in os.walk(LEI_folder):\n",
     "    if os.path.exists(os.path.join(root, \"GHSL.tif\")):\n",
     "        GHSL_files.append(os.path.join(root, \"GHSL.tif\"))\n",
     "    try:\n",
-    "        results[os.path.basename(root)] = [len(files), os.stat(os.path.join(root, \"GHSL.tif\")).st_size]\n",
+    "        results[os.path.basename(root)] = [\n",
+    "            len(files),\n",
+    "            os.stat(os.path.join(root, \"GHSL.tif\")).st_size,\n",
+    "        ]\n",
     "        if len(files) != 6:\n",
-    "            print(\"%s - %s\" % (os.path.basename(root), os.stat(os.path.join(root, \"GHSL.tif\")).st_size))\n",
+    "            print(\n",
+    "                \"%s - %s\"\n",
+    "                % (\n",
+    "                    os.path.basename(root),\n",
+    "                    os.stat(os.path.join(root, \"GHSL.tif\")).st_size,\n",
+    "                )\n",
+    "            )\n",
     "    except:\n",
     "        pass"
    ]
@@ -82,13 +90,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "root = '/home/wb411133/data/Projects/LEI/634/'\n",
+    "root = \"/home/wb411133/data/Projects/LEI/634/\"\n",
     "inputGHSL = os.path.join(root, \"GHSL.tif\")\n",
     "inRaster = rasterio.open(inputGHSL)\n",
     "inR = inRaster.read()\n",
     "\n",
-    "newR = (inR == 3).astype('int')\n",
-    "oldR = (np.isin(inR, [4,5,6])).astype('int')"
+    "newR = (inR == 3).astype(\"int\")\n",
+    "oldR = (np.isin(inR, [4, 5, 6])).astype(\"int\")"
    ]
   },
   {
@@ -97,9 +105,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "fig, (axr, axg) = pyplot.subplots(1, 2, figsize=(20,20))\n",
-    "show(oldR, ax=axr, title='OLD')\n",
-    "show(newR, ax=axg, title='NEW')"
+    "fig, (axr, axg) = pyplot.subplots(1, 2, figsize=(20, 20))\n",
+    "show(oldR, ax=axr, title=\"OLD\")\n",
+    "show(newR, ax=axg, title=\"NEW\")"
    ]
   },
   {
@@ -108,11 +116,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#write out raster to file\n",
+    "# write out raster to file\n",
     "outProperties = inRaster.profile\n",
-    "outRaster = outRaster.astype('int32')\n",
-    "outProperties['dtype'] = 'int32'\n",
-    "with rasterio.open(inputGHSL.replace(\".tif\", \"_LEI.tif\"), 'w', **outProperties) as out:\n",
+    "outRaster = outRaster.astype(\"int32\")\n",
+    "outProperties[\"dtype\"] = \"int32\"\n",
+    "with rasterio.open(inputGHSL.replace(\".tif\", \"_LEI.tif\"), \"w\", **outProperties) as out:\n",
     "    out.write(outRaster)"
    ]
   },
@@ -133,12 +141,12 @@
    "outputs": [],
    "source": [
     "for ghsl_file in GHSL_files:\n",
-    "    print(f'{ghsl_file}')\n",
+    "    print(f\"{ghsl_file}\")\n",
     "    out_file = ghsl_file.replace(\".tif\", \"new_LEI_90_00.csv\")\n",
     "    if not os.path.exists(out_file):\n",
-    "        lei = calculate_LEI(ghsl_file, old_list = [5,6], new_list=[4])\n",
-    "        xx = pd.DataFrame(lei, columns=['geometry', 'old', 'total'])\n",
-    "        xx['LEI'] = xx['old'] / xx['total']\n",
+    "        lei = calculate_LEI(ghsl_file, old_list=[5, 6], new_list=[4])\n",
+    "        xx = pd.DataFrame(lei, columns=[\"geometry\", \"old\", \"total\"])\n",
+    "        xx[\"LEI\"] = xx[\"old\"] / xx[\"total\"]\n",
     "        xx.to_csv(out_file)"
    ]
   },
@@ -149,12 +157,12 @@
    "outputs": [],
    "source": [
     "# Process LEI results\n",
-    "base_folder = '/home/wb411133/data/Projects/LEI'\n",
+    "base_folder = \"/home/wb411133/data/Projects/LEI\"\n",
     "all_results_files = []\n",
     "for root, folders, files in os.walk(base_folder):\n",
     "    for f in files:\n",
     "        if \"GHSLnew_LEI_90_00\" in f:\n",
-    "            all_results_files.append(os.path.join(root, f))\n"
+    "            all_results_files.append(os.path.join(root, f))"
    ]
   },
   {
@@ -178,7 +186,7 @@
    "source": [
     "all_results = pd.DataFrame(summarized_results).transpose()\n",
     "# Old test to determine which files were not processed correctly\n",
-    "#bas_res = all_results[all_results['Expansion'] == 123282000.0].index\n",
+    "# bas_res = all_results[all_results['Expansion'] == 123282000.0].index\n",
     "all_results.head()"
    ]
   },
@@ -225,8 +233,8 @@
    "outputs": [],
    "source": [
     "xx = pd.DataFrame(all_res).head().transpose()\n",
-    "xx.columns = ['built75', 'built90', 'built00', 'built14']\n",
-    "#xx[xx.index.isin(['1'])]\n",
+    "xx.columns = [\"built75\", \"built90\", \"built00\", \"built14\"]\n",
+    "# xx[xx.index.isin(['1'])]\n",
     "xx.head()"
    ]
   },
@@ -254,17 +262,23 @@
    "outputs": [],
    "source": [
     "csv_files = [x for x in os.listdir(LEI_folder) if x[-4:] == \".csv\"]\n",
-    "lei0014 = pd.read_csv(os.path.join(LEI_folder, 'Summarized_LEI_Results.csv'),index_col=0)\n",
+    "lei0014 = pd.read_csv(\n",
+    "    os.path.join(LEI_folder, \"Summarized_LEI_Results.csv\"), index_col=0\n",
+    ")\n",
     "lei0014.columns = [\"%s_0014\" % x for x in lei0014.columns]\n",
     "\n",
-    "lei9014 = pd.read_csv(os.path.join(LEI_folder, 'Summarized_LEI_Results_90_0014.csv'),index_col=0)\n",
+    "lei9014 = pd.read_csv(\n",
+    "    os.path.join(LEI_folder, \"Summarized_LEI_Results_90_0014.csv\"), index_col=0\n",
+    ")\n",
     "lei9014.columns = [\"%s_9014\" % x for x in lei9014.columns]\n",
     "\n",
-    "lei9000 = pd.read_csv(os.path.join(LEI_folder, 'Summarized_LEI_Results_90_00.csv'),index_col=0)\n",
+    "lei9000 = pd.read_csv(\n",
+    "    os.path.join(LEI_folder, \"Summarized_LEI_Results_90_00.csv\"), index_col=0\n",
+    ")\n",
     "lei9000.columns = [\"%s_9000\" % x for x in lei9000.columns]\n",
     "\n",
-    "built_area = pd.read_csv(\"/home/wb411133/temp/LEI_cities_built.csv\",index_col=0)\n",
-    "built_area.columns = [\"%s_BUILT\" % x for x in built_area.columns]\n"
+    "built_area = pd.read_csv(\"/home/wb411133/temp/LEI_cities_built.csv\", index_col=0)\n",
+    "built_area.columns = [\"%s_BUILT\" % x for x in built_area.columns]"
    ]
   },
   {
@@ -282,7 +296,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "combined_results.to_csv(os.path.join(LEI_folder, 'LEI_COMBINED.csv'))"
+    "combined_results.to_csv(os.path.join(LEI_folder, \"LEI_COMBINED.csv\"))"
    ]
   },
   {
@@ -291,7 +305,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "combined_results['Expansion_0014'] + combined_results['Infill_0014'] + combined_results['Leapfrog_0014'] - (combined_results['built14_BUILT'] - combined_results['built00_BUILT'])"
+    "(\n",
+    "    combined_results[\"Expansion_0014\"]\n",
+    "    + combined_results[\"Infill_0014\"]\n",
+    "    + combined_results[\"Leapfrog_0014\"]\n",
+    "    - (combined_results[\"built14_BUILT\"] - combined_results[\"built00_BUILT\"])\n",
+    ")"
    ]
   },
   {
@@ -319,7 +338,7 @@
    "source": [
     "in_ghsl = \"/home/wb411133/data/Projects/LEI/1/GHSL.tif\"\n",
     "inR = rasterio.open(in_ghsl)\n",
-    "inD = inR.read()\n"
+    "inD = inR.read()"
    ]
   },
   {
@@ -357,8 +376,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "lei_2000_2014 = calculate_LEI(in_ghsl, old_list = [4,5,6], new_list=[3])\n",
-    "lei_1990_2000 = calculate_LEI(in_ghsl, old_list = [5,6], new_list=[4])"
+    "lei_2000_2014 = calculate_LEI(in_ghsl, old_list=[4, 5, 6], new_list=[3])\n",
+    "lei_1990_2000 = calculate_LEI(in_ghsl, old_list=[5, 6], new_list=[4])"
    ]
   },
   {
@@ -367,8 +386,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "xx = pd.DataFrame(lei, columns=['geometry', 'old', 'total'])\n",
-    "xx['LEI'] = xx['old'] / xx['total']        "
+    "xx = pd.DataFrame(lei, columns=[\"geometry\", \"old\", \"total\"])\n",
+    "xx[\"LEI\"] = xx[\"old\"] / xx[\"total\"]"
    ]
   },
   {