Illegal_Gold_Mining_WA_TF_Training.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<table class=\"ee-notebook-buttons\" align=\"left\"><td>\n",
        "<a target=\"_blank\"  href=\"http://colab.research.google.com/github/biplovbhandari/tensorflow-ml-models/blob/master/Illegal_Gold_Mining_WA_TF_Training.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab</a>\n",
        "</td><td>\n",
        "<a target=\"_blank\"  href=\"https://github.com/biplovbhandari/tensorflow-ml-models/blob/master/Illegal_Gold_Mining_WA_TF_Training.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> View source on GitHub</a></td>\n",
        "<td>\n",
        "<a target=\"_blank\"  href=\"https://nbviewer.org/github/biplovbhandari/tensorflow-ml-models/blob/master/Illegal_Gold_Mining_WA_TF_Training.ipynb?flush_cache=True\"><img width=60px src=\"https://nbviewer.org/static/img/nav_logo.svg\" /> View on nbviewer</a></td>\n",
        "\n",
        "</table>"
      ],
      "metadata": {
        "id": "5qjSUsUizxgQ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7brFlUdrFYE"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "This is an Earth Engine <> TensorFlow demonstration notebook for developing a Deep Neural Networks (DNN) model for the Galamsey gold mining in West Africa. This notebook was developed as a part of the capacity building training to the West Africa Hub of the SERVIR.\n",
        "\n",
        "Specifically, this notebook shows:\n",
        "\n",
        "1.   Exporting training/testing data from Earth Engine in TFRecord format.\n",
        "2.   Preparing the data for use in a TensorFlow model.\n",
        "2.   Training and validating a simple model (Keras `Sequential` neural network and functional model) in TensorFlow.\n",
        "3.   Making predictions on image data exported from Earth Engine in TFRecord format.\n",
        "4.   Ingesting classified image data to Earth Engine in TFRecord format.\n",
        "5.   Using AI Platform for inference.\n",
        "6.   Using Vertex AI Platform for inference (coming soon)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Setup software libraries\n",
        "\n",
        "Import software libraries and/or authenticate as necessary."
      ],
      "metadata": {
        "id": "wv2-iNnx0vGW"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "5qMKG1hEXuML",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d67b8a97-142c-4425-91fb-5ee501964022"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using EE version  0.1.363\n",
            "Using Folium version  0.14.0\n",
            "Using TF version  2.12.0\n"
          ]
        }
      ],
      "source": [
        "import ee\n",
        "print(\"Using EE version \", ee.__version__)\n",
        "import folium\n",
        "print(\"Using Folium version \", folium.__version__)\n",
        "\n",
        "from google.api_core import exceptions, retry\n",
        "import google.auth\n",
        "from google.colab import auth\n",
        "\n",
        "import io\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "from datetime import datetime\n",
        "from functools import partial\n",
        "from pprint import pprint\n",
        "\n",
        "import random\n",
        "import requests\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from keras import layers\n",
        "print(\"Using TF version \", tf.__version__)\n",
        "\n",
        "from typing import Dict, Iterable, List, Tuple"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dEM3FP4YakJg"
      },
      "source": [
        "## Authenticate to Colab, Cloud and Earth Engine\n",
        "\n",
        "Identify yourself to Google Cloud, so you have access to storage and other resources.  When you run the code below, it will ask you to provide necessary permissions.  Follow the link to a page that will let you grant permission to the Cloud SDK to access your resources.\n",
        "\n",
        "(You may need to run this again if you get a credentials error later.)\n",
        "\n",
        "See [the EE auth reference](https://developers.google.com/earth-engine/guides/auth) for more info."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wszoNy7PrrZ1"
      },
      "outputs": [],
      "source": [
        "# Replace your-project with your project name.\n",
        "PROJECT = \"servir-ee\"\n",
        "\n",
        "BUCKET = \"wa-tf-training\"\n",
        "\n",
        "auth.authenticate_user()\n",
        "\n",
        "credentials, _ = google.auth.default(\n",
        "    scopes=[\n",
        "        \"https://www.googleapis.com/auth/cloud-platform\",\n",
        "        \"https://www.googleapis.com/auth/earthengine\",\n",
        "    ]\n",
        ")\n",
        "ee.Initialize(\n",
        "    credentials,\n",
        "    project=PROJECT,\n",
        "    opt_url=\"https://earthengine-highvolume.googleapis.com\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMkNljPGMzyG"
      },
      "source": [
        "# Define variables\n",
        "\n",
        "This set of global variables will be used throughout.  For this demo, you must have a Cloud Storage bucket into which you can write files.  ([learn more about creating Cloud Storage buckets](https://cloud.google.com/storage/docs/creating-buckets))."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zU_FsXzZr34G"
      },
      "outputs": [],
      "source": [
        "START = '2019-01-01'\n",
        "END = '2019-12-31'\n",
        "\n",
        "# A random spot near the study area\n",
        "COORDS = [-1.981041, 6.074769]\n",
        "TEST_POINT = ee.Geometry.Point(COORDS)\n",
        "\n",
        "REGION = ee.Geometry.Polygon(\n",
        "        [[[-3.2689309056402194, 8.045285352640722],\n",
        "          [-3.2689309056402194, 4.693792623646198],\n",
        "          [0.48839331310976064, 4.693792623646198],\n",
        "          [0.48839331310976064, 8.045285352640722]]])\n",
        "\n",
        "SCALE = 30        # Meters per pixel\n",
        "VALIDATION_RATIO = 0.2\n",
        "TEST_RATIO = 0.2\n",
        "N_CLASSES = 2 # mining/no-mining\n",
        "\n",
        "# Predictors.\n",
        "S1_BANDS = ['VV', 'VH', 's1_ratio', 's1_ndratio']\n",
        "\n",
        "S2_BANDS = ['B1', 'B2', 'B3', 'B4', 'B5', 'B6', 'B7',\n",
        "               'B8', 'B8A', 'B9', 'B11', 'B12']\n",
        "\n",
        "# Target variable.\n",
        "OUTPUT_BANDS = ['mining']\n",
        "\n",
        "OLD_LABEL = 'CID'\n",
        "\n",
        "# File name for the prediction (image) dataset.  The trained model will read\n",
        "# this dataset and make predictions in each pixel.\n",
        "IMAGE_FILE_PREFIX = 'WA_image'\n",
        "\n",
        "# The output path for the classified image (i.e. predictions) TFRecord file.\n",
        "OUTPUT_IMAGE_FILE = 'gs://' + BUCKET + '/WA_Classified.TFRecord'\n",
        "\n",
        "# Input stack.\n",
        "INPUT_BANDS = S1_BANDS + S2_BANDS\n",
        "BANDS = INPUT_BANDS + OUTPUT_BANDS\n",
        "\n",
        "EPOCHS = 20\n",
        "BATCH_SIZE = 32"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ICmelIuVvwCS"
      },
      "source": [
        "# Inspect and get your datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use Sentinel-2 and Sentinel-1 Dataset, plus several indices for this training. Feel free to add more dataset as you see appropriate."
      ],
      "metadata": {
        "id": "5YB80jxHmBPK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4DABPFcG42Fc"
      },
      "outputs": [],
      "source": [
        "# Helper function to set the class label (label=1)\n",
        "def set_class_label(f: ee.Feature) -> ee.Feature:\n",
        "    f = f.set(OUTPUT_BANDS[0], 1)\n",
        "    f = f.select([OUTPUT_BANDS[0]])\n",
        "    return f"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WFyvWM3O423M"
      },
      "outputs": [],
      "source": [
        "# Helper function to set the class label to the existing one\n",
        "def set_label(f: ee.Feature) -> ee.Feature:\n",
        "    f = f.set(OUTPUT_BANDS[0], f.get(OLD_LABEL))\n",
        "    f = f.select([OUTPUT_BANDS[0]])\n",
        "    return f"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "At this point, let's load our collected data of galamsey and non-galamsey points. (Check this [script](https://code.earthengine.google.com/84d3ba1026399cd581792da94a6cb36e) for the collection)."
      ],
      "metadata": {
        "id": "SINNQ78EsjWv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_added_galamsey = ee.FeatureCollection(\"projects/servir-wa/tf_training_aug_2023/user_added_galamsey\")\n",
        "user_added_non_galamsey = ee.FeatureCollection(\"projects/servir-wa/tf_training_aug_2023/user_added_non_galamsey\")"
      ],
      "metadata": {
        "id": "Oc3VuKRyswjJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CQTPmkhu3nRB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d9db561e-1f62-43f3-9d8c-67b76210e839"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'0': 2000}\n"
          ]
        }
      ],
      "source": [
        "galamsey_pts = ee.FeatureCollection(\"projects/servir-wa/services/IllegalMining/2019_galamsey_pts\")\n",
        "pprint(galamsey_pts.aggregate_histogram(OLD_LABEL).getInfo())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7LOFSLvt4cnc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5b171f8c-a51d-43c8-da09-066951317c38"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'1': 2000}\n"
          ]
        }
      ],
      "source": [
        "galamsey_pts = galamsey_pts.map(set_class_label)\n",
        "\n",
        "pprint(galamsey_pts.aggregate_histogram(OUTPUT_BANDS[0]).getInfo())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "djxrklFe3apo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d9c73498-0421-48fc-926a-a759e0fa20a9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'geometry': {'coordinates': [-2.0106086649987467, 5.854679287445995],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'mining': 1},\n",
            " 'type': 'Feature'}\n",
            "{'geometry': {'coordinates': [-2.055461814492921, 6.530161515380249],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'CID': 1},\n",
            " 'type': 'Feature'}\n",
            "{'geometry': {'coordinates': [-0.5952137390609912, 6.87835374216136],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'CID': 0},\n",
            " 'type': 'Feature'}\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, None, None)"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ],
      "source": [
        "# old labels still exists\n",
        "\n",
        "pprint(galamsey_pts.first().getInfo()), pprint(user_added_galamsey.first().getInfo()), pprint(user_added_non_galamsey.first().getInfo())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YPbdcll-3-m1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c074f30-a939-4dcd-89b4-70ee43ae0aed"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'geometry': {'coordinates': [-2.0106086649987467, 5.854679287445995],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'mining': 1},\n",
            " 'type': 'Feature'}\n",
            "{'geometry': {'coordinates': [-2.055461814492921, 6.530161515380249],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'mining': 1},\n",
            " 'type': 'Feature'}\n",
            "{'geometry': {'coordinates': [-0.5952137390609912, 6.87835374216136],\n",
            "              'type': 'Point'},\n",
            " 'id': '00000000000000000000',\n",
            " 'properties': {'mining': 0},\n",
            " 'type': 'Feature'}\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, None, None)"
            ]
          },
          "metadata": {},
          "execution_count": 103
        }
      ],
      "source": [
        "user_added_galamsey = user_added_galamsey.map(set_label)\n",
        "user_added_non_galamsey = user_added_non_galamsey.map(set_label)\n",
        "\n",
        "pprint(galamsey_pts.first().getInfo()), pprint(user_added_galamsey.first().getInfo()), pprint(user_added_non_galamsey.first().getInfo())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## All points data"
      ],
      "metadata": {
        "id": "6DQR8rmw2Yq0"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qdoT-OuOxTZi",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 771
        },
        "outputId": "5449f5ab-7a87-409e-d07e-10f309005b62"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<folium.folium.Map at 0x7ad5b82d0cd0>"
            ],
            "text/html": [
              "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><span style=\"color:#565656\">Make this Notebook Trusted to load map: File -> Trust Notebook</span><iframe srcdoc=\"&lt;!DOCTYPE html&gt;\n",
              "&lt;html&gt;\n",
              "&lt;head&gt;\n",
              "    \n",
              "    &lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;\n",
              "    \n",
              "        &lt;script&gt;\n",
              "            L_NO_TOUCH = false;\n",
              "            L_DISABLE_3D = false;\n",
              "        &lt;/script&gt;\n",
              "    \n",
              "    &lt;style&gt;html, body {width: 100%;height: 100%;margin: 0;padding: 0;}&lt;/style&gt;\n",
              "    &lt;style&gt;#map {position:absolute;top:0;bottom:0;right:0;left:0;}&lt;/style&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://code.jquery.com/jquery-1.12.4.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.0/css/all.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/gh/python-visualization/folium/folium/templates/leaflet.awesome.rotate.min.css&quot;/&gt;\n",
              "    \n",
              "            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,\n",
              "                initial-scale=1.0, maximum-scale=1.0, user-scalable=no&quot; /&gt;\n",
              "            &lt;style&gt;\n",
              "                #map_85bb099096c2d4335443c47f45ed741b {\n",
              "                    position: relative;\n",
              "                    width: 100.0%;\n",
              "                    height: 100.0%;\n",
              "                    left: 0.0%;\n",
              "                    top: 0.0%;\n",
              "                }\n",
              "                .leaflet-container { font-size: 1rem; }\n",
              "            &lt;/style&gt;\n",
              "        \n",
              "&lt;/head&gt;\n",
              "&lt;body&gt;\n",
              "    \n",
              "    \n",
              "            &lt;div class=&quot;folium-map&quot; id=&quot;map_85bb099096c2d4335443c47f45ed741b&quot; &gt;&lt;/div&gt;\n",
              "        \n",
              "&lt;/body&gt;\n",
              "&lt;script&gt;\n",
              "    \n",
              "    \n",
              "            var map_85bb099096c2d4335443c47f45ed741b = L.map(\n",
              "                &quot;map_85bb099096c2d4335443c47f45ed741b&quot;,\n",
              "                {\n",
              "                    center: [6.074769, -1.981041],\n",
              "                    crs: L.CRS.EPSG3857,\n",
              "                    zoom: 8,\n",
              "                    zoomControl: true,\n",
              "                    preferCanvas: false,\n",
              "                }\n",
              "            );\n",
              "\n",
              "            \n",
              "\n",
              "        \n",
              "    \n",
              "            var tile_layer_65359a58c6aa16ba478bd2b07a6b2f76 = L.tileLayer(\n",
              "                &quot;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Data by \\u0026copy; \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://openstreetmap.org\\&quot;\\u003eOpenStreetMap\\u003c/a\\u003e, under \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://www.openstreetmap.org/copyright\\&quot;\\u003eODbL\\u003c/a\\u003e.&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_85bb099096c2d4335443c47f45ed741b);\n",
              "        \n",
              "    \n",
              "            var tile_layer_cc0f4d9e86c54ad9c5b3d551fb0f4c84 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/ebb324517967bf72c0d062e51efa482d-fa1bf8f495bd51765b276f36e5148691/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_85bb099096c2d4335443c47f45ed741b);\n",
              "        \n",
              "    \n",
              "            var tile_layer_990b8bbd5f69079c18f4f824c492e26f = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/f1760ad778267ed6b1cbc52367a6f070-f50f916b6957e6e076bfd45a5f22e101/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_85bb099096c2d4335443c47f45ed741b);\n",
              "        \n",
              "    \n",
              "            var tile_layer_8b541ec2bad41ed372162bd68238c140 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/6ffe7d881b677840529222dc533b8999-576ef1efcf62d8f175959780a775029b/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_85bb099096c2d4335443c47f45ed741b);\n",
              "        \n",
              "    \n",
              "            var layer_control_cc4e483d0a3eaa442191e96ec758242e = {\n",
              "                base_layers : {\n",
              "                    &quot;openstreetmap&quot; : tile_layer_65359a58c6aa16ba478bd2b07a6b2f76,\n",
              "                },\n",
              "                overlays :  {\n",
              "                    &quot;galamsey_pts&quot; : tile_layer_cc0f4d9e86c54ad9c5b3d551fb0f4c84,\n",
              "                    &quot;non_galamsey_pts&quot; : tile_layer_990b8bbd5f69079c18f4f824c492e26f,\n",
              "                    &quot;REGION&quot; : tile_layer_8b541ec2bad41ed372162bd68238c140,\n",
              "                },\n",
              "            };\n",
              "            L.control.layers(\n",
              "                layer_control_cc4e483d0a3eaa442191e96ec758242e.base_layers,\n",
              "                layer_control_cc4e483d0a3eaa442191e96ec758242e.overlays,\n",
              "                {&quot;autoZIndex&quot;: true, &quot;collapsed&quot;: true, &quot;position&quot;: &quot;topright&quot;}\n",
              "            ).addTo(map_85bb099096c2d4335443c47f45ed741b);\n",
              "        \n",
              "&lt;/script&gt;\n",
              "&lt;/html&gt;\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ],
      "source": [
        "all_points = galamsey_pts.merge(user_added_galamsey).merge(user_added_non_galamsey)\n",
        "\n",
        "map = folium.Map(location=[COORDS[1], COORDS[0]], zoom_start=8)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=galamsey_pts.merge(user_added_galamsey).getMapId({'color': 'green'})[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"galamsey_pts\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=user_added_non_galamsey.getMapId({'color': 'red'})[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"non_galamsey_pts\",\n",
        "  ).add_to(map)\n",
        "\n",
        "roi_outline = ee.Image().byte()\\\n",
        "    .paint(featureCollection=REGION,\n",
        "           color=1,\n",
        "           width=3).getMapId()\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=roi_outline[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"REGION\",\n",
        "  ).add_to(map)\n",
        "\n",
        "\n",
        "map.add_child(folium.LayerControl())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JknaYTOUuPuc"
      },
      "source": [
        "## Sentinel-2 surface reflectance\n",
        "\n",
        "We will use a cloud-freee composite of Sentinel-2 surface reflectance data for predictors.  See [the Code Editor example](https://code.earthengine.google.com/?scriptPath=Examples%3ACloud%20Masking%2FSentinel2%20Cloud%20And%20Shadow) for details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_VRf8asUuYPG"
      },
      "outputs": [],
      "source": [
        "def get_s2_composite(roi, start, end):\n",
        "  s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')\n",
        "  s2sr = ee.ImageCollection('COPERNICUS/S2_SR_HARMONIZED')\n",
        "  s2c = s2c.filterBounds(roi.buffer(100, 1000)).filterDate(start, end)\n",
        "  s2sr = s2sr.filterBounds(roi.buffer(100, 1000)).filterDate(start, end)\n",
        "\n",
        "  def index_join(collection_a, collection_b, property_name):\n",
        "    joined = ee.ImageCollection(\n",
        "        ee.Join.saveFirst(property_name).apply(\n",
        "            primary=collection_a,\n",
        "            secondary=collection_b,\n",
        "            condition=ee.Filter.equals(\n",
        "                leftField='system:index',\n",
        "                rightField='system:index')))\n",
        "    return joined.map(\n",
        "        lambda image: image.addBands(ee.Image(image.get(property_name))))\n",
        "\n",
        "  def mask_image(image):\n",
        "    prob = image.select('probability')\n",
        "    return image.select('B.*').divide(10000).updateMask(prob.lt(50))\n",
        "\n",
        "  with_cloud_probability = index_join(s2sr, s2c, 'cloud_probability')\n",
        "  masked = ee.ImageCollection(with_cloud_probability.map(mask_image))\n",
        "  return masked.select(S2_BANDS).median().float().unmask(0)\n",
        "\n",
        "image = get_s2_composite(REGION, START, END)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k_kSU6fa_Tzg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 771
        },
        "outputId": "25c89023-a550-4c3d-970d-fd88566067b4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<folium.folium.Map at 0x7ad64bff12d0>"
            ],
            "text/html": [
              "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><span style=\"color:#565656\">Make this Notebook Trusted to load map: File -> Trust Notebook</span><iframe srcdoc=\"&lt;!DOCTYPE html&gt;\n",
              "&lt;html&gt;\n",
              "&lt;head&gt;\n",
              "    \n",
              "    &lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;\n",
              "    \n",
              "        &lt;script&gt;\n",
              "            L_NO_TOUCH = false;\n",
              "            L_DISABLE_3D = false;\n",
              "        &lt;/script&gt;\n",
              "    \n",
              "    &lt;style&gt;html, body {width: 100%;height: 100%;margin: 0;padding: 0;}&lt;/style&gt;\n",
              "    &lt;style&gt;#map {position:absolute;top:0;bottom:0;right:0;left:0;}&lt;/style&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://code.jquery.com/jquery-1.12.4.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.0/css/all.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/gh/python-visualization/folium/folium/templates/leaflet.awesome.rotate.min.css&quot;/&gt;\n",
              "    \n",
              "            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,\n",
              "                initial-scale=1.0, maximum-scale=1.0, user-scalable=no&quot; /&gt;\n",
              "            &lt;style&gt;\n",
              "                #map_bdb6673022197a65fb4a3ded51086006 {\n",
              "                    position: relative;\n",
              "                    width: 100.0%;\n",
              "                    height: 100.0%;\n",
              "                    left: 0.0%;\n",
              "                    top: 0.0%;\n",
              "                }\n",
              "                .leaflet-container { font-size: 1rem; }\n",
              "            &lt;/style&gt;\n",
              "        \n",
              "&lt;/head&gt;\n",
              "&lt;body&gt;\n",
              "    \n",
              "    \n",
              "            &lt;div class=&quot;folium-map&quot; id=&quot;map_bdb6673022197a65fb4a3ded51086006&quot; &gt;&lt;/div&gt;\n",
              "        \n",
              "&lt;/body&gt;\n",
              "&lt;script&gt;\n",
              "    \n",
              "    \n",
              "            var map_bdb6673022197a65fb4a3ded51086006 = L.map(\n",
              "                &quot;map_bdb6673022197a65fb4a3ded51086006&quot;,\n",
              "                {\n",
              "                    center: [6.074769, -1.981041],\n",
              "                    crs: L.CRS.EPSG3857,\n",
              "                    zoom: 8,\n",
              "                    zoomControl: true,\n",
              "                    preferCanvas: false,\n",
              "                }\n",
              "            );\n",
              "\n",
              "            \n",
              "\n",
              "        \n",
              "    \n",
              "            var tile_layer_690b8b9ed3f0181a876eb83d69c6ca7b = L.tileLayer(\n",
              "                &quot;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Data by \\u0026copy; \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://openstreetmap.org\\&quot;\\u003eOpenStreetMap\\u003c/a\\u003e, under \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://www.openstreetmap.org/copyright\\&quot;\\u003eODbL\\u003c/a\\u003e.&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_bdb6673022197a65fb4a3ded51086006);\n",
              "        \n",
              "    \n",
              "            var tile_layer_12e24476e9a7b8c73630fe0ad384c4a3 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/6f504a485aeb859130bff75024884c6a-aa7fd10ef3caaedb19fd434d1b3b64e7/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_bdb6673022197a65fb4a3ded51086006);\n",
              "        \n",
              "    \n",
              "            var tile_layer_2e33a1283138814283553878cbd2bffe = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/6ffe7d881b677840529222dc533b8999-18645adedd93e1def063a40f22e5d4a8/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_bdb6673022197a65fb4a3ded51086006);\n",
              "        \n",
              "    \n",
              "            var layer_control_1cee9eac1068a9a92fd581ef81ba9570 = {\n",
              "                base_layers : {\n",
              "                    &quot;openstreetmap&quot; : tile_layer_690b8b9ed3f0181a876eb83d69c6ca7b,\n",
              "                },\n",
              "                overlays :  {\n",
              "                    &quot;image&quot; : tile_layer_12e24476e9a7b8c73630fe0ad384c4a3,\n",
              "                    &quot;REGION&quot; : tile_layer_2e33a1283138814283553878cbd2bffe,\n",
              "                },\n",
              "            };\n",
              "            L.control.layers(\n",
              "                layer_control_1cee9eac1068a9a92fd581ef81ba9570.base_layers,\n",
              "                layer_control_1cee9eac1068a9a92fd581ef81ba9570.overlays,\n",
              "                {&quot;autoZIndex&quot;: true, &quot;collapsed&quot;: true, &quot;position&quot;: &quot;topright&quot;}\n",
              "            ).addTo(map_bdb6673022197a65fb4a3ded51086006);\n",
              "        \n",
              "&lt;/script&gt;\n",
              "&lt;/html&gt;\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "vis_params = {\n",
        "  'min': 0,\n",
        "  'max': 0.3,\n",
        "  'bands': ['B4', 'B3', 'B2'],\n",
        "}\n",
        "\n",
        "\n",
        "map = folium.Map(location=[COORDS[1], COORDS[0]], zoom_start=8)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=image.getMapId(vis_params)[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"image\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=roi_outline[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"REGION\",\n",
        "  ).add_to(map)\n",
        "\n",
        "map.add_child(folium.LayerControl())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h29GLQlYvAVL"
      },
      "source": [
        "# Generate training data\n",
        "\n",
        "There are a number of ways to generate training data in Earth Engine.  This notebook demonstrates using `image.sample` method to generate training data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l18gUBsgvlBn"
      },
      "source": [
        "Divide data into training (60%), validation (20%) and testing (20%) datasets. Also get the distribution of the galamsey and non-galamsey points in each of the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ddAvYu6AzLtP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aad20aed-bdae-4d22-88c8-b52b79f4cf00"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'geometry': {'coordinates': [-2.0106086649987467, 5.854679287445995],\n",
            "              'type': 'Point'},\n",
            " 'id': '1_1_00000000000000000000',\n",
            " 'properties': {'mining': 1, 'random': 0.4438691942788049},\n",
            " 'type': 'Feature'}\n",
            "('Sample size:', 3132)\n",
            "('Training sample size:', 1901)\n",
            "('Validation sample size:', 589)\n",
            "('Test sample size:', 589)\n",
            "('training_sample_locations', {'0': 629, '1': 1272})\n",
            "('validation_sample_locations', {'0': 189, '1': 400})\n",
            "('test_sample_locations', {'0': 194, '1': 448})\n"
          ]
        }
      ],
      "source": [
        "all_points = all_points.randomColumn(\"random\", 100)\n",
        "\n",
        "pprint(all_points.first().getInfo())\n",
        "\n",
        "pprint((\"Sample size:\", all_points.size().getInfo()))\n",
        "\n",
        "training_sample_locations = all_points.filter(ee.Filter.gt(\"random\", VALIDATION_RATIO + TEST_RATIO)) # > 0.4\n",
        "training_sample_size = training_sample_locations.size().getInfo()\n",
        "pprint((\"Training sample size:\", training_sample_size))\n",
        "\n",
        "validation_sample_locations = all_points.filter(ee.Filter.lte(\"random\", VALIDATION_RATIO)) # <= 0.2\n",
        "validation_sample_size = validation_sample_locations.size().getInfo()\n",
        "pprint((\"Validation sample size:\", validation_sample_size))\n",
        "\n",
        "test_sample_locations = all_points.filter(ee.Filter.And(ee.Filter.gt(\"random\", VALIDATION_RATIO),\n",
        "                                                        ee.Filter.lte(\"random\", VALIDATION_RATIO + TEST_RATIO))) # > 0.2 and <= 0.4\n",
        "test_sample_size = validation_sample_locations.size().getInfo()\n",
        "pprint((\"Test sample size:\", test_sample_size))\n",
        "\n",
        "pprint((\"training_sample_locations\", training_sample_locations.aggregate_histogram(OUTPUT_BANDS[0]).getInfo()))\n",
        "pprint((\"validation_sample_locations\", validation_sample_locations.aggregate_histogram(OUTPUT_BANDS[0]).getInfo()))\n",
        "pprint((\"test_sample_locations\", test_sample_locations.aggregate_histogram(OUTPUT_BANDS[0]).getInfo()))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AQa8a-5W_utH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 771
        },
        "outputId": "fe537444-fbee-4576-8cc2-9d318363e81e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<folium.folium.Map at 0x7ad64bba0310>"
            ],
            "text/html": [
              "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><span style=\"color:#565656\">Make this Notebook Trusted to load map: File -> Trust Notebook</span><iframe srcdoc=\"&lt;!DOCTYPE html&gt;\n",
              "&lt;html&gt;\n",
              "&lt;head&gt;\n",
              "    \n",
              "    &lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;\n",
              "    \n",
              "        &lt;script&gt;\n",
              "            L_NO_TOUCH = false;\n",
              "            L_DISABLE_3D = false;\n",
              "        &lt;/script&gt;\n",
              "    \n",
              "    &lt;style&gt;html, body {width: 100%;height: 100%;margin: 0;padding: 0;}&lt;/style&gt;\n",
              "    &lt;style&gt;#map {position:absolute;top:0;bottom:0;right:0;left:0;}&lt;/style&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://code.jquery.com/jquery-1.12.4.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.0/css/all.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/gh/python-visualization/folium/folium/templates/leaflet.awesome.rotate.min.css&quot;/&gt;\n",
              "    \n",
              "            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,\n",
              "                initial-scale=1.0, maximum-scale=1.0, user-scalable=no&quot; /&gt;\n",
              "            &lt;style&gt;\n",
              "                #map_ab57eefa2bac6a08ad6c0a16b589f8ce {\n",
              "                    position: relative;\n",
              "                    width: 100.0%;\n",
              "                    height: 100.0%;\n",
              "                    left: 0.0%;\n",
              "                    top: 0.0%;\n",
              "                }\n",
              "                .leaflet-container { font-size: 1rem; }\n",
              "            &lt;/style&gt;\n",
              "        \n",
              "&lt;/head&gt;\n",
              "&lt;body&gt;\n",
              "    \n",
              "    \n",
              "            &lt;div class=&quot;folium-map&quot; id=&quot;map_ab57eefa2bac6a08ad6c0a16b589f8ce&quot; &gt;&lt;/div&gt;\n",
              "        \n",
              "&lt;/body&gt;\n",
              "&lt;script&gt;\n",
              "    \n",
              "    \n",
              "            var map_ab57eefa2bac6a08ad6c0a16b589f8ce = L.map(\n",
              "                &quot;map_ab57eefa2bac6a08ad6c0a16b589f8ce&quot;,\n",
              "                {\n",
              "                    center: [6.074769, -1.981041],\n",
              "                    crs: L.CRS.EPSG3857,\n",
              "                    zoom: 8,\n",
              "                    zoomControl: true,\n",
              "                    preferCanvas: false,\n",
              "                }\n",
              "            );\n",
              "\n",
              "            \n",
              "\n",
              "        \n",
              "    \n",
              "            var tile_layer_f87654bf3b41be2a17bef05d17e2ce87 = L.tileLayer(\n",
              "                &quot;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Data by \\u0026copy; \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://openstreetmap.org\\&quot;\\u003eOpenStreetMap\\u003c/a\\u003e, under \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://www.openstreetmap.org/copyright\\&quot;\\u003eODbL\\u003c/a\\u003e.&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "    \n",
              "            var tile_layer_cb8df9ec7574a3974a3b8e859a5aa9f8 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/0326c2c0f5a9929851aaf976de49477f-fc83adacd72e2fcc92053ac307f021cf/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "    \n",
              "            var tile_layer_51621f7321cc204451c0102ae1f31e0e = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/ded86cb2f9a32bbf2f6391297ba59da9-099f28124353dd2c493c326eeae56c5a/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "    \n",
              "            var tile_layer_4a77fdea96882024c0f6228ab5dfbe72 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/bfe4450684189a139eda6fe4e7290278-6e88318e91c05a9da14256a733279f59/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "    \n",
              "            var tile_layer_77cda7512aa882f4d4f1eb777f17230a = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/6ffe7d881b677840529222dc533b8999-18645adedd93e1def063a40f22e5d4a8/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "    \n",
              "            var layer_control_9492fca4b446eda5bbfe6f46dbe554de = {\n",
              "                base_layers : {\n",
              "                    &quot;openstreetmap&quot; : tile_layer_f87654bf3b41be2a17bef05d17e2ce87,\n",
              "                },\n",
              "                overlays :  {\n",
              "                    &quot;training_sample_locations&quot; : tile_layer_cb8df9ec7574a3974a3b8e859a5aa9f8,\n",
              "                    &quot;validation_sample_locations&quot; : tile_layer_51621f7321cc204451c0102ae1f31e0e,\n",
              "                    &quot;test_sample_locations&quot; : tile_layer_4a77fdea96882024c0f6228ab5dfbe72,\n",
              "                    &quot;REGION&quot; : tile_layer_77cda7512aa882f4d4f1eb777f17230a,\n",
              "                },\n",
              "            };\n",
              "            L.control.layers(\n",
              "                layer_control_9492fca4b446eda5bbfe6f46dbe554de.base_layers,\n",
              "                layer_control_9492fca4b446eda5bbfe6f46dbe554de.overlays,\n",
              "                {&quot;autoZIndex&quot;: true, &quot;collapsed&quot;: true, &quot;position&quot;: &quot;topright&quot;}\n",
              "            ).addTo(map_ab57eefa2bac6a08ad6c0a16b589f8ce);\n",
              "        \n",
              "&lt;/script&gt;\n",
              "&lt;/html&gt;\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "map = folium.Map(location=[COORDS[1], COORDS[0]], zoom_start=8)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=training_sample_locations.getMapId({'color': 'green'})[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"training_sample_locations\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=validation_sample_locations.getMapId({'color': 'red'})[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"validation_sample_locations\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=test_sample_locations.getMapId({'color': 'blue'})[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"test_sample_locations\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=roi_outline[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"REGION\",\n",
        "  ).add_to(map)\n",
        "\n",
        "map.add_child(folium.LayerControl())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Paint the label data into an image and add that as a band to the S2 Image"
      ],
      "metadata": {
        "id": "LvvJlgdAmTPO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bCPSKKvaDPa8"
      },
      "outputs": [],
      "source": [
        "label_img = ee.Image().byte().paint(featureCollection=all_points, color=OUTPUT_BANDS[0]).rename(OUTPUT_BANDS[0])\n",
        "image = image.addBands(label_img)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Export Image before sampling"
      ],
      "metadata": {
        "id": "R5ELGEvP4R-l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you try to sample without exporting you will run into Computational Time Out. This is because there are a lot of processing going on with the S2 data using cloud score etc. So better to export the image to avoid the Computational Time out issue."
      ],
      "metadata": {
        "id": "kNdx2kpFmZx7"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iCl8RipohN_4"
      },
      "outputs": [],
      "source": [
        "def export_image_to_asset(image: ee.Image, start_training: bool, **kwargs: dict) -> ee.batch.Task:\n",
        "    asset_id = kwargs.get(\"asset_id\", \"\")\n",
        "    print(f\"Exporting image to {asset_id}..\")\n",
        "\n",
        "    training_task = ee.batch.Export.image.toAsset(\n",
        "        image=image,\n",
        "        description=kwargs.get(\"description\", \"myExportImageTask\"),\n",
        "        assetId=asset_id,\n",
        "        region=kwargs.get(\"region\", None),\n",
        "        scale=kwargs.get(\"scale\", SCALE),\n",
        "        maxPixels=kwargs.get(\"max_pixels\", 1E13),\n",
        "    )\n",
        "    if start_training: training_task.start()\n",
        "    return training_task"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u5kniofjhLHd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24952621-b462-4152-fe12-8484c2be168d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Exporting image to projects/servir-wa/tf_training_aug_2023/s2_2019_demo..\n"
          ]
        }
      ],
      "source": [
        "# Change False to True if you're actually exporting\n",
        "export_image = False\n",
        "training_task = export_image_to_asset(image, export_image, **{\"asset_id\": \"projects/servir-wa/tf_training_aug_2023/s2_2019_demo\", \"region\": REGION})\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n7ctCmk9hRO-"
      },
      "outputs": [],
      "source": [
        "#@title Don't run unless you are exporting the image\n",
        "\n",
        "# Print all tasks.\n",
        "# print(ee.batch.Task.list())\n",
        "\n",
        "# Poll the training task until it's done.\n",
        "import time\n",
        "while training_task.active():\n",
        "  print('Polling for task (id: {}).'.format(training_task.id))\n",
        "  time.sleep(30)\n",
        "print('Done with training export.')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally assemble all the data for the sampling purpose. Here we are adding already exported Sentinel-2 and Sentinel-1 dataset."
      ],
      "metadata": {
        "id": "3ZsDy3IAmqhJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pUBJd_PvhTav",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ea4d09cc-0a85-44e5-8296-774403b8678b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['VV',\n",
            " 'VH',\n",
            " 's1_ratio',\n",
            " 's1_ndratio',\n",
            " 'B1',\n",
            " 'B2',\n",
            " 'B3',\n",
            " 'B4',\n",
            " 'B5',\n",
            " 'B6',\n",
            " 'B7',\n",
            " 'B8',\n",
            " 'B8A',\n",
            " 'B9',\n",
            " 'B11',\n",
            " 'B12',\n",
            " 'mining']\n"
          ]
        }
      ],
      "source": [
        "s2 = ee.Image(\"projects/servir-wa/tf_training_aug_2023/s2_2019_demo\")\n",
        "# s1 = ee.Image(\"projects/servir-wa/tf_training_aug_2023/s1_asc_2019_demo\")\n",
        "s1 = ee.Image(\"projects/servir-wa/tf_training_aug_2023/s1_asc_2019_demo_30_m\")\n",
        "s1 = s1.select(['VV', 'VH', 'ratio', 'ndratio'], S1_BANDS)\n",
        "image = s1.addBands(s2)\n",
        "pprint(image.bandNames().getInfo())"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sample_image(image: ee.Image, region: ee.FeatureCollection, **kwargs: dict) -> ee.FeatureCollection:\n",
        "    sample = image.sample(region=ee.FeatureCollection(region),\n",
        "                          scale=kwargs.get(\"seed\", SCALE),\n",
        "                          seed=kwargs.get(\"seed\", 100),\n",
        "                          geometries=kwargs.get(\"geometries\", False))\n",
        "    return sample"
      ],
      "metadata": {
        "id": "ddheOsJ-31Z-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "giuvS5Z15wdW"
      },
      "outputs": [],
      "source": [
        "training_samples = sample_image(image, training_sample_locations)\n",
        "validation_samples = sample_image(image, validation_sample_locations)\n",
        "test_samples = sample_image(image, test_sample_locations)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "owyYGeLNTPn4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "845c4126-3885-4559-8e82-5dfb3d17bffe"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'geometry': None,\n",
            " 'id': '0',\n",
            " 'properties': {'B1': 0.07715000212192535,\n",
            "                'B11': 0.2533000111579895,\n",
            "                'B12': 0.16979999840259552,\n",
            "                'B2': 0.08049999922513962,\n",
            "                'B3': 0.11225000023841858,\n",
            "                'B4': 0.12084999680519104,\n",
            "                'B5': 0.1732500046491623,\n",
            "                'B6': 0.243599995970726,\n",
            "                'B7': 0.26205000281333923,\n",
            "                'B8': 0.25804999470710754,\n",
            "                'B8A': 0.28334999084472656,\n",
            "                'B9': 0.30869999527931213,\n",
            "                'VH': -14.61683173873444,\n",
            "                'VV': -8.161584434856472,\n",
            "                'mining': 1,\n",
            "                's1_ndratio': -0.28339315844804536,\n",
            "                's1_ratio': 0.5583689120008384},\n",
            " 'type': 'Feature'}\n"
          ]
        }
      ],
      "source": [
        "# sanity check before exporting them\n",
        "example_samples = sample_image(image, ee.FeatureCollection(ee.Feature(training_sample_locations.first())), **{\"scale\": SCALE})\n",
        "pprint(example_samples.first().getInfo())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jaEZvYsgSvmq"
      },
      "outputs": [],
      "source": [
        "def export_collection_to_cloud_storage(collection: ee.FeatureCollection, start_training: bool, **kwargs: dict) -> ee.batch.Task:\n",
        "    description = kwargs.get(\"description\", \"myExportTableTask\")\n",
        "    bucket = kwargs.get(\"bucket\", \"myBucket\")\n",
        "    file_prefix = kwargs.get(\"file_prefix\") if kwargs.get(\"file_prefix\") is not None else description\n",
        "    print(f\"Exporting training data to gs://{bucket}/{file_prefix}..\")\n",
        "    training_task = ee.batch.Export.table.toCloudStorage(\n",
        "        collection=collection,\n",
        "        description=description,\n",
        "        fileNamePrefix=file_prefix,\n",
        "        bucket=bucket,\n",
        "        fileFormat=kwargs.get(\"file_format\", \"TFRecord\"),\n",
        "        selectors=kwargs.get(\"selectors\", collection.first().propertyNames().getInfo()),\n",
        "    )\n",
        "    if start_training: training_task.start()\n",
        "    return training_task"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kk6LnXXgSLCz"
      },
      "outputs": [],
      "source": [
        "# Names for output files.\n",
        "train_file_prefix = \"illegal_mining_training\"\n",
        "test_file_prefix = \"illegal_mining_testing\"\n",
        "validate_file_prefix = \"illegal_mining_validation\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, let's export the samples in TFRecord format for using with the DL."
      ],
      "metadata": {
        "id": "PCaERQpEm3bU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JEZDY9N8SoEz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "62aab824-9041-4ea0-9e09-40f843171fbb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Exporting training data to gs://wa-tf-training/illegal_mining_training..\n",
            "Exporting training data to gs://wa-tf-training/illegal_mining_testing..\n",
            "Exporting training data to gs://wa-tf-training/illegal_mining_validation..\n"
          ]
        }
      ],
      "source": [
        "# Change start_training=True if you're actually exporting\n",
        "start_training = False\n",
        "\n",
        "kwargs = { \"bucket\": BUCKET, \"selectors\": BANDS }\n",
        "training_task = export_collection_to_cloud_storage(training_samples, start_training=start_training, **{**kwargs, \"file_prefix\": train_file_prefix, \"description\": \"Training\"})\n",
        "testing_task = export_collection_to_cloud_storage(test_samples, start_training=start_training, **{**kwargs, \"file_prefix\": test_file_prefix, \"description\": \"Testing\"})\n",
        "validation_task = export_collection_to_cloud_storage(validation_samples, start_training=start_training, **{**kwargs, \"file_prefix\": validate_file_prefix, \"description\": \"Validation\"})\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QJDLNmmqdFI1"
      },
      "outputs": [],
      "source": [
        "#@title Don't run unless you are exporting the training data\n",
        "\n",
        "# print(ee.batch.Task.list())\n",
        "\n",
        "# Poll the training task until it's done.\n",
        "import time\n",
        "while training_task.active():\n",
        "  print('Polling for task (id: {}).'.format(training_task.id))\n",
        "  time.sleep(30)\n",
        "print('Done with training export.')\n",
        "\n",
        "while testing_task.active():\n",
        "  print('Polling for task (id: {}).'.format(testing_task.id))\n",
        "  time.sleep(30)\n",
        "print('Done with testing export.')\n",
        "\n",
        "\n",
        "while validation_task.active():\n",
        "  print('Polling for task (id: {}).'.format(validation_task.id))\n",
        "  time.sleep(30)\n",
        "print('Done with validation export.')\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xub4XayKiRTm"
      },
      "source": [
        "### Check existence of the exported files\n",
        "\n",
        "If you've seen the status of the export tasks change to `COMPLETED`, then check for the existince of the files in the output Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g-T1neRsiSli",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7fbc6f67-0f53-4817-d85c-f3b33dfe8ffd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found training file.\n",
            "Found testing file.\n",
            "Found validation file.\n"
          ]
        }
      ],
      "source": [
        "file_name_suffix = '.tfrecord.gz'\n",
        "train_file_path = 'gs://' + BUCKET + '/' + train_file_prefix + file_name_suffix\n",
        "test_file_path = 'gs://' + BUCKET + '/' + test_file_prefix + file_name_suffix\n",
        "validate_file_path = 'gs://' + BUCKET + '/' + validate_file_prefix + file_name_suffix\n",
        "\n",
        "print('Found training file.' if tf.io.gfile.exists(train_file_path) else 'No training file found.')\n",
        "print('Found testing file.' if tf.io.gfile.exists(test_file_path) else 'No testing file found.')\n",
        "print('Found validation file.' if tf.io.gfile.exists(validate_file_path) else 'No validation file found.')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define the structure of your data\n",
        "\n",
        "For parsing the exported TFRecord files, `FEATURES_DICT` is a mapping between feature names (recall that `BANDS` contains the band and label names) and `float32` [`tf.io.FixedLenFeature`](https://www.tensorflow.org/api_docs/python/tf/io/FixedLenFeature) objects.  This mapping is necessary for telling TensorFlow how to read data in a TFRecord file into tensors.  Specifically, **all numeric data exported from Earth Engine is exported as `float32`**.\n",
        "\n",
        "(Note: *features* in the TensorFlow context (i.e. [`tf.train.Feature`](https://www.tensorflow.org/api_docs/python/tf/train/Feature)) are not to be confused with Earth Engine features (i.e. [`ee.Feature`](https://developers.google.com/earth-engine/api_docs#eefeature)), where the former is a protocol message type for serialized data input to the model and the latter is a geometry-based geographic data structure.)"
      ],
      "metadata": {
        "id": "6lsbP-pB6vnB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "COLUMNS = [\n",
        "  tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
        "  for k in BANDS\n",
        "]\n",
        "\n",
        "FEATURES_DICT = dict(zip(BANDS, COLUMNS))"
      ],
      "metadata": {
        "id": "fkBzzaH7609h"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bE3NH0ijU8G"
      },
      "source": [
        "# Data preparation and pre-processing\n",
        "\n",
        "Read data from the TFRecord file into a `tf.data.Dataset`.  Pre-process the dataset to get it into a suitable format for input to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3lYYa7wgj4YZ"
      },
      "outputs": [],
      "source": [
        "def create_tfrecord_from_file(filename: str) -> tf.data.TFRecordDataset:\n",
        "    return tf.data.TFRecordDataset(filename, compression_type=\"GZIP\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Read into a `tf.data.Dataset`\n",
        "\n",
        "Here we are going to read a file in Cloud Storage into a `tf.data.Dataset`.  ([these TensorFlow docs](https://www.tensorflow.org/guide/data) explain more about reading data into a `Dataset`).  Check that you can read examples from the file.  The purpose here is to ensure that we can read from the file without an error.  The actual content is not necessarily human readable."
      ],
      "metadata": {
        "id": "oz23zlye6hmY"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_VuOTtHZjb5L"
      },
      "outputs": [],
      "source": [
        "train_dataset = tf.data.Dataset.list_files(train_file_path).interleave(create_tfrecord_from_file)\n",
        "test_dataset = tf.data.Dataset.list_files(test_file_path).interleave(create_tfrecord_from_file)\n",
        "validate_dataset = tf.data.Dataset.list_files(validate_file_path).interleave(create_tfrecord_from_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parse the dataset\n",
        "\n",
        "Now we need to make a parsing function for the data in the TFRecord files.  The data comes in flattened 2D arrays per record and we want to use the first part of the array for input to the model and the last element of the array as the class label.  The parsing function reads data from a serialized [`Example` proto](https://www.tensorflow.org/api_docs/python/tf/train/Example) into a dictionary in which the keys are the feature names and the values are the tensors storing the value of the features for that example.  ([These TensorFlow docs](https://www.tensorflow.org/tutorials/load_data/tfrecord) explain more about reading `Example` protos from TFRecord files)."
      ],
      "metadata": {
        "id": "dBFMFrqu7FVe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def create_tfrecord_from_file(filename: str) -> tf.data.TFRecordDataset:\n",
        "    return tf.data.TFRecordDataset(filename, compression_type=\"GZIP\")"
      ],
      "metadata": {
        "id": "vpzf9nww7IOl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def parse_tfrecord_dnn(example_proto: tf.Tensor) -> Tuple:\n",
        "    parsed_features = tf.io.parse_single_example(example_proto, FEATURES_DICT)\n",
        "    label = parsed_features.pop(OUTPUT_BANDS[0])\n",
        "    label = tf.cast(label, tf.int32)\n",
        "    return parsed_features, label"
      ],
      "metadata": {
        "id": "JIN4UvUw7JcQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def to_tuple_dnn(dataset: dict, label: tf.Tensor, depth: int = 1) -> Tuple:\n",
        "    return tf.transpose(list(dataset.values())), tf.one_hot(indices=label, depth=depth)"
      ],
      "metadata": {
        "id": "KlBXHhYu7Kpb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "parser = partial(parse_tfrecord_dnn)\n",
        "tupler = partial(to_tuple_dnn, depth=N_CLASSES)"
      ],
      "metadata": {
        "id": "cVU3w1Nx7LlA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fTDFfgSvm21S"
      },
      "outputs": [],
      "source": [
        "train_dataset = train_dataset.map(parser, num_parallel_calls=tf.data.AUTOTUNE)\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(iter(train_dataset).next())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kgwUDVya6XPm",
        "outputId": "06b8b6bd-d789-479a-e83b-7210513082da"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "({'B1': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.04895], dtype=float32)>,\n",
            "  'B11': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.02475], dtype=float32)>,\n",
            "  'B12': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.0203], dtype=float32)>,\n",
            "  'B2': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.0496], dtype=float32)>,\n",
            "  'B3': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.05795], dtype=float32)>,\n",
            "  'B4': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.04185], dtype=float32)>,\n",
            "  'B5': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.04425], dtype=float32)>,\n",
            "  'B6': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.0384], dtype=float32)>,\n",
            "  'B7': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.03915], dtype=float32)>,\n",
            "  'B8': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.03395], dtype=float32)>,\n",
            "  'B8A': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.03505], dtype=float32)>,\n",
            "  'B9': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.06845], dtype=float32)>,\n",
            "  'VH': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([-28.888039], dtype=float32)>,\n",
            "  'VV': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([-18.517916], dtype=float32)>,\n",
            "  's1_ndratio': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([-0.21875148], dtype=float32)>,\n",
            "  's1_ratio': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.64102364], dtype=float32)>},\n",
            " <tf.Tensor: shape=(1,), dtype=int32, numpy=array([0], dtype=int32)>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that each record of the parsed dataset contains a tuple.  The first element of the tuple is a dictionary with bands for keys and the numeric value of the bands for values.  The second element of the tuple is a class label."
      ],
      "metadata": {
        "id": "_rE3M_pk8l9Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create additional features\n",
        "\n",
        "Another thing we might want to do as part of the input process is to create new features, for example NDVI, a vegetation index computed from reflectance in two spectral bands.  Here are some helper functions for that."
      ],
      "metadata": {
        "id": "rFcAem_e8nBt"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kBKZIQG2n4sK"
      },
      "outputs": [],
      "source": [
        "def normalized_difference(a, b):\n",
        "    nd = (a - b) / (a + b)\n",
        "    nd_inf = (a - b) / (a + b + 0.000001)\n",
        "    return tf.where(tf.math.is_finite(nd), nd, nd_inf)\n",
        "\n",
        "def add_NDVI(features, label):\n",
        "    NIR = features[\"B5\"]\n",
        "    RED = features[\"B4\"]\n",
        "    features[\"NDVI\"] = normalized_difference(NIR, RED)\n",
        "    return features, label\n",
        "\n",
        "def add_EVI(features, label):\n",
        "    NIR = features[\"B5\"]\n",
        "    RED = features[\"B4\"]\n",
        "    BLUE = features[\"B2\"]\n",
        "    EVI = 2.5 * ((NIR - RED) / (NIR + 6 * RED - 7.5 * BLUE + 1))\n",
        "    features[\"EVI\"] = EVI\n",
        "    return features, label\n",
        "\n",
        "def add_MNDWI(features, label):\n",
        "    GREEN = features[\"B3\"]\n",
        "    NIR = features[\"B5\"]\n",
        "    MNDWI = normalized_difference(GREEN, NIR)\n",
        "    features[\"MNDWI\"] = MNDWI\n",
        "    return features, label\n",
        "\n",
        "def add_SAVI(features, label):\n",
        "    NIR = features[\"B5\"]\n",
        "    RED = features[\"B4\"]\n",
        "    SAVI = (NIR - RED) * (1 + 0.5)/(NIR + RED + 0.5)\n",
        "    features[\"SAVI\"] = SAVI\n",
        "    return features, label\n",
        "\n",
        "def add_IBI(features, label):\n",
        "    #Add Index-Based Built-Up Index (IBI)\n",
        "    RED = features[\"B4\"]\n",
        "    GREEN = features[\"B3\"]\n",
        "    SWIR1 = features[\"B11\"]\n",
        "    NIR = features[\"B5\"]\n",
        "\n",
        "    ibiA = (2 * SWIR1) / (SWIR1 + NIR)\n",
        "    ibiB = (NIR / (NIR + RED)) + (GREEN / (GREEN + SWIR1))\n",
        "    IBI = normalized_difference(ibiA, ibiB)\n",
        "    features[\"IBI\"] = IBI\n",
        "    return features, label\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-VUO2ygdnJDX"
      },
      "outputs": [],
      "source": [
        "train_dataset = train_dataset.map(add_NDVI)\n",
        "train_dataset = train_dataset.map(add_EVI)\n",
        "train_dataset = train_dataset.map(add_MNDWI)\n",
        "train_dataset = train_dataset.map(add_SAVI)\n",
        "train_dataset = train_dataset.map(add_IBI)\n",
        "\n",
        "\n",
        "train_dataset = train_dataset.map(tupler, num_parallel_calls=tf.data.AUTOTUNE)\n",
        "train_dataset = train_dataset.shuffle(512)\n",
        "train_dataset = train_dataset.batch(BATCH_SIZE)\n",
        "train_dataset = train_dataset.prefetch(buffer_size=tf.data.AUTOTUNE)\n",
        "train_dataset = train_dataset.repeat()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TcucVyGsoGhX"
      },
      "outputs": [],
      "source": [
        "validate_dataset = validate_dataset.map(parser, num_parallel_calls=tf.data.AUTOTUNE)\n",
        "validate_dataset = validate_dataset.map(add_NDVI)\n",
        "validate_dataset = validate_dataset.map(add_EVI)\n",
        "validate_dataset = validate_dataset.map(add_MNDWI)\n",
        "validate_dataset = validate_dataset.map(add_SAVI)\n",
        "validate_dataset = validate_dataset.map(add_IBI)\n",
        "\n",
        "validate_dataset = validate_dataset.map(tupler, num_parallel_calls=tf.data.AUTOTUNE)\n",
        "validate_dataset = validate_dataset.shuffle(512)\n",
        "validate_dataset = validate_dataset.batch(1)\n",
        "validate_dataset = validate_dataset.prefetch(buffer_size=tf.data.AUTOTUNE)\n",
        "validate_dataset = validate_dataset.repeat()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model setup\n",
        "\n",
        "The basic workflow for classification in TensorFlow is:\n",
        "\n",
        "1.  Create the model.\n",
        "2.  Train the model (i.e. `fit()`).\n",
        "3.  Use the trained model for inference (i.e. `predict()`).\n",
        "\n",
        "Here we'll create a `Sequential` neural network model using Keras.  This simple model is inspired by examples in:\n",
        "\n",
        "* [The TensorFlow Get Started tutorial](https://www.tensorflow.org/tutorials/)\n",
        "* [The TensorFlow Keras guide](https://www.tensorflow.org/guide/keras#build_a_simple_model)\n",
        "* [The Keras `Sequential` model examples](https://keras.io/getting-started/sequential-model-guide/#multilayer-perceptron-mlp-for-multi-class-softmax-classification)\n",
        "\n",
        "Note that the model used here is purely for demonstration purposes and hasn't gone through any performance tuning.\n",
        "\n",
        "We will look at both Sequential and Functional based approach to making models."
      ],
      "metadata": {
        "id": "gMPHHc3n_e7b"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.   Sequential Model with keras"
      ],
      "metadata": {
        "id": "IpKC01_L_pxD"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NHcXI3ehox-L"
      },
      "outputs": [],
      "source": [
        "# Define the layers in the model.\n",
        "model1 = tf.keras.models.Sequential([\n",
        "  tf.keras.layers.Dense(128, activation=\"relu\"),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(64, activation=\"relu\"),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(32, activation=\"relu\"),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  # 2 class = sigmoid, multi-class = softmax\n",
        "  tf.keras.layers.Dense(N_CLASSES, activation=\"sigmoid\")\n",
        "])\n",
        "\n",
        "# Compile the model with the specified loss function.\n",
        "model1.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LiVhO9TQpY93",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "21928210-b6a5-49bc-f3f9-ad666eaf985b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/25\n",
            "59/59 [==============================] - 11s 24ms/step - loss: 0.6726 - accuracy: 0.6483 - val_loss: 0.5854 - val_accuracy: 0.7778\n",
            "Epoch 2/25\n",
            "59/59 [==============================] - 1s 22ms/step - loss: 0.6355 - accuracy: 0.6752 - val_loss: 0.5319 - val_accuracy: 0.7778\n",
            "Epoch 3/25\n",
            "59/59 [==============================] - 1s 22ms/step - loss: 0.5979 - accuracy: 0.7250 - val_loss: 0.5737 - val_accuracy: 0.7778\n",
            "Epoch 4/25\n",
            "59/59 [==============================] - 2s 33ms/step - loss: 0.5410 - accuracy: 0.7533 - val_loss: 0.4579 - val_accuracy: 0.7778\n",
            "Epoch 5/25\n",
            "59/59 [==============================] - 2s 27ms/step - loss: 0.4713 - accuracy: 0.7988 - val_loss: 0.3266 - val_accuracy: 0.8333\n",
            "Epoch 6/25\n",
            "59/59 [==============================] - 2s 33ms/step - loss: 0.4107 - accuracy: 0.8283 - val_loss: 0.1976 - val_accuracy: 0.9444\n",
            "Epoch 7/25\n",
            "59/59 [==============================] - 1s 19ms/step - loss: 0.3779 - accuracy: 0.8459 - val_loss: 0.1966 - val_accuracy: 0.9444\n",
            "Epoch 8/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.3603 - accuracy: 0.8529 - val_loss: 0.2266 - val_accuracy: 0.8889\n",
            "Epoch 9/25\n",
            "59/59 [==============================] - 1s 18ms/step - loss: 0.3724 - accuracy: 0.8486 - val_loss: 0.1989 - val_accuracy: 0.9444\n",
            "Epoch 10/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.3242 - accuracy: 0.8700 - val_loss: 0.3798 - val_accuracy: 0.8333\n",
            "Epoch 11/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.3137 - accuracy: 0.8807 - val_loss: 0.0860 - val_accuracy: 1.0000\n",
            "Epoch 12/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.3154 - accuracy: 0.8711 - val_loss: 0.4335 - val_accuracy: 0.7778\n",
            "Epoch 13/25\n",
            "59/59 [==============================] - 2s 32ms/step - loss: 0.3017 - accuracy: 0.8759 - val_loss: 0.1328 - val_accuracy: 1.0000\n",
            "Epoch 14/25\n",
            "59/59 [==============================] - 2s 36ms/step - loss: 0.2922 - accuracy: 0.8764 - val_loss: 0.2328 - val_accuracy: 0.8889\n",
            "Epoch 15/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.3011 - accuracy: 0.8727 - val_loss: 0.2001 - val_accuracy: 0.9444\n",
            "Epoch 16/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2751 - accuracy: 0.8876 - val_loss: 0.2390 - val_accuracy: 0.8889\n",
            "Epoch 17/25\n",
            "59/59 [==============================] - 1s 19ms/step - loss: 0.2689 - accuracy: 0.8876 - val_loss: 0.2625 - val_accuracy: 0.8889\n",
            "Epoch 18/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2808 - accuracy: 0.8887 - val_loss: 0.2860 - val_accuracy: 0.8333\n",
            "Epoch 19/25\n",
            "59/59 [==============================] - 1s 19ms/step - loss: 0.2658 - accuracy: 0.9016 - val_loss: 0.2374 - val_accuracy: 0.8889\n",
            "Epoch 20/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2588 - accuracy: 0.8946 - val_loss: 0.2750 - val_accuracy: 0.9444\n",
            "Epoch 21/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2668 - accuracy: 0.8935 - val_loss: 0.1370 - val_accuracy: 0.9444\n",
            "Epoch 22/25\n",
            "59/59 [==============================] - 2s 32ms/step - loss: 0.2594 - accuracy: 0.8898 - val_loss: 0.1017 - val_accuracy: 0.9444\n",
            "Epoch 23/25\n",
            "59/59 [==============================] - 2s 42ms/step - loss: 0.2668 - accuracy: 0.8909 - val_loss: 0.0500 - val_accuracy: 1.0000\n",
            "Epoch 24/25\n",
            "59/59 [==============================] - 2s 34ms/step - loss: 0.2597 - accuracy: 0.8855 - val_loss: 0.2521 - val_accuracy: 0.8333\n",
            "Epoch 25/25\n",
            "59/59 [==============================] - 2s 28ms/step - loss: 0.2680 - accuracy: 0.8860 - val_loss: 0.1249 - val_accuracy: 0.9444\n"
          ]
        }
      ],
      "source": [
        "# Fit the model to the training data.\n",
        "history1 = model1.fit(\n",
        "            x=train_dataset,\n",
        "            epochs=EPOCHS,\n",
        "            steps_per_epoch=(training_sample_size // BATCH_SIZE),\n",
        "            validation_data=validate_dataset,\n",
        "            validation_steps=(validation_sample_size // BATCH_SIZE)\n",
        "        )"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model1.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BRhrRZwTAzAH",
        "outputId": "420dd648-8948-4e2c-e5ad-976e793226a3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 1, 128)            2816      \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 1, 128)            0         \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1, 64)             8256      \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 1, 64)             0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1, 32)             2080      \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 1, 32)             0         \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 1, 2)              66        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 13,218\n",
            "Trainable params: 13,218\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Functional Model with keras"
      ],
      "metadata": {
        "id": "VLL7oEusADN7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "inputs = keras.Input(shape=(None, len(INPUT_BANDS) + 5), name=\"input_layer\")\n",
        "y = keras.layers.Conv1D(128, 3, activation=\"relu\", padding=\"same\", name=\"conv1\")(inputs)\n",
        "y = keras.layers.MaxPooling1D(2, padding=\"same\")(y)\n",
        "y = keras.layers.Conv1D(64, 3, activation=\"relu\", padding=\"same\", name=\"conv2\")(y)\n",
        "y = keras.layers.MaxPooling1D(2, padding=\"same\")(y)\n",
        "y = keras.layers.Conv1D(len(INPUT_BANDS) + 5, 2, activation=\"relu\", padding=\"same\", name=\"conv4\")(y)\n",
        "\n",
        "all_inputs = keras.layers.concatenate([inputs, y])\n",
        "\n",
        "x = keras.layers.Dense(128, activation=\"relu\")(all_inputs)\n",
        "x = keras.layers.Dropout(0.2)(x)\n",
        "x = keras.layers.Dense(64, activation=\"relu\")(x)\n",
        "x = keras.layers.Dropout(0.2)(x)\n",
        "x = keras.layers.Dense(32, activation=\"relu\")(x)\n",
        "x = keras.layers.Dropout(0.2)(x)\n",
        "output = keras.layers.Dense(N_CLASSES, activation=\"sigmoid\")(x)\n",
        "\n",
        "model2 = keras.models.Model(inputs=inputs, outputs=output)\n",
        "\n",
        "# Compile the model with the specified loss function.\n",
        "model2.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "               loss='binary_crossentropy',\n",
        "               metrics=['accuracy'])\n"
      ],
      "metadata": {
        "id": "26OHvlb6AJ9m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model2.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zwm2-CqBA1kp",
        "outputId": "480ea324-30de-498b-c663-4de47846c3ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_2\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " input_layer (InputLayer)       [(None, None, 21)]   0           []                               \n",
            "                                                                                                  \n",
            " conv1 (Conv1D)                 (None, None, 128)    8192        ['input_layer[0][0]']            \n",
            "                                                                                                  \n",
            " max_pooling1d_4 (MaxPooling1D)  (None, None, 128)   0           ['conv1[0][0]']                  \n",
            "                                                                                                  \n",
            " conv2 (Conv1D)                 (None, None, 64)     24640       ['max_pooling1d_4[0][0]']        \n",
            "                                                                                                  \n",
            " max_pooling1d_5 (MaxPooling1D)  (None, None, 64)    0           ['conv2[0][0]']                  \n",
            "                                                                                                  \n",
            " conv4 (Conv1D)                 (None, None, 21)     2709        ['max_pooling1d_5[0][0]']        \n",
            "                                                                                                  \n",
            " concatenate_2 (Concatenate)    (None, None, 42)     0           ['input_layer[0][0]',            \n",
            "                                                                  'conv4[0][0]']                  \n",
            "                                                                                                  \n",
            " dense_14 (Dense)               (None, None, 128)    5504        ['concatenate_2[0][0]']          \n",
            "                                                                                                  \n",
            " dropout_11 (Dropout)           (None, None, 128)    0           ['dense_14[0][0]']               \n",
            "                                                                                                  \n",
            " dense_15 (Dense)               (None, None, 64)     8256        ['dropout_11[0][0]']             \n",
            "                                                                                                  \n",
            " dropout_12 (Dropout)           (None, None, 64)     0           ['dense_15[0][0]']               \n",
            "                                                                                                  \n",
            " dense_16 (Dense)               (None, None, 32)     2080        ['dropout_12[0][0]']             \n",
            "                                                                                                  \n",
            " dropout_13 (Dropout)           (None, None, 32)     0           ['dense_16[0][0]']               \n",
            "                                                                                                  \n",
            " dense_17 (Dense)               (None, None, 2)      66          ['dropout_13[0][0]']             \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 51,447\n",
            "Trainable params: 51,447\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the model to the training data.\n",
        "history2 = model2.fit(\n",
        "            x=train_dataset,\n",
        "            epochs=EPOCHS,\n",
        "            steps_per_epoch=(training_sample_size // BATCH_SIZE),\n",
        "            validation_data=validate_dataset,\n",
        "            validation_steps=(validation_sample_size // BATCH_SIZE)\n",
        "        )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zNnX0NYtBWvc",
        "outputId": "1fe7264a-bb8e-4f31-e297-ec0c37c8c766"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/25\n",
            "59/59 [==============================] - 13s 31ms/step - loss: 0.7147 - accuracy: 0.6298 - val_loss: 0.6391 - val_accuracy: 0.6667\n",
            "Epoch 2/25\n",
            "59/59 [==============================] - 2s 24ms/step - loss: 0.6520 - accuracy: 0.6768 - val_loss: 0.6339 - val_accuracy: 0.6667\n",
            "Epoch 3/25\n",
            "59/59 [==============================] - 1s 22ms/step - loss: 0.6116 - accuracy: 0.7159 - val_loss: 0.6631 - val_accuracy: 0.7778\n",
            "Epoch 4/25\n",
            "59/59 [==============================] - 3s 43ms/step - loss: 0.5224 - accuracy: 0.7822 - val_loss: 0.5511 - val_accuracy: 0.7222\n",
            "Epoch 5/25\n",
            "59/59 [==============================] - 1s 23ms/step - loss: 0.4045 - accuracy: 0.8432 - val_loss: 0.8022 - val_accuracy: 0.6111\n",
            "Epoch 6/25\n",
            "59/59 [==============================] - 1s 23ms/step - loss: 0.3785 - accuracy: 0.8502 - val_loss: 0.1110 - val_accuracy: 1.0000\n",
            "Epoch 7/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.3380 - accuracy: 0.8609 - val_loss: 0.0725 - val_accuracy: 1.0000\n",
            "Epoch 8/25\n",
            "59/59 [==============================] - 1s 23ms/step - loss: 0.3609 - accuracy: 0.8443 - val_loss: 0.3241 - val_accuracy: 0.8889\n",
            "Epoch 9/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.2980 - accuracy: 0.8769 - val_loss: 0.3957 - val_accuracy: 0.7778\n",
            "Epoch 10/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2957 - accuracy: 0.8882 - val_loss: 0.2511 - val_accuracy: 0.8889\n",
            "Epoch 11/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2840 - accuracy: 0.8866 - val_loss: 0.2698 - val_accuracy: 0.8333\n",
            "Epoch 12/25\n",
            "59/59 [==============================] - 2s 35ms/step - loss: 0.3236 - accuracy: 0.8684 - val_loss: 0.2524 - val_accuracy: 0.9444\n",
            "Epoch 13/25\n",
            "59/59 [==============================] - 2s 30ms/step - loss: 0.2762 - accuracy: 0.8882 - val_loss: 0.3844 - val_accuracy: 0.8333\n",
            "Epoch 14/25\n",
            "59/59 [==============================] - 2s 28ms/step - loss: 0.2883 - accuracy: 0.8850 - val_loss: 0.1984 - val_accuracy: 0.9444\n",
            "Epoch 15/25\n",
            "59/59 [==============================] - 2s 41ms/step - loss: 0.2575 - accuracy: 0.8887 - val_loss: 0.1853 - val_accuracy: 0.8889\n",
            "Epoch 16/25\n",
            "59/59 [==============================] - 2s 41ms/step - loss: 0.2818 - accuracy: 0.8850 - val_loss: 0.2529 - val_accuracy: 0.8889\n",
            "Epoch 17/25\n",
            "59/59 [==============================] - 2s 41ms/step - loss: 0.2726 - accuracy: 0.8871 - val_loss: 0.2316 - val_accuracy: 0.9444\n",
            "Epoch 18/25\n",
            "59/59 [==============================] - 2s 34ms/step - loss: 0.2577 - accuracy: 0.8876 - val_loss: 0.1154 - val_accuracy: 1.0000\n",
            "Epoch 19/25\n",
            "59/59 [==============================] - 2s 26ms/step - loss: 0.2483 - accuracy: 0.9005 - val_loss: 0.2087 - val_accuracy: 0.8889\n",
            "Epoch 20/25\n",
            "59/59 [==============================] - 1s 22ms/step - loss: 0.2547 - accuracy: 0.8925 - val_loss: 0.0853 - val_accuracy: 1.0000\n",
            "Epoch 21/25\n",
            "59/59 [==============================] - 1s 23ms/step - loss: 0.2628 - accuracy: 0.8871 - val_loss: 0.1350 - val_accuracy: 0.9444\n",
            "Epoch 22/25\n",
            "59/59 [==============================] - 1s 21ms/step - loss: 0.2365 - accuracy: 0.9026 - val_loss: 0.1943 - val_accuracy: 0.8889\n",
            "Epoch 23/25\n",
            "59/59 [==============================] - 1s 20ms/step - loss: 0.2468 - accuracy: 0.8941 - val_loss: 0.2590 - val_accuracy: 0.8333\n",
            "Epoch 24/25\n",
            "59/59 [==============================] - 2s 35ms/step - loss: 0.2549 - accuracy: 0.8876 - val_loss: 0.2155 - val_accuracy: 0.8889\n",
            "Epoch 25/25\n",
            "59/59 [==============================] - 4s 67ms/step - loss: 0.2493 - accuracy: 0.9010 - val_loss: 0.1347 - val_accuracy: 0.9444\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-XoOC-zRvhJ7"
      },
      "source": [
        "## Check model accuracy on the test set\n",
        "\n",
        "Now that we have a trained model, we can evaluate it using the test dataset.  To do that, read and prepare the test dataset in the same way as the training dataset.  Here we specify a batch size of 1 so that each example in the test set is used exactly once to compute model accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q2_hzf8GvglN"
      },
      "outputs": [],
      "source": [
        "test_dataset = test_dataset.map(parser, num_parallel_calls=tf.data.AUTOTUNE)\n",
        "test_dataset = test_dataset.map(add_NDVI)\n",
        "test_dataset = test_dataset.map(add_EVI)\n",
        "test_dataset = test_dataset.map(add_MNDWI)\n",
        "test_dataset = test_dataset.map(add_SAVI)\n",
        "test_dataset = test_dataset.map(add_IBI)\n",
        "\n",
        "test_dataset = test_dataset.map(tupler, num_parallel_calls=tf.data.AUTOTUNE)\n",
        "test_dataset = test_dataset.batch(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "45jrbUvIvtIj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "09497c15-0f5e-4e06-c914-782b423efb21"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "642/642 [==============================] - 7s 10ms/step - loss: 0.2569 - accuracy: 0.8863\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.25685566663742065, 0.8862928152084351]"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ],
      "source": [
        "model1.evaluate(test_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model2.evaluate(test_dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bNLBppFjBDql",
        "outputId": "3b1fdebc-40f1-4bf1-d655-1369c0d9b57e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "642/642 [==============================] - 4s 6ms/step - loss: 0.2922 - accuracy: 0.8551\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.29222628474235535, 0.855140209197998]"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Inferences"
      ],
      "metadata": {
        "id": "AiGSiz7RGcSc"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VlI5Z8J4uXfF"
      },
      "source": [
        "## 1.   Use the trained model to classify an image from Earth Engine\n",
        "\n",
        "\n",
        "\n",
        "Now it's time to classify the image that was exported from Earth Engine.  If the exported image is large, it will be split into multiple TFRecord files in its destination folder.  There will also be a JSON sidecar file called \"the mixer\" that describes the format and georeferencing of the image.  Here we will find the image files and the mixer file, getting some info out of the mixer that will be useful during model inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPOJwkdQuXhu"
      },
      "source": [
        "### Export the imagery\n",
        "\n",
        "You also need to export imagery using TFRecord format.  Specifically, export whatever imagery you want to be classified by the trained model into the output Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qVq94dDJ4y0n",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e742cb93-5733-4143-b9ee-5e0df25753d6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'WA_image'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 60
        }
      ],
      "source": [
        "IMAGE_FILE_PREFIX"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def export_image_to_cloud_storage(image: ee.Image, start_training: bool, region: ee.Geometry, **kwargs: dict) -> ee.batch.Task:\n",
        "\n",
        "    print(f\"Exporting image to BUCKET {BUCKET}..\")\n",
        "\n",
        "    # Specify patch and file dimensions.\n",
        "    image_export_options = {\n",
        "    \"patchDimensions\": [256, 256],\n",
        "    \"maxFileSize\": 104857600,\n",
        "    \"compressed\": True\n",
        "    }\n",
        "\n",
        "    training_task = ee.batch.Export.image.toCloudStorage(\n",
        "        image=image,\n",
        "        description=kwargs.get(\"description\", \"myExportImageTask\"),\n",
        "        fileNamePrefix=kwargs.get(\"file_name_prefix\", IMAGE_FILE_PREFIX),\n",
        "        bucket=BUCKET,\n",
        "        fileFormat=\"TFRecord\",\n",
        "        region=region.getInfo()['coordinates'],\n",
        "        scale=kwargs.get(\"scale\", SCALE),\n",
        "        maxPixels=kwargs.get(\"max_pixels\", 1E13),\n",
        "        formatOptions=image_export_options,\n",
        "    )\n",
        "    if start_training: training_task.start()\n",
        "    return training_task"
      ],
      "metadata": {
        "id": "sBarxkxqC3Co"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Getting a smaller area for prediction because this might take long to complete\n",
        "\n",
        "EXPORT_REGION = ee.Geometry.Polygon(\n",
        "        [[[-2.2499489720464894, 6.806096384151279],\n",
        "          [-2.2499489720464894, 5.675769064032216],\n",
        "          [-1.3435769017339894, 5.675769064032216],\n",
        "          [-1.3435769017339894, 6.806096384151279]]])\n",
        "\n"
      ],
      "metadata": {
        "id": "GcKqKjwMFBBU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "start_cloud_export = False\n",
        "export_task = export_image_to_cloud_storage(image, start_training=start_cloud_export, region=EXPORT_REGION)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JoCoydoPE3E6",
        "outputId": "afcb483d-17c0-4cd9-dca6-0b57340b3dae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Exporting image to BUCKET wa-tf-training..\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Don't run unless you are exporting the image\n",
        "\n",
        "# Print all tasks.\n",
        "# print(ee.batch.Task.list())\n",
        "\n",
        "# Poll the training task until it's done.\n",
        "import time\n",
        "while export_task.active():\n",
        "  print('Polling for task (id: {}).'.format(training_task.id))\n",
        "  time.sleep(30)\n",
        "print('Done with training export.')"
      ],
      "metadata": {
        "id": "h7Eyh78GFkLk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sd8mMoYFuXki"
      },
      "source": [
        "### Find the image files and JSON mixer file in Cloud Storage\n",
        "\n",
        "Use `gsutil` to locate the files of interest in the output Cloud Storage bucket.  Check to make sure your image export task finished before running the following.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oyZNd-wq0gLv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84ab2a9e-b04e-47f0-e2db-df8a22932c2e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['gs://wa-tf-training/WA_image00000.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00001.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00002.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00003.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00004.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00005.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00006.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00007.tfrecord.gz',\n",
            " 'gs://wa-tf-training/WA_image00008.tfrecord.gz']\n",
            "'gs://wa-tf-training/WA_imagemixer.json'\n"
          ]
        }
      ],
      "source": [
        "# Get a list of all the files in the output bucket.\n",
        "files_list = !gsutil ls 'gs://'{BUCKET}\n",
        "# Get only the files generated by the image export.\n",
        "exported_files_list = [s for s in files_list if IMAGE_FILE_PREFIX in s]\n",
        "\n",
        "# Get the list of image files and the JSON mixer file.\n",
        "image_files_list = []\n",
        "json_file = None\n",
        "for f in exported_files_list:\n",
        "  if f.endswith('.tfrecord.gz'):\n",
        "    image_files_list.append(f)\n",
        "  elif f.endswith('.json'):\n",
        "    json_file = f\n",
        "\n",
        "# Make sure the files are in the right order.\n",
        "image_files_list.sort()\n",
        "\n",
        "pprint(image_files_list)\n",
        "pprint(json_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tk5lMR-auXnN"
      },
      "source": [
        "### Read the JSON mixer file\n",
        "\n",
        "The mixer contains metadata and georeferencing information for the exported patches, each of which is in a different file.  Read the mixer to get some information needed for prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BYljsHsN0jUI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a5d467bc-4a94-485c-bb7c-32fecb93aa18"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'patchDimensions': [256, 256],\n",
            " 'patchesPerRow': 13,\n",
            " 'projection': {'affine': {'doubleMatrix': [0.0002694945852358564,\n",
            "                                            0.0,\n",
            "                                            -2.2500102921341663,\n",
            "                                            0.0,\n",
            "                                            -0.0002694945852358564,\n",
            "                                            6.806355244716791]},\n",
            "                'crs': 'EPSG:4326'},\n",
            " 'totalPatches': 208}\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "\n",
        "# Load the contents of the mixer file to a JSON object.\n",
        "json_text = !gsutil cat {json_file}\n",
        "# Get a single string w/ newlines from the IPython.utils.text.SList\n",
        "mixer = json.loads(json_text.nlstr)\n",
        "pprint(mixer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WIHHtj_luXpo"
      },
      "source": [
        "### Read the image files into a dataset\n",
        "\n",
        "You can feed the list of files directly to the `TFRecordDataset` constructor to make a combined dataset on which to perform inference.  The input needs to be preprocessed differently than the training and testing.  Mainly, this is because the pixels are written into records as patches, we need to read the patches in as one big tensor (one patch for each band), then flatten them into lots of little tensors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5aSxoB0I0q05"
      },
      "outputs": [],
      "source": [
        "# Get relevant info from the JSON mixer file.\n",
        "patch_width = mixer['patchDimensions'][0]\n",
        "patch_height = mixer['patchDimensions'][1]\n",
        "patches = mixer['totalPatches']\n",
        "patch_dimensions_flat = [patch_width * patch_height, 1]\n",
        "\n",
        "# Note that the tensors are in the shape of a patch, one patch for each band.\n",
        "image_columns = [\n",
        "  tf.io.FixedLenFeature(shape=patch_dimensions_flat, dtype=tf.float32)\n",
        "    for k in BANDS\n",
        "]\n",
        "\n",
        "# Parsing dictionary.\n",
        "image_features_dict = dict(zip(INPUT_BANDS, image_columns))\n",
        "\n",
        "# Note that you can make one dataset from many files by specifying a list.\n",
        "image_dataset = tf.data.TFRecordDataset(image_files_list, compression_type='GZIP')\n",
        "\n",
        "# Parsing function.\n",
        "def parse_image(example_proto):\n",
        "  return tf.io.parse_single_example(example_proto, image_features_dict)\n",
        "\n",
        "# Parse the data into tensors, one long tensor per patch.\n",
        "image_dataset = image_dataset.map(parse_image, num_parallel_calls=5)\n",
        "\n",
        "# Break our long tensors into many little ones.\n",
        "image_dataset = image_dataset.flat_map(\n",
        "  lambda features: tf.data.Dataset.from_tensor_slices(features)\n",
        ")\n",
        "\n",
        "# Add additional features (NDVI).\n",
        "image_dataset = image_dataset.map(\n",
        "  # Add NDVI to a feature that doesn't have a label.\n",
        "  lambda features: add_NDVI(features, None)[0]\n",
        ")\n",
        "\n",
        "image_dataset = image_dataset.map(\n",
        "  # Add NDVI to a feature that doesn't have a label.\n",
        "  lambda features: add_EVI(features, None)[0]\n",
        ")\n",
        "\n",
        "image_dataset = image_dataset.map(\n",
        "  # Add NDVI to a feature that doesn't have a label.\n",
        "  lambda features: add_MNDWI(features, None)[0]\n",
        ")\n",
        "\n",
        "image_dataset = image_dataset.map(\n",
        "  # Add NDVI to a feature that doesn't have a label.\n",
        "  lambda features: add_SAVI(features, None)[0]\n",
        ")\n",
        "\n",
        "image_dataset = image_dataset.map(\n",
        "  # Add NDVI to a feature that doesn't have a label.\n",
        "  lambda features: add_IBI(features, None)[0]\n",
        ")\n",
        "\n",
        "\n",
        "# Turn the dictionary in each record into a tuple without a label.\n",
        "image_dataset = image_dataset.map(\n",
        "  lambda data_dict: (tf.transpose(list(data_dict.values())), )\n",
        ")\n",
        "\n",
        "# Turn each patch into a batch.\n",
        "image_dataset = image_dataset.batch(patch_width * patch_height)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkXViLq5uXsS"
      },
      "source": [
        "### Generate predictions for the image pixels\n",
        "\n",
        "To get predictions in each pixel, run the image dataset through the trained model using `model.predict()`.  Print the first prediction to see that the output is a list of the three class probabilities for each pixel.  Running all predictions might take a while."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PqHIfNWd0ues",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1dd88310-46b3-4f2c-c398-7eaa65f22bd3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "208/208 [==============================] - 2889s 14s/step\n",
            "[[0.8816099 0.1042067]]\n"
          ]
        }
      ],
      "source": [
        "# Run prediction in batches, with as many steps as there are patches.\n",
        "predictions = model1.predict(image_dataset, steps=patches, verbose=1)\n",
        "\n",
        "# Note that the predictions come as a numpy array.  Check the first one.\n",
        "print(predictions[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K3N_gwwEIRz3",
        "outputId": "d2a6cdbb-6eb3-4ab9-f09d-949ce46e0277"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "13631488\n",
            "array([[0.67055327, 0.31307673]], dtype=float32)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, None)"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ],
      "source": [
        "pprint(len(predictions)), pprint(predictions[3045])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cr920RNyuXu_"
      },
      "source": [
        "### Write the predictions to a TFRecord file\n",
        "\n",
        "Now that there's a list of class probabilities in `predictions`, it's time to write them back into a file, optionally including a class label which is simply the index of the maximum probability.  We'll write directly from TensorFlow to a file in the output Cloud Storage bucket.\n",
        "\n",
        "Iterate over the list, compute class label and write the class and the probabilities in patches.  Specifically, we need to write the pixels into the file as patches in the same order they came out.  The records are written as serialized `tf.train.Example` protos.  This might take a while."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_2PYYSAb0z3v",
        "outputId": "b27dba73-74c2-42d4-e35b-634820b3ae9e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Done with patch 1 of 208 ...\n",
            "Done with patch 2 of 208 ...\n",
            "Done with patch 3 of 208 ...\n",
            "Done with patch 4 of 208 ...\n",
            "Done with patch 5 of 208 ...\n",
            "Done with patch 6 of 208 ...\n",
            "Done with patch 7 of 208 ...\n",
            "Done with patch 8 of 208 ...\n",
            "Done with patch 9 of 208 ...\n",
            "Done with patch 10 of 208 ...\n",
            "Done with patch 11 of 208 ...\n",
            "Done with patch 12 of 208 ...\n",
            "Done with patch 13 of 208 ...\n",
            "Done with patch 14 of 208 ...\n",
            "Done with patch 15 of 208 ...\n",
            "Done with patch 16 of 208 ...\n",
            "Done with patch 17 of 208 ...\n",
            "Done with patch 18 of 208 ...\n",
            "Done with patch 19 of 208 ...\n",
            "Done with patch 20 of 208 ...\n",
            "Done with patch 21 of 208 ...\n",
            "Done with patch 22 of 208 ...\n",
            "Done with patch 23 of 208 ...\n",
            "Done with patch 24 of 208 ...\n",
            "Done with patch 25 of 208 ...\n",
            "Done with patch 26 of 208 ...\n",
            "Done with patch 27 of 208 ...\n",
            "Done with patch 28 of 208 ...\n",
            "Done with patch 29 of 208 ...\n",
            "Done with patch 30 of 208 ...\n",
            "Done with patch 31 of 208 ...\n",
            "Done with patch 32 of 208 ...\n",
            "Done with patch 33 of 208 ...\n",
            "Done with patch 34 of 208 ...\n",
            "Done with patch 35 of 208 ...\n",
            "Done with patch 36 of 208 ...\n",
            "Done with patch 37 of 208 ...\n",
            "Done with patch 38 of 208 ...\n",
            "Done with patch 39 of 208 ...\n",
            "Done with patch 40 of 208 ...\n",
            "Done with patch 41 of 208 ...\n",
            "Done with patch 42 of 208 ...\n",
            "Done with patch 43 of 208 ...\n",
            "Done with patch 44 of 208 ...\n",
            "Done with patch 45 of 208 ...\n",
            "Done with patch 46 of 208 ...\n",
            "Done with patch 47 of 208 ...\n",
            "Done with patch 48 of 208 ...\n",
            "Done with patch 49 of 208 ...\n",
            "Done with patch 50 of 208 ...\n",
            "Done with patch 51 of 208 ...\n",
            "Done with patch 52 of 208 ...\n",
            "Done with patch 53 of 208 ...\n",
            "Done with patch 54 of 208 ...\n",
            "Done with patch 55 of 208 ...\n",
            "Done with patch 56 of 208 ...\n",
            "Done with patch 57 of 208 ...\n",
            "Done with patch 58 of 208 ...\n",
            "Done with patch 59 of 208 ...\n",
            "Done with patch 60 of 208 ...\n",
            "Done with patch 61 of 208 ...\n",
            "Done with patch 62 of 208 ...\n",
            "Done with patch 63 of 208 ...\n",
            "Done with patch 64 of 208 ...\n",
            "Done with patch 65 of 208 ...\n",
            "Done with patch 66 of 208 ...\n",
            "Done with patch 67 of 208 ...\n",
            "Done with patch 68 of 208 ...\n",
            "Done with patch 69 of 208 ...\n",
            "Done with patch 70 of 208 ...\n",
            "Done with patch 71 of 208 ...\n",
            "Done with patch 72 of 208 ...\n",
            "Done with patch 73 of 208 ...\n",
            "Done with patch 74 of 208 ...\n",
            "Done with patch 75 of 208 ...\n",
            "Done with patch 76 of 208 ...\n",
            "Done with patch 77 of 208 ...\n",
            "Done with patch 78 of 208 ...\n",
            "Done with patch 79 of 208 ...\n",
            "Done with patch 80 of 208 ...\n",
            "Done with patch 81 of 208 ...\n",
            "Done with patch 82 of 208 ...\n",
            "Done with patch 83 of 208 ...\n",
            "Done with patch 84 of 208 ...\n",
            "Done with patch 85 of 208 ...\n",
            "Done with patch 86 of 208 ...\n",
            "Done with patch 87 of 208 ...\n",
            "Done with patch 88 of 208 ...\n",
            "Done with patch 89 of 208 ...\n",
            "Done with patch 90 of 208 ...\n",
            "Done with patch 91 of 208 ...\n",
            "Done with patch 92 of 208 ...\n",
            "Done with patch 93 of 208 ...\n",
            "Done with patch 94 of 208 ...\n",
            "Done with patch 95 of 208 ...\n",
            "Done with patch 96 of 208 ...\n",
            "Done with patch 97 of 208 ...\n",
            "Done with patch 98 of 208 ...\n",
            "Done with patch 99 of 208 ...\n",
            "Done with patch 100 of 208 ...\n",
            "Done with patch 101 of 208 ...\n",
            "Done with patch 102 of 208 ...\n",
            "Done with patch 103 of 208 ...\n",
            "Done with patch 104 of 208 ...\n",
            "Done with patch 105 of 208 ...\n",
            "Done with patch 106 of 208 ...\n",
            "Done with patch 107 of 208 ...\n",
            "Done with patch 108 of 208 ...\n",
            "Done with patch 109 of 208 ...\n",
            "Done with patch 110 of 208 ...\n",
            "Done with patch 111 of 208 ...\n",
            "Done with patch 112 of 208 ...\n",
            "Done with patch 113 of 208 ...\n",
            "Done with patch 114 of 208 ...\n",
            "Done with patch 115 of 208 ...\n",
            "Done with patch 116 of 208 ...\n",
            "Done with patch 117 of 208 ...\n",
            "Done with patch 118 of 208 ...\n",
            "Done with patch 119 of 208 ...\n",
            "Done with patch 120 of 208 ...\n",
            "Done with patch 121 of 208 ...\n",
            "Done with patch 122 of 208 ...\n",
            "Done with patch 123 of 208 ...\n",
            "Done with patch 124 of 208 ...\n",
            "Done with patch 125 of 208 ...\n",
            "Done with patch 126 of 208 ...\n",
            "Done with patch 127 of 208 ...\n",
            "Done with patch 128 of 208 ...\n",
            "Done with patch 129 of 208 ...\n",
            "Done with patch 130 of 208 ...\n",
            "Done with patch 131 of 208 ...\n",
            "Done with patch 132 of 208 ...\n",
            "Done with patch 133 of 208 ...\n",
            "Done with patch 134 of 208 ...\n",
            "Done with patch 135 of 208 ...\n",
            "Done with patch 136 of 208 ...\n",
            "Done with patch 137 of 208 ...\n",
            "Done with patch 138 of 208 ...\n",
            "Done with patch 139 of 208 ...\n",
            "Done with patch 140 of 208 ...\n",
            "Done with patch 141 of 208 ...\n",
            "Done with patch 142 of 208 ...\n",
            "Done with patch 143 of 208 ...\n",
            "Done with patch 144 of 208 ...\n",
            "Done with patch 145 of 208 ...\n",
            "Done with patch 146 of 208 ...\n",
            "Done with patch 147 of 208 ...\n",
            "Done with patch 148 of 208 ...\n",
            "Done with patch 149 of 208 ...\n",
            "Done with patch 150 of 208 ...\n",
            "Done with patch 151 of 208 ...\n",
            "Done with patch 152 of 208 ...\n",
            "Done with patch 153 of 208 ...\n",
            "Done with patch 154 of 208 ...\n",
            "Done with patch 155 of 208 ...\n",
            "Done with patch 156 of 208 ...\n",
            "Done with patch 157 of 208 ...\n",
            "Done with patch 158 of 208 ...\n",
            "Done with patch 159 of 208 ...\n",
            "Done with patch 160 of 208 ...\n",
            "Done with patch 161 of 208 ...\n",
            "Done with patch 162 of 208 ...\n",
            "Done with patch 163 of 208 ...\n",
            "Done with patch 164 of 208 ...\n",
            "Done with patch 165 of 208 ...\n",
            "Done with patch 166 of 208 ...\n",
            "Done with patch 167 of 208 ...\n",
            "Done with patch 168 of 208 ...\n",
            "Done with patch 169 of 208 ...\n",
            "Done with patch 170 of 208 ...\n",
            "Done with patch 171 of 208 ...\n",
            "Done with patch 172 of 208 ...\n",
            "Done with patch 173 of 208 ...\n",
            "Done with patch 174 of 208 ...\n",
            "Done with patch 175 of 208 ...\n",
            "Done with patch 176 of 208 ...\n",
            "Done with patch 177 of 208 ...\n",
            "Done with patch 178 of 208 ...\n",
            "Done with patch 179 of 208 ...\n",
            "Done with patch 180 of 208 ...\n",
            "Done with patch 181 of 208 ...\n",
            "Done with patch 182 of 208 ...\n",
            "Done with patch 183 of 208 ...\n",
            "Done with patch 184 of 208 ...\n",
            "Done with patch 185 of 208 ...\n",
            "Done with patch 186 of 208 ...\n",
            "Done with patch 187 of 208 ...\n",
            "Done with patch 188 of 208 ...\n",
            "Done with patch 189 of 208 ...\n",
            "Done with patch 190 of 208 ...\n",
            "Done with patch 191 of 208 ...\n",
            "Done with patch 192 of 208 ...\n",
            "Done with patch 193 of 208 ...\n",
            "Done with patch 194 of 208 ...\n",
            "Done with patch 195 of 208 ...\n",
            "Done with patch 196 of 208 ...\n",
            "Done with patch 197 of 208 ...\n",
            "Done with patch 198 of 208 ...\n",
            "Done with patch 199 of 208 ...\n",
            "Done with patch 200 of 208 ...\n",
            "Done with patch 201 of 208 ...\n",
            "Done with patch 202 of 208 ...\n",
            "Done with patch 203 of 208 ...\n",
            "Done with patch 204 of 208 ...\n",
            "Done with patch 205 of 208 ...\n",
            "Done with patch 206 of 208 ...\n",
            "Done with patch 207 of 208 ...\n",
            "Done with patch 208 of 208 ...\n"
          ]
        }
      ],
      "source": [
        "# Instantiate the writer.\n",
        "writer = tf.io.TFRecordWriter(OUTPUT_IMAGE_FILE)\n",
        "\n",
        "# Every patch-worth of predictions we'll dump an example into the output\n",
        "# file with a single feature that holds our predictions. Since our predictions\n",
        "# are already in the order of the exported data, the patches we create here\n",
        "# will also be in the right order.\n",
        "patch = [[], [], []]\n",
        "cur_patch = 1\n",
        "for prediction in predictions:\n",
        "  patch[0].append(tf.argmax(prediction, 1).numpy()[0])\n",
        "  patch[1].append(prediction[0][0])\n",
        "  patch[2].append(prediction[0][1])\n",
        "  # Once we've seen a patches-worth of class_ids...\n",
        "  if (len(patch[0]) == patch_width * patch_height):\n",
        "    print(f\"Done with patch {cur_patch} of {patches} ...\")\n",
        "    # Create an example\n",
        "    example = tf.train.Example(\n",
        "      features=tf.train.Features(\n",
        "        feature={\n",
        "          \"prediction\": tf.train.Feature(\n",
        "              int64_list=tf.train.Int64List(\n",
        "                  value=patch[0])),\n",
        "          \"nonMiningProb\": tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=patch[1])),\n",
        "          \"miningProb\": tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=patch[2]))\n",
        "        }\n",
        "      )\n",
        "    )\n",
        "    # Write the example to the file and clear our patch array so it's ready for\n",
        "    # another batch of class ids\n",
        "    writer.write(example.SerializeToString())\n",
        "    patch = [[], [], []]\n",
        "    cur_patch += 1\n",
        "\n",
        "writer.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ubr0PHzBuXxX"
      },
      "source": [
        "### Upload the classifications to an Earth Engine asset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEs1gprquX0M"
      },
      "source": [
        "#### Verify the existence of the predictions file\n",
        "\n",
        "At this stage, there should be a predictions TFRecord file sitting in the output Cloud Storage bucket.  Use the `gsutil` command to verify that the predictions image (and associated mixer JSON) exist and have non-zero size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "drdPJFj11Aan",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "945dd998-5031-4f65-a0a2-d0cd718c4084"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 122705648  2023-08-15T19:25:16Z  gs://wa-tf-training/WA_Classified.TFRecord\n",
            "TOTAL: 1 objects, 122705648 bytes (117.02 MiB)\n"
          ]
        }
      ],
      "source": [
        "!gsutil ls -l {OUTPUT_IMAGE_FILE}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObxFS6JMuX20"
      },
      "source": [
        "### Upload the classified image to Earth Engine\n",
        "\n",
        "Upload the image to Earth Engine directly from the Cloud Storage bucket with the [`earthengine` command](https://developers.google.com/earth-engine/command_line#upload).  Provide both the image TFRecord file and the JSON file as arguments to `earthengine upload`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KZwkrwbW1Eoa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "62c2017c-0d50-4320-a0d0-845367171544"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Uploading to projects/servir-wa/tf_training_aug_2023/tf_training_prediction_example\n"
          ]
        }
      ],
      "source": [
        "OUTPUT_ASSET_ID = \"projects/servir-wa/tf_training_aug_2023/tf_training_prediction_example\"\n",
        "print('Uploading to ' + OUTPUT_ASSET_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DM9ASE9S1Gx0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d7f67189-3c01-477d-c0cb-1a2ac3e5b8b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Started upload task with ID: 6O5HX5D6HJWA4OC3VIOO7H3J\n"
          ]
        }
      ],
      "source": [
        "# Start the upload.\n",
        "!earthengine upload image --asset_id={OUTPUT_ASSET_ID} --pyramiding_policy=mode {OUTPUT_IMAGE_FILE} {json_file}"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the upload doesn't work, you would need to authenticate again with earthengine using `ee.Authenticate()`"
      ],
      "metadata": {
        "id": "kFBA31QWqR7e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ee.Authenticate()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8LLtIHmUqaaS",
        "outputId": "ece05ae4-803b-41b9-d03d-5a22d505e76c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions. If the web browser does not start automatically, please manually browse the URL below.\n",
            "\n",
            "    https://code.earthengine.google.com/client-auth?scopes=https%3A//www.googleapis.com/auth/earthengine%20https%3A//www.googleapis.com/auth/devstorage.full_control&request_id=H-Vz0BkJ29dwEiqDHm7rJ40rvXBLTrPv4pNzIB80j_w&tc=Lr6Oaf-fstI2udOGnjFJSA2tCk3ykToxVn4I5fV6V48&cc=te9e6iHe1OjeuO8QqdXK24IYLDFkcJD1OoHlcJqq46o\n",
            "\n",
            "The authorization workflow will generate a code, which you should paste in the box below.\n",
            "Enter verification code: 4/1Adeu5BU2RIBeAlurwEBu5hY89Se05ozGmSWeElJZ1noxnuAcnbkGRUzQAtM\n",
            "\n",
            "Successfully saved authorization token.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhL7cZnsuX5V"
      },
      "source": [
        "### Check the status of the asset ingestion\n",
        "\n",
        "You can also use the Earth Engine API to check the status of your asset upload.  It might take a while.  The upload of the image is an asset ingestion task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iQ6O1sMh1MQ1"
      },
      "outputs": [],
      "source": [
        "ee.batch.Task.list()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kWbfTJwNuX77"
      },
      "source": [
        "### View the ingested asset\n",
        "\n",
        "Display the vector of class probabilities as an RGB image with colors corresponding to the probability of bare, vegetation, water in a pixel.  Also display the winning class using the same color palette."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PSD-9kMe1NyX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 771
        },
        "outputId": "c868ed25-f71b-46f5-ded7-a81a750f9aa1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<folium.folium.Map at 0x7ad5b82d2c50>"
            ],
            "text/html": [
              "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><span style=\"color:#565656\">Make this Notebook Trusted to load map: File -> Trust Notebook</span><iframe srcdoc=\"&lt;!DOCTYPE html&gt;\n",
              "&lt;html&gt;\n",
              "&lt;head&gt;\n",
              "    \n",
              "    &lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;\n",
              "    \n",
              "        &lt;script&gt;\n",
              "            L_NO_TOUCH = false;\n",
              "            L_DISABLE_3D = false;\n",
              "        &lt;/script&gt;\n",
              "    \n",
              "    &lt;style&gt;html, body {width: 100%;height: 100%;margin: 0;padding: 0;}&lt;/style&gt;\n",
              "    &lt;style&gt;#map {position:absolute;top:0;bottom:0;right:0;left:0;}&lt;/style&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://code.jquery.com/jquery-1.12.4.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.0/css/all.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/gh/python-visualization/folium/folium/templates/leaflet.awesome.rotate.min.css&quot;/&gt;\n",
              "    \n",
              "            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,\n",
              "                initial-scale=1.0, maximum-scale=1.0, user-scalable=no&quot; /&gt;\n",
              "            &lt;style&gt;\n",
              "                #map_f8d0656ec97d28a1fef19eb452feba92 {\n",
              "                    position: relative;\n",
              "                    width: 100.0%;\n",
              "                    height: 100.0%;\n",
              "                    left: 0.0%;\n",
              "                    top: 0.0%;\n",
              "                }\n",
              "                .leaflet-container { font-size: 1rem; }\n",
              "            &lt;/style&gt;\n",
              "        \n",
              "&lt;/head&gt;\n",
              "&lt;body&gt;\n",
              "    \n",
              "    \n",
              "            &lt;div class=&quot;folium-map&quot; id=&quot;map_f8d0656ec97d28a1fef19eb452feba92&quot; &gt;&lt;/div&gt;\n",
              "        \n",
              "&lt;/body&gt;\n",
              "&lt;script&gt;\n",
              "    \n",
              "    \n",
              "            var map_f8d0656ec97d28a1fef19eb452feba92 = L.map(\n",
              "                &quot;map_f8d0656ec97d28a1fef19eb452feba92&quot;,\n",
              "                {\n",
              "                    center: [6.074769, -1.981041],\n",
              "                    crs: L.CRS.EPSG3857,\n",
              "                    zoom: 10,\n",
              "                    zoomControl: true,\n",
              "                    preferCanvas: false,\n",
              "                }\n",
              "            );\n",
              "\n",
              "            \n",
              "\n",
              "        \n",
              "    \n",
              "            var tile_layer_fd56a5880213cc4ea86d3c908890bc0e = L.tileLayer(\n",
              "                &quot;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Data by \\u0026copy; \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://openstreetmap.org\\&quot;\\u003eOpenStreetMap\\u003c/a\\u003e, under \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://www.openstreetmap.org/copyright\\&quot;\\u003eODbL\\u003c/a\\u003e.&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_f8d0656ec97d28a1fef19eb452feba92);\n",
              "        \n",
              "    \n",
              "            var tile_layer_0b4e5a5201eeef41e843a755fa568c19 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/dfc99c5a468dc05b0e8c4b7bd52e51ce-3fe476d2090d54f57d61244253030310/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Map Data \\u0026copy; \\u003ca href=\\u0027https://earthengine.google.com/\\u0027\\u003eGoogle Earth Engine\\u003c/a\\u003e&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_f8d0656ec97d28a1fef19eb452feba92);\n",
              "        \n",
              "    \n",
              "            var tile_layer_8b09406eec5ca445d8088978bf1b9bcb = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/0fe24b8c91a536578283d18a285ca869-cf495c85eca75774398c0fc7b3ea7aba/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Map Data \\u0026copy; \\u003ca href=\\u0027https://earthengine.google.com/\\u0027\\u003eGoogle Earth Engine\\u003c/a\\u003e&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_f8d0656ec97d28a1fef19eb452feba92);\n",
              "        \n",
              "    \n",
              "            var tile_layer_fa13f49ea9fe294fbe35d38f062357f6 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/8be8ff46baebded0ce215db6e63f944a-03889605b798b2f225db299d747ed038/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Map Data \\u0026copy; \\u003ca href=\\u0027https://earthengine.google.com/\\u0027\\u003eGoogle Earth Engine\\u003c/a\\u003e&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_f8d0656ec97d28a1fef19eb452feba92);\n",
              "        \n",
              "    \n",
              "            var layer_control_8628a762ded7754d86c2444e516e31a0 = {\n",
              "                base_layers : {\n",
              "                    &quot;openstreetmap&quot; : tile_layer_fd56a5880213cc4ea86d3c908890bc0e,\n",
              "                },\n",
              "                overlays :  {\n",
              "                    &quot;Prediction&quot; : tile_layer_0b4e5a5201eeef41e843a755fa568c19,\n",
              "                    &quot;Mining Probability&quot; : tile_layer_8b09406eec5ca445d8088978bf1b9bcb,\n",
              "                    &quot;Non-Mining Probability&quot; : tile_layer_fa13f49ea9fe294fbe35d38f062357f6,\n",
              "                },\n",
              "            };\n",
              "            L.control.layers(\n",
              "                layer_control_8628a762ded7754d86c2444e516e31a0.base_layers,\n",
              "                layer_control_8628a762ded7754d86c2444e516e31a0.overlays,\n",
              "                {&quot;autoZIndex&quot;: true, &quot;collapsed&quot;: true, &quot;position&quot;: &quot;topright&quot;}\n",
              "            ).addTo(map_f8d0656ec97d28a1fef19eb452feba92);\n",
              "        \n",
              "&lt;/script&gt;\n",
              "&lt;/html&gt;\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ],
      "source": [
        "predictions_image = ee.Image(OUTPUT_ASSET_ID)\n",
        "\n",
        "prediction_vis = {\n",
        "  \"bands\": \"prediction\",\n",
        "  \"min\": 0,\n",
        "  \"max\": 1,\n",
        "}\n",
        "mining_probability_vis = {\"bands\": [\"miningProb\"], \"min\": 0, \"max\": 1}\n",
        "non_mining_probability_vis = {\"bands\": [\"nonMiningProb\"], \"min\": 0, \"max\": 1}\n",
        "\n",
        "prediction_map_id = predictions_image.getMapId(prediction_vis)\n",
        "mining_probability_map_id = predictions_image.getMapId(mining_probability_vis)\n",
        "non_mining_probability_map_id = predictions_image.getMapId(non_mining_probability_vis)\n",
        "\n",
        "map = folium.Map(location=[COORDS[1], COORDS[0]])\n",
        "\n",
        "folium.TileLayer(\n",
        "  tiles=prediction_map_id[\"tile_fetcher\"].url_format,\n",
        "  attr=\"Map Data &copy; <a href='https://earthengine.google.com/'>Google Earth Engine</a>\",\n",
        "  overlay=True,\n",
        "  name=\"Prediction\",\n",
        ").add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "  tiles=mining_probability_map_id[\"tile_fetcher\"].url_format,\n",
        "  attr=\"Map Data &copy; <a href='https://earthengine.google.com/'>Google Earth Engine</a>\",\n",
        "  overlay=True,\n",
        "  name=\"Mining Probability\",\n",
        ").add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "  tiles=non_mining_probability_map_id[\"tile_fetcher\"].url_format,\n",
        "  attr=\"Map Data &copy; <a href='https://earthengine.google.com/'>Google Earth Engine</a>\",\n",
        "  overlay=True,\n",
        "  name=\"Non-Mining Probability\",\n",
        ").add_to(map)\n",
        "\n",
        "map.add_child(folium.LayerControl())\n",
        "map"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The result does not look terrible. But this can always be improved with more training data or adding new features.\n",
        "You can see the difference in using only Sentinel-2 vs adding Sentinel-1 in this [script](https://code.earthengine.google.com/bbf335ba829569d6ac8f0f3c5cedc063)."
      ],
      "metadata": {
        "id": "P7ntj9LXwa0B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Using the AI Platform"
      ],
      "metadata": {
        "id": "eaFDqRtbG5-l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that the notebook VM is sometimes not heavy-duty enough to get through a whole training job, especially if you have a large buffer size or a large number of epochs. You can still use this notebook for training, but may need to set up an alternative VM (learn more) for production use. Alternatively, you can package your code for running large training jobs on Google's AI Platform as described here. The following code loads a pre-trained model, which you can use for predictions right away."
      ],
      "metadata": {
        "id": "wwiU1riXWagm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MODEL_DIR = \"gs://\" + BUCKET + \"/model\"\n",
        "print(f\"Saving model to {MODEL_DIR}\")\n",
        "model1.save(MODEL_DIR)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "04vJf_k1WcDt",
        "outputId": "5a213262-9f76-4659-ca96-49c856be9511"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving model to gs://wa-tf-training/model\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hb9Xg9bY8ze6"
      },
      "source": [
        "### Prepare the model for making predictions in Earth Engine\n",
        "\n",
        "Before we can use the model in Earth Engine, it needs to be hosted by AI Platform.  But before we can host the model on AI Platform we need to *EEify* (a new word!) it.  The EEification process merely appends some extra operations to the input and outputs of the model in order to accomdate the interchange format between pixels from Earth Engine (float32) and inputs to AI Platform (base64).  (See [this doc](https://cloud.google.com/ml-engine/docs/online-predict#binary_data_in_prediction_input) for details.)  \n",
        "\n",
        "## `earthengine model prepare`\n",
        "The EEification process is handled for you using the Earth Engine command `earthengine model prepare`.  To use that command, we need to specify the input and output model directories and the name of the input and output nodes in the TensorFlow computation graph.  We can do all that programmatically:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nHTYyyOA-_cR"
      },
      "outputs": [],
      "source": [
        "# ee.Authenticate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uNV43HpU8z3O",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d48fbc6e-de73-4144-83b0-062d2a4a6f6e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved project id\n",
            "Warning: TensorFlow Addons not found. Models that use non-standard ops may not work.\n",
            "Success: model at 'gs://wa-tf-training/eeified' is ready to be hosted in AI Platform.\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.python.tools import saved_model_utils\n",
        "\n",
        "meta_graph_def = saved_model_utils.get_meta_graph_def(MODEL_DIR, 'serve')\n",
        "inputs = meta_graph_def.signature_def['serving_default'].inputs\n",
        "outputs = meta_graph_def.signature_def['serving_default'].outputs\n",
        "\n",
        "# Just get the first thing(s) from the serving signature def.  i.e. this\n",
        "# model only has a single input and a single output.\n",
        "input_name = None\n",
        "for k,v in inputs.items():\n",
        "  input_name = v.name\n",
        "  break\n",
        "\n",
        "output_name = None\n",
        "for k,v in outputs.items():\n",
        "  output_name = v.name\n",
        "  break\n",
        "\n",
        "# Make a dictionary that maps Earth Engine outputs and inputs to\n",
        "# AI Platform inputs and outputs, respectively.\n",
        "import json\n",
        "input_dict = \"'\" + json.dumps({input_name: \"array\"}) + \"'\"\n",
        "output_dict = \"'\" + json.dumps({output_name: OUTPUT_BANDS[0]}) + \"'\"\n",
        "\n",
        "# Put the EEified model next to the trained model directory.\n",
        "EEIFIED_DIR = 'gs://' + BUCKET + '/eeified'\n",
        "\n",
        "# You need to set the project before using the model prepare command.\n",
        "# if you get config not found; rerun the ee.Authenticate() (above cell)\n",
        "!earthengine set_project {PROJECT}\n",
        "!earthengine model prepare --source_dir {MODEL_DIR} --dest_dir {EEIFIED_DIR} --input {input_dict} --output {output_dict}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5455372S_SdF"
      },
      "source": [
        "### Perform inference using the trained model in Earth Engine\n",
        "\n",
        "Before it's possible to get predictions from the trained and EEified model, it needs to be deployed on AI Platform.  The first step is to create the model.  The second step is to create a version.  See [this guide](https://cloud.google.com/ml-engine/docs/tensorflow/deploying-models) for details.  Note that models and versions can be monitored from the [AI Platform models page](http://console.cloud.google.com/ai-platform/models) of the Cloud Console.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sf1G0L31856T",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "55196886-3c7f-418b-fdd4-0538e509d012"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating version: v1692129518\n",
            "Please specify a region:\n",
            "(For the global endpoint the region needs to be specified as 'global'.)\n",
            " [1] global\n",
            " [2] asia-east1\n",
            " [3] asia-northeast1\n",
            " [4] asia-southeast1\n",
            " [5] australia-southeast1\n",
            " [6] europe-west1\n",
            " [7] europe-west2\n",
            " [8] europe-west3\n",
            " [9] europe-west4\n",
            " [10] northamerica-northeast1\n",
            " [11] us-central1\n",
            " [12] us-east1\n",
            " [13] us-east4\n",
            " [14] us-west1\n",
            " [15] cancel\n",
            "Please enter your numeric choice:  11\n",
            "\n",
            "To make this the default region, run `gcloud config set ai_platform/region us-central1`.\n",
            "\n",
            "Using endpoint [https://us-central1-ml.googleapis.com/]\n",
            "Created ai platform model [projects/servir-ee/models/wa_tf_training_model].\n",
            "Please specify a region:\n",
            "(For the global endpoint the region needs to be specified as 'global'.)\n",
            " [1] global\n",
            " [2] asia-east1\n",
            " [3] asia-northeast1\n",
            " [4] asia-southeast1\n",
            " [5] australia-southeast1\n",
            " [6] europe-west1\n",
            " [7] europe-west2\n",
            " [8] europe-west3\n",
            " [9] europe-west4\n",
            " [10] northamerica-northeast1\n",
            " [11] us-central1\n",
            " [12] us-east1\n",
            " [13] us-east4\n",
            " [14] us-west1\n",
            " [15] cancel\n",
            "Please enter your numeric choice:  11\n",
            "\n",
            "To make this the default region, run `gcloud config set ai_platform/region us-central1`.\n",
            "\n",
            "Using endpoint [https://us-central1-ml.googleapis.com/]\n"
          ]
        }
      ],
      "source": [
        "MODEL_NAME = 'wa_tf_training_model'\n",
        "VERSION_NAME = 'v' + str(int(time.time()))\n",
        "print('Creating version: ' + VERSION_NAME)\n",
        "\n",
        "!gcloud ai-platform models create {MODEL_NAME} --project {PROJECT}\n",
        "!gcloud ai-platform versions create {VERSION_NAME} \\\n",
        "  --project {PROJECT} \\\n",
        "  --model {MODEL_NAME} \\\n",
        "  --origin {EEIFIED_DIR} \\\n",
        "  --runtime-version=2.8 \\\n",
        "  --framework \"TENSORFLOW\" \\\n",
        "  --python-version=3.7"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tXerLpdO_6Rp"
      },
      "source": [
        "There is now a trained model, prepared for serving to Earth Engine, hosted and versioned on AI Platform.  We can now connect Earth Engine directly to the trained model for inference.  You do that with the `ee.Model.fromAiPlatformPredictor` command.\n",
        "\n",
        "#### `ee.Model.fromAiPlatformPredictor`\n",
        "For this command to work, we need to know a lot about the model.  To connect to the model, you need to know the name and version.\n",
        "\n",
        "##### Inputs\n",
        "You need to be able to recreate the imagery on which it was trained in order to perform inference.  Specifically, you need to create an array-valued input from the scaled data and use that for input.  (Recall that the new input node is named `array`, which is convenient because the array image has one band, named `array` by default.)  The inputs will be provided as 16x16 patches (`inputTileSize`), at 30-meter resolution (`proj`), but 8 pixels will be thrown out (`inputOverlapSize`) to minimize boundary effects.\n",
        "\n",
        "##### Outputs\n",
        "The output (which you also need to know), is a single float band named `mining`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AluqGcyEDaUI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6884407a-7dde-429c-b274-d41158674ebb"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['VV',\n",
              " 'VH',\n",
              " 's1_ratio',\n",
              " 's1_ndratio',\n",
              " 'B1',\n",
              " 'B2',\n",
              " 'B3',\n",
              " 'B4',\n",
              " 'B5',\n",
              " 'B6',\n",
              " 'B7',\n",
              " 'B8',\n",
              " 'B8A',\n",
              " 'B9',\n",
              " 'B11',\n",
              " 'B12']"
            ]
          },
          "metadata": {},
          "execution_count": 115
        }
      ],
      "source": [
        "image.select(INPUT_BANDS).bandNames().getInfo()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5MBsMkHw_6xV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 771
        },
        "outputId": "bf572061-9f79-4810-8b75-b0e024b3ee8f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<folium.folium.Map at 0x7ad5b9670f40>"
            ],
            "text/html": [
              "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><span style=\"color:#565656\">Make this Notebook Trusted to load map: File -> Trust Notebook</span><iframe srcdoc=\"&lt;!DOCTYPE html&gt;\n",
              "&lt;html&gt;\n",
              "&lt;head&gt;\n",
              "    \n",
              "    &lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;\n",
              "    \n",
              "        &lt;script&gt;\n",
              "            L_NO_TOUCH = false;\n",
              "            L_DISABLE_3D = false;\n",
              "        &lt;/script&gt;\n",
              "    \n",
              "    &lt;style&gt;html, body {width: 100%;height: 100%;margin: 0;padding: 0;}&lt;/style&gt;\n",
              "    &lt;style&gt;#map {position:absolute;top:0;bottom:0;right:0;left:0;}&lt;/style&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://code.jquery.com/jquery-1.12.4.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.js&quot;&gt;&lt;/script&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/leaflet@1.9.3/dist/leaflet.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.0/css/all.min.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/Leaflet.awesome-markers/2.0.2/leaflet.awesome-markers.css&quot;/&gt;\n",
              "    &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdn.jsdelivr.net/gh/python-visualization/folium/folium/templates/leaflet.awesome.rotate.min.css&quot;/&gt;\n",
              "    \n",
              "            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,\n",
              "                initial-scale=1.0, maximum-scale=1.0, user-scalable=no&quot; /&gt;\n",
              "            &lt;style&gt;\n",
              "                #map_396c9a7f7aebfa7fa9aa973aff5c400b {\n",
              "                    position: relative;\n",
              "                    width: 100.0%;\n",
              "                    height: 100.0%;\n",
              "                    left: 0.0%;\n",
              "                    top: 0.0%;\n",
              "                }\n",
              "                .leaflet-container { font-size: 1rem; }\n",
              "            &lt;/style&gt;\n",
              "        \n",
              "&lt;/head&gt;\n",
              "&lt;body&gt;\n",
              "    \n",
              "    \n",
              "            &lt;div class=&quot;folium-map&quot; id=&quot;map_396c9a7f7aebfa7fa9aa973aff5c400b&quot; &gt;&lt;/div&gt;\n",
              "        \n",
              "&lt;/body&gt;\n",
              "&lt;script&gt;\n",
              "    \n",
              "    \n",
              "            var map_396c9a7f7aebfa7fa9aa973aff5c400b = L.map(\n",
              "                &quot;map_396c9a7f7aebfa7fa9aa973aff5c400b&quot;,\n",
              "                {\n",
              "                    center: [6.074769, -1.981041],\n",
              "                    crs: L.CRS.EPSG3857,\n",
              "                    zoom: 8,\n",
              "                    zoomControl: true,\n",
              "                    preferCanvas: false,\n",
              "                }\n",
              "            );\n",
              "\n",
              "            \n",
              "\n",
              "        \n",
              "    \n",
              "            var tile_layer_fc57caa560e84afe94728a2c8ac0c1b3 = L.tileLayer(\n",
              "                &quot;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Data by \\u0026copy; \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://openstreetmap.org\\&quot;\\u003eOpenStreetMap\\u003c/a\\u003e, under \\u003ca target=\\&quot;_blank\\&quot; href=\\&quot;http://www.openstreetmap.org/copyright\\&quot;\\u003eODbL\\u003c/a\\u003e.&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_396c9a7f7aebfa7fa9aa973aff5c400b);\n",
              "        \n",
              "    \n",
              "            var tile_layer_faad3f2d3729110533898bf07821e3d3 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/40259306db56bb01afa514fa58350110-81ec143a434d1534eebdf818ea367729/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_396c9a7f7aebfa7fa9aa973aff5c400b);\n",
              "        \n",
              "    \n",
              "            var tile_layer_6e96de23633f54bf6ad422dcff2f5f86 = L.tileLayer(\n",
              "                &quot;https://earthengine-highvolume.googleapis.com/v1/projects/servir-ee/maps/e75501c528e05d98e9c096b2d56b698e-a9a5c63acad8b7af54ef4ac178e38c71/tiles/{z}/{x}/{y}&quot;,\n",
              "                {&quot;attribution&quot;: &quot;Google Earth Engine&quot;, &quot;detectRetina&quot;: false, &quot;maxNativeZoom&quot;: 18, &quot;maxZoom&quot;: 18, &quot;minZoom&quot;: 0, &quot;noWrap&quot;: false, &quot;opacity&quot;: 1, &quot;subdomains&quot;: &quot;abc&quot;, &quot;tms&quot;: false}\n",
              "            ).addTo(map_396c9a7f7aebfa7fa9aa973aff5c400b);\n",
              "        \n",
              "    \n",
              "            var layer_control_e6ad22357e3068805ecbe04bbcde8f30 = {\n",
              "                base_layers : {\n",
              "                    &quot;openstreetmap&quot; : tile_layer_fc57caa560e84afe94728a2c8ac0c1b3,\n",
              "                },\n",
              "                overlays :  {\n",
              "                    &quot;Image&quot; : tile_layer_faad3f2d3729110533898bf07821e3d3,\n",
              "                    &quot;predictions&quot; : tile_layer_6e96de23633f54bf6ad422dcff2f5f86,\n",
              "                },\n",
              "            };\n",
              "            L.control.layers(\n",
              "                layer_control_e6ad22357e3068805ecbe04bbcde8f30.base_layers,\n",
              "                layer_control_e6ad22357e3068805ecbe04bbcde8f30.overlays,\n",
              "                {&quot;autoZIndex&quot;: true, &quot;collapsed&quot;: true, &quot;position&quot;: &quot;topright&quot;}\n",
              "            ).addTo(map_396c9a7f7aebfa7fa9aa973aff5c400b);\n",
              "        \n",
              "&lt;/script&gt;\n",
              "&lt;/html&gt;\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ],
      "source": [
        "# Load the trained model and use it for prediction.\n",
        "model = ee.Model.fromAiPlatformPredictor(\n",
        "    projectName = PROJECT,\n",
        "    modelName = MODEL_NAME,\n",
        "    version = VERSION_NAME,\n",
        "    inputTileSize = [16, 16],\n",
        "    inputOverlapSize = [8, 8],\n",
        "    proj = ee.Projection('EPSG:4326').atScale(SCALE),\n",
        "    fixInputProj = True,\n",
        "    outputBands = {OUTPUT_BANDS[0]: {\n",
        "        'type': ee.PixelType.float()\n",
        "      }\n",
        "    }\n",
        ")\n",
        "predictions = model.predictImage(image.select(INPUT_BANDS).toArray())\n",
        "\n",
        "\n",
        "vis_params = {\n",
        "  'min': 0,\n",
        "  'max': 0.4,\n",
        "  'bands': ['B4', 'B3', 'B2'],\n",
        "}\n",
        "\n",
        "\n",
        "map = folium.Map(location=[COORDS[1], COORDS[0]], zoom_start=8)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=image.clip(EXPORT_REGION).getMapId(vis_params)[\"tile_fetcher\"].url_format,\n",
        "    attr=\"Google Earth Engine\",\n",
        "    overlay=True,\n",
        "    name=\"Image\",\n",
        "  ).add_to(map)\n",
        "\n",
        "folium.TileLayer(\n",
        "    tiles=predictions.getMapId({'min': 0, 'max': 1})[\"tile_fetcher\"].url_format,\n",
        "    attr='Google Earth Engine',\n",
        "    overlay=True,\n",
        "    name='predictions',\n",
        "  ).add_to(map)\n",
        "map.add_child(folium.LayerControl())\n",
        "map"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. Using Vertex AI Platform (coming soon ...)"
      ],
      "metadata": {
        "id": "q2QTGuPGHI2d"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Next steps . . ."
      ],
      "metadata": {
        "id": "24gNeiOcyjPy"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}