feature_layers/update_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Update Data\n",
    "A couple different workflows for updating data:\n",
    "* Apply edits -- good for small, transactional edits\n",
    "* Overwrite -- good for ETL when extracting the entire dataset to replace all the existing data\n",
    "* Append -- good for ETL, batch updating existing layers and other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import tempfile\n",
    "import os\n",
    "import shutil\n",
    "import pandas as pd\n",
    "from arcgis.gis import GIS\n",
    "from arcgis.features import FeatureLayerCollection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note**, if you are unable to import local `partnerutils`, **copy the following functions** from [`cool_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/cool_utils.py) and [`feature_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/feature_utils.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from partnerutils.cool_utils import chunk_df\n",
    "from partnerutils.feature_utils import sdf_from_xyz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter password:  ········\n"
     ]
    }
   ],
   "source": [
    "# this will prompt for a password\n",
    "# can also do GIS(\"https://www.arcgis.com\", \"<USERNAME>\", \"<PASSWORD>\")\n",
    "gis = GIS(username=\"mpayson_startups\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply Edits\n",
    "This uses the **[`edit_features`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.features.toc.html#arcgis.features.FeatureLayer.edit_features)** function to append data when working with a [Spatially Enabled DataFrame](https://developers.arcgis.com/python/guide/introduction-to-the-spatially-enabled-dataframe/). This method is best for small updates as it sends individual features as JSON and applies row-by-row edits. You can also update an existing feature using the `updates` parameter in `edit_features` by specifying a common Object ID.\n",
    "\n",
    "Note, after a few updates, I'd [rebuild the indexes](https://github.com/mpayson/esri-partner-tools/blob/master/feature_layers/manage_indexes.ipynb) as they can become unbalanced with many transactional edits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to data\n",
    "csv_path = \"../sample_data/NYC_Restaurant_Inspections.csv\"\n",
    "x_col = \"Longitude\"\n",
    "y_col = \"Latitude\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read csv and construct spatial dataframe\n",
    "df = pd.read_csv(csv_path)\n",
    "sdf = sdf_from_xyz(df, x_col, y_col)\n",
    "len(sdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://Startups.maps.arcgis.com/home/item.html?id=4fa5d18ff8e14ed99a06cd1725d92450' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://Startups.maps.arcgis.com/home/item.html?id=4fa5d18ff8e14ed99a06cd1725d92450' target='_blank'><b>MyFeatureService</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://Startups.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/featureshosted16.png' style=\"vertical-align:middle;\">Feature Layer Collection by mpayson_startups\n",
       "                        <br/>Last Modified: December 10, 2019\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"MyFeatureService\" type:Feature Layer Collection owner:mpayson_startups>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iterate through chunks to create and append data\n",
    "lyr = None\n",
    "for c_df in chunk_df(sdf):\n",
    "    if not lyr:\n",
    "        item = c_df.spatial.to_featurelayer(\"MyFeatureService\", tags=\"esri_partner_tools_sample\")\n",
    "        lyr = item.layers[0]\n",
    "    else:\n",
    "        # THIS IS THE APPEND DATA PART\n",
    "        fs = c_df.spatial.to_featureset()\n",
    "        success = lyr.edit_features(adds=fs)\n",
    "item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overwrite\n",
    "This uses the **[`ovewrite`](https://developers.arcgis.com/python/api-reference/arcgis.features.managers.html?highlight=overwrite#arcgis.features.managers.FeatureLayerCollectionManager.overwrite)** method. It deletes all data in an existing service and replaces it with updated data from the specified file. This is the most efficient method if you are completely updating a service.\n",
    "\n",
    "Note, the overwriting file type **must be the same** as the type used to create the initial service (more info in doc). If you publish from a data frame (like above) the Python API zips to a FGDB or SHP before publishing, so updating with geojson or CSV won't work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get two geojson subsets to showcase the workflow\n",
    "file_path = \"../sample_data/NYC_Restaurant_Inspections.geojson\"\n",
    "with open(file_path) as file:\n",
    "    geojson = json.load(file)\n",
    "\n",
    "features = geojson.pop('features')\n",
    "init_geo = {**geojson, **{'features': features[0:3000]}}\n",
    "next_geo = {**geojson, **{'features': features[3000:5000]}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://Startups.maps.arcgis.com/home/item.html?id=6de4b01ab0d64562a3d1f2638e874e10' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://Startups.maps.arcgis.com/home/item.html?id=6de4b01ab0d64562a3d1f2638e874e10' target='_blank'><b>Inspections</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://Startups.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/featureshosted16.png' style=\"vertical-align:middle;\">Feature Layer Collection by mpayson_startups\n",
       "                        <br/>Last Modified: December 10, 2019\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"Inspections\" type:Feature Layer Collection owner:mpayson_startups>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# publish the initial layer\n",
    "temp_dir_path = tempfile.mkdtemp()\n",
    "temp_file_path = os.path.join(temp_dir_path, 'Inspections.geojson')\n",
    "with open(temp_file_path, 'w') as temp_file:\n",
    "    json.dump(init_geo, temp_file)\n",
    "\n",
    "item = gis.content.add({\n",
    "    \"type\": \"GeoJson\",\n",
    "    \"title\": \"Inspections\",\n",
    "    \"tags\": \"esri_partner_tools_sample\",\n",
    "    #... additional properties like descriptions\n",
    "}, data=temp_file_path)\n",
    "shutil.rmtree(temp_dir_path)\n",
    "\n",
    "lyr_item = item.publish()\n",
    "lyr_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://Startups.maps.arcgis.com/home/item.html?id=6de4b01ab0d64562a3d1f2638e874e10' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://Startups.maps.arcgis.com/home/item.html?id=6de4b01ab0d64562a3d1f2638e874e10' target='_blank'><b>Inspections</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://Startups.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/featureshosted16.png' style=\"vertical-align:middle;\">Feature Layer Collection by mpayson_startups\n",
       "                        <br/>Last Modified: December 10, 2019\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"Inspections\" type:Feature Layer Collection owner:mpayson_startups>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# overwrite the layer with second geojson \n",
    "temp_dir_path = tempfile.mkdtemp()\n",
    "temp_file_path = os.path.join(temp_dir_path, 'Inspections.geojson')\n",
    "with open(temp_file_path, 'w') as temp_file:\n",
    "    json.dump(next_geo, temp_file)\n",
    "\n",
    "# THIS IS THE IMPORTANT PART\n",
    "flc = FeatureLayerCollection.fromitem(lyr_item)\n",
    "flc.manager.overwrite(temp_file_path)\n",
    "\n",
    "shutil.rmtree(temp_dir_path)\n",
    "\n",
    "lyr_item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Append\n",
    "This uses the **[`append`](https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#arcgis.features.FeatureLayer.append)** method (exclusive to ArcGIS Online). It upserts data from a previously existing file item (of any supported type) and is more efficient for larger updates. It can also update existing features. By default, this uses the Object ID attribute to determine which features to update, but you can also specify your own field, this field just [has to be indexed](https://github.com/mpayson/esri-partner-tools/blob/master/feature_layers/manage_indexes.ipynb) in ArcGIS Online."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get two geojson subsets to showcase the workflow\n",
    "file_path = \"../sample_data/NYC_Restaurant_Inspections.geojson\"\n",
    "with open(file_path) as file:\n",
    "    geojson = json.load(file)\n",
    "\n",
    "features = geojson.pop('features')\n",
    "init_geo = {**geojson, **{'features': features[0:3000]}}\n",
    "next_geo = {**geojson, **{'features': features[3000:5000]}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://Startups.maps.arcgis.com/home/item.html?id=21871e0fe19041d2af3792ab9e1c54e9' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://Startups.maps.arcgis.com/home/item.html?id=21871e0fe19041d2af3792ab9e1c54e9' target='_blank'><b>Inspections</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://Startups.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/featureshosted16.png' style=\"vertical-align:middle;\">Feature Layer Collection by mpayson_startups\n",
       "                        <br/>Last Modified: January 29, 2020\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"Inspections\" type:Feature Layer Collection owner:mpayson_startups>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# publish the initial layer\n",
    "temp_dir_path = tempfile.mkdtemp()\n",
    "temp_file_path = os.path.join(temp_dir_path, 'Inspections.geojson')\n",
    "with open(temp_file_path, 'w') as temp_file:\n",
    "    json.dump(init_geo, temp_file)\n",
    "\n",
    "item = gis.content.add({\n",
    "    \"type\": \"GeoJson\",\n",
    "    \"title\": \"Inspections\",\n",
    "    \"tags\": \"esri_partner_tools_sample\",\n",
    "    # ... additional properties like descriptions\n",
    "}, data=temp_file_path)\n",
    "shutil.rmtree(temp_dir_path)\n",
    "\n",
    "lyr_item = item.publish()\n",
    "lyr_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://Startups.maps.arcgis.com/home/item.html?id=b5cbd5325cd74cb9badee93fd8a7d283' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://Startups.maps.arcgis.com/home/item.html?id=b5cbd5325cd74cb9badee93fd8a7d283' target='_blank'><b>Append Inspections</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://Startups.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/layers16.png' style=\"vertical-align:middle;\">GeoJson by mpayson_startups\n",
       "                        <br/>Last Modified: January 29, 2020\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"Append Inspections\" type:GeoJson owner:mpayson_startups>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# overwrite the layer with second geojson \n",
    "temp_dir_path = tempfile.mkdtemp()\n",
    "temp_file_path = os.path.join(temp_dir_path, 'AppendInspections.geojson')\n",
    "with open(temp_file_path, 'w') as temp_file:\n",
    "    json.dump(next_geo, temp_file)\n",
    "    \n",
    "# add geojson as an item\n",
    "append_item = gis.content.add({\n",
    "    \"type\": \"GeoJson\",\n",
    "    \"title\": \"Append Inspections\",\n",
    "    \"tags\": \"esri_partner_tools_sample\"\n",
    "}, data=temp_file_path)\n",
    "shutil.rmtree(temp_dir_path)\n",
    "\n",
    "lyr = lyr_item.layers[0]\n",
    "lyr.append(\n",
    "    item_id=append_item.id,\n",
    "    upload_format='geojson',\n",
    "    # optional field to identify existing data to update from new data\n",
    "    # field needs to be uniquely indexed (see manage indexes notebook)\n",
    "    # by default, this is the ObjectID\n",
    "#     upsert_matching_field=\"field_name\"\n",
    ")\n",
    "\n",
    "lyr_item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "delete_items = gis.content.search(\"tags:esri_partner_tools_sample\")\n",
    "gis.content.delete_items(delete_items)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}