Research Task - ArcPro template for updating the definition associated with each field

[tiffanychu90]

Research Task

Use this template to create issues that represent small chunks of work (.5 - 2 days) that stand alone, or represent incremental work as part of an Epic Issue Template.

• Relevant PR (if applicable):

• Question or Goal: We always submit DDP6/DDP8 excel spreadsheets for new open data layers to be published. One is the metadata XML for the entire layer, the other is the data dictionary. During the Geoportal inventory, the GIS team has asked for the data dictionary portion to be supplied in the automatic update, and not only in the email request. Previously, it's been ok to use a placeholder url because the spreadsheet gets uploaded manually on their end.

• Research Required: Figure out how to work this into our templating process.
  The field name overwrite occurs with arcpy, and the definitions likely will be populated in that step.
  Need a robust enough process to accommodate the fact that we often add new columns, and they will need definitions.

The field info don't appear in the XML template -- double check this to be true.

Epic: #544
Related to: #802, #810

[tiffanychu90]

ArcPy docs...everything related to working with fields:

• arcpy functions / fields - add field delimiters, parse field name, validate field name
• arcpy class / fields - field, field info, field map, field mappings
• arcpy data access module / describe

Nothing in docs indicates ability to change field definition beyond doing it manually. Changing it manually and exporting metadata....fields do not appear in XML, so we can't populate it ourselves.

[tiffanychu90]

TODO

• see if using arcpy, we can overwrite an entire table, that way, we don't have to keep updating field description
• filling out metadata xml template...use the optional field to include URL to readme
• for metadata xml template...tags is nowhere to be found, and it shouldn't be in the theme keywords section. theme keywords should be found from this list, and we should use transportation. use arcpy.metadata class to also add tags when we're renaming fields arcpy metadata class
• neither of these docs show we can add field description (field definition)
  • https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/add-field.htm
  • https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/add-fields.htm
  • https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/batch-update-fields.htm

[tiffanychu90]

Temporary resolution

• Move to metadata.yml and data_dictionary.yml as a simpler way for us to document what we want to input as metadata
• Create for each dataset an FGDC template of fields. Then go in a update it update_data_dict.py and update_fields_fgdc.py
• Apply the ISO 19139 template for each dataset and programmatically update that each month
• Sync that in ESRI
• Sync the FGDC template to bring in those field definitions/definition sources
• Sync that to enterprise gdb

Temporary because FGDC apparently is being phased out and we won't be able to make use of this loophole as a workaround. We can take advantage of the fact that FGDC metadata can have ArcGIS metadata added to it, but not vice versa.
https://pro.arcgis.com/en/pro-app/latest/help/metadata/upgrade-metadata-to-the-arcgis-format.htm