WBRef Web Science Activity Pages.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook explores a new technique for working through the USGS \"/science/\" pages. These are the web pages created and maintained by USGS Science Centers describing their work and activities. They represent the best (and almost only) online, public documentation of USGS projects. They sometimes trace back to internal project management processes.\n",
    "\n",
    "The notebook has two parts:\n",
    "* New(ish) function for scraping project web pages (there is still no API available for any other type of access to the material)\n",
    "* Exploratory process using an OpenAI large language model to read the textual descripive information from the page and return linkable concepts\n",
    "\n",
    "The basic prompt explored here ends up doing a more thorough job with a much faster response time than any other methods I've experimented with in the past. I run through two examples, one in a more ecosystems-related field and the other in a geoscience field. The basic concept of operations with this is to use AI to identify important things we want to know about from our texts and then link those to defined concepts in the GeoKB. We won't link everything a given pass with an LLM identifies, but the confirmed linkages we make will theoretically prove more meaningful and usable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import openai\n",
    "import dateutil.parser\n",
    "import json\n",
    "from wbmaker import WikibaseConnection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_project(url):\n",
    "    usgs_base_url = \"https://www.usgs.gov\"\n",
    "    if \"science\" not in [i.lower() for i in url.split(\"/\")]:\n",
    "        return\n",
    "\n",
    "    r = requests.get(url)\n",
    "    if r.status_code != 200:\n",
    "        return\n",
    "\n",
    "    soup = BeautifulSoup(r.content, 'html.parser')\n",
    "    project = {}\n",
    "\n",
    "    title_section = soup.find('h1')\n",
    "    title_section_parts = [i.strip() for i in title_section.text.split('\\n') if len(i.strip()) > 0]\n",
    "    project[\"label\"] = title_section_parts[0]\n",
    "    project[\"status\"] = title_section_parts[1]\n",
    "\n",
    "    by_line_link = soup.find('span', {'class': 'by-line'}).find('a')\n",
    "    project[\"org_owner\"] = by_line_link.text.strip()\n",
    "    project[\"org_owner_link\"] = f\"{usgs_base_url}{by_line_link['href']}\"\n",
    "\n",
    "    project[\"article_date\"] = dateutil.parser.parse(soup.find('span', {'class': 'date'}).text.strip()).isoformat()\n",
    "\n",
    "    project[\"contacts\"] = []\n",
    "\n",
    "    contact_section = soup.find('div', {'class': 'paragraph--type--contacts'})\n",
    "    for c in contact_section.find_all('div', {'class': 'field-contacts'}):\n",
    "        contact = {}\n",
    "        name_section = c.find('h4')\n",
    "        contact[\"name\"] = name_section.find('span').text.strip()\n",
    "        contact[\"link\"] = f\"{usgs_base_url}{name_section.find('a')['href']}\"\n",
    "        contact[\"email\"] = c.find('div', {'class': 'field-email'}).text.strip()\n",
    "        project[\"contacts\"].append(contact)\n",
    "\n",
    "    project[\"keywords\"] = []\n",
    "    explore_section = soup.find('div', {'class': 'c-usgs-explore-science'})\n",
    "    if explore_section:\n",
    "        for item in explore_section.find('ul').find_all('li'):\n",
    "            project[\"keywords\"].append({\n",
    "                \"term\": item.text.strip(),\n",
    "                \"link\": f\"{usgs_base_url}{item.find('a')['href']}\"\n",
    "            })\n",
    "\n",
    "    project[\"partners\"] = []\n",
    "    partner_section = soup.find('div', {'id': 'partners'})\n",
    "    if partner_section:\n",
    "        for item in partner_section.find_all('a'):\n",
    "            project[\"partners\"].append({\n",
    "                \"partner_name\": item.text.strip(),\n",
    "                \"partner_link\": item['href']\n",
    "            })\n",
    "\n",
    "    overview_section = soup.find('div', {'id': 'overview'})\n",
    "    project[\"overview_texts\"] = []\n",
    "    for p in overview_section.find_all('p'):\n",
    "        p_text = p.text.strip()\n",
    "        if not p_text.startswith('Return to') and len(p_text) > 50:\n",
    "            project[\"overview_texts\"].append(p.text.strip())\n",
    "\n",
    "    return project\n",
    "\n",
    "prompts = {\n",
    "    \"concepts\": \"Read the description of a scientific project provided after this sentence and return the following in objects and lists of a JSON string: concise summary of less than 250 characters, list of study objectives, named geographic places that can be found on a map, other types of places described in words, organization names and descriptions of their role, scientific disciplines, scientific methods and techniques being used, types of scientific data used or produced, types of scientific models used or produced, geologic formations discussed, rock types identified, geologic time periods described, mineral and/or energy commodities identified, mineral species mentioned, biological species mentioned.\",\n",
    "}\n",
    "\n",
    "def llm_text(long_text, prompt=\"summary\", model=\"gpt-3.5-turbo\"):\n",
    "    prompt_str = prompts[prompt]\n",
    "    response = openai.ChatCompletion.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"You are a very helpful assistant.\"\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"{prompt_str}\\n\\n {long_text}\"\n",
    "            },\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    if response and \"choices\" in response:\n",
    "        return json.loads(response[\"choices\"][0][\"message\"][\"content\"])\n",
    "    \n",
    "    return\n",
    "\n",
    "def llm_definitions(terms, context, model=\"gpt-3.5-turbo\"):\n",
    "    response = openai.ChatCompletion.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"You are a very helpful assistant.\"\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"For the following list of terms you provided in a previous response about {context}, please provide concise definitions for each term and a link to an online source of definition (if available) in a list of JSON objects: {', '.join(terms)}\"\n",
    "            },\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    if response and \"choices\" in response:\n",
    "        return response[\"choices\"][0][\"message\"][\"content\"]\n",
    "    \n",
    "    return"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bio/Eco Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': 'Impacts of coastal and watershed changes on upper estuaries: causes and implications of wetland ecosystem transitions along the US Atlantic and Gulf Coasts',\n",
       " 'status': 'Active',\n",
       " 'org_owner': 'Climate Research and Development Program',\n",
       " 'org_owner_link': 'https://www.usgs.gov/programs/climate-research-and-development-program',\n",
       " 'article_date': '2020-02-26T00:00:00',\n",
       " 'contacts': [{'name': 'Ken Krauss, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/ken-krauss',\n",
       "   'email': 'kraussk@usgs.gov'},\n",
       "  {'name': 'Gregory Noe',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/gregory-noe',\n",
       "   'email': 'gnoe@usgs.gov'},\n",
       "  {'name': 'Camille LaFosse Stagg, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/camille-lafosse-stagg',\n",
       "   'email': 'staggc@usgs.gov'},\n",
       "  {'name': 'Hongqing Wang, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/hongqing-wang',\n",
       "   'email': 'wangh@usgs.gov'},\n",
       "  {'name': 'Eric J Ward, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/eric-j-ward',\n",
       "   'email': 'eward@usgs.gov'}],\n",
       " 'keywords': [{'term': 'Biogeochemical Cycling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Biogeochemical Cycling'},\n",
       "  {'term': 'Climate Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Change'},\n",
       "  {'term': 'Drought and Floods',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Drought and Floods'},\n",
       "  {'term': 'Ecosystem Modelling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Modelling'},\n",
       "  {'term': 'Ecosystem Services',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Services'},\n",
       "  {'term': 'Forests', 'link': 'https://www.usgs.gov/search?keywords=Forests'},\n",
       "  {'term': 'Oceans and Estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Oceans and Estuaries'},\n",
       "  {'term': 'Sea Level Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Change'},\n",
       "  {'term': 'Soils', 'link': 'https://www.usgs.gov/search?keywords=Soils'},\n",
       "  {'term': 'Water Quality and Quantity',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Quality and Quantity'},\n",
       "  {'term': 'Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetlands'},\n",
       "  {'term': 'Adaptation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Adaptation'},\n",
       "  {'term': 'Climate Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Change'},\n",
       "  {'term': 'Coastal Ecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Coastal Ecology'},\n",
       "  {'term': 'Conservation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Conservation'},\n",
       "  {'term': 'Conservation Planning',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Conservation Planning'},\n",
       "  {'term': 'Drought', 'link': 'https://www.usgs.gov/search?keywords=Drought'},\n",
       "  {'term': 'Ecological Modeling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecological Modeling'},\n",
       "  {'term': 'Eutrophication',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Eutrophication'},\n",
       "  {'term': 'Geospatial',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Geospatial'},\n",
       "  {'term': 'Gulf of Mexico',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Gulf of Mexico'},\n",
       "  {'term': 'Hurricane/Storms',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hurricane/Storms'},\n",
       "  {'term': 'Hydrological Alteration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrological Alteration'},\n",
       "  {'term': 'Hydrology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrology'},\n",
       "  {'term': 'Land Use / Land Cover Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Land Use / Land Cover Change'},\n",
       "  {'term': 'Plants', 'link': 'https://www.usgs.gov/search?keywords=Plants'},\n",
       "  {'term': 'Restoration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Restoration'},\n",
       "  {'term': 'Sea Level Rise',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Rise'},\n",
       "  {'term': 'Sedimentation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sedimentation'},\n",
       "  {'term': 'Wetland Ecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetland Ecology'},\n",
       "  {'term': 'Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetlands'},\n",
       "  {'term': 'Brackish',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Brackish'},\n",
       "  {'term': 'Coastal', 'link': 'https://www.usgs.gov/search?keywords=Coastal'},\n",
       "  {'term': 'Estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Estuaries'},\n",
       "  {'term': 'Floodplains',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Floodplains'},\n",
       "  {'term': 'Forested Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Forested Wetlands'},\n",
       "  {'term': 'Forests', 'link': 'https://www.usgs.gov/search?keywords=Forests'},\n",
       "  {'term': 'Freshwater',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Freshwater'},\n",
       "  {'term': 'GIS', 'link': 'https://www.usgs.gov/search?keywords=GIS'},\n",
       "  {'term': 'Habitat Mapping',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Habitat Mapping'},\n",
       "  {'term': 'Habitat Models',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Habitat Models'},\n",
       "  {'term': 'Marshes', 'link': 'https://www.usgs.gov/search?keywords=Marshes'},\n",
       "  {'term': 'Remote Sensing',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Remote Sensing'},\n",
       "  {'term': 'Climate Hazards',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Hazards'},\n",
       "  {'term': 'Climate Variability',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Variability'},\n",
       "  {'term': 'Ecosystem Resilience',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Resilience'},\n",
       "  {'term': 'Sea Level',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level'},\n",
       "  {'term': 'Sea Level Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Change'},\n",
       "  {'term': 'Water Resources',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Resources'},\n",
       "  {'term': 'Coastal and Wetland Ecosystems',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Coastal and Wetland Ecosystems'},\n",
       "  {'term': 'Cycling of Carbon and Nutrients',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Cycling of Carbon and Nutrients'},\n",
       "  {'term': 'Hydrologic Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrologic Change'},\n",
       "  {'term': 'Landscape Change and Impacts',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Landscape Change and Impacts'},\n",
       "  {'term': 'Paleoclimate',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Paleoclimate'},\n",
       "  {'term': 'Species Stressors',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Species Stressors'},\n",
       "  {'term': 'Priority Landscapes',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Priority Landscapes'},\n",
       "  {'term': 'Ecological Stressors',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecological Stressors'},\n",
       "  {'term': 'Management and Restoration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Management and Restoration'},\n",
       "  {'term': 'Paleoclimate and Paleoecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Paleoclimate and Paleoecology'},\n",
       "  {'term': 'Landscape Science',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Landscape Science'},\n",
       "  {'term': 'Climate', 'link': 'https://www.usgs.gov/search?keywords=Climate'},\n",
       "  {'term': 'Coasts', 'link': 'https://www.usgs.gov/search?keywords=Coasts'},\n",
       "  {'term': 'Biology', 'link': 'https://www.usgs.gov/search?keywords=Biology'},\n",
       "  {'term': 'Ecosystems',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystems'},\n",
       "  {'term': 'estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=estuaries'},\n",
       "  {'term': 'tidal freshwater forested wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=tidal freshwater forested wetlands'},\n",
       "  {'term': 'sea level rise',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=sea level rise'},\n",
       "  {'term': 'Water Quality',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Quality'},\n",
       "  {'term': 'land use change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=land use change'}],\n",
       " 'partners': [{'partner_name': 'Chesapeake Bay Program',\n",
       "   'partner_link': 'http://www.chesapeakebay.net/'},\n",
       "  {'partner_name': 'Chesapeake Bay-Virginia National Estuarine Research Reserve',\n",
       "   'partner_link': 'https://www.vims.edu/cbnerr/'},\n",
       "  {'partner_name': 'Clemson University',\n",
       "   'partner_link': 'http://www.clemson.edu/'}],\n",
       " 'overview_texts': ['Estuaries and their surrounding wetlands are coastal transition zones where freshwater rivers meet tidal seawater.\\xa0 As sea levels rise, tidal forces move saltier water farther upstream, extending into freshwater wetland areas. Human changes to the surrounding landscape may amplify the effects of this tidal extension, impacting the resiliency and function of the upper estuarine wetlands. One visible indicator is the rapid conversion of some Southeast and mid-Atlantic tidal freshwater forested wetlands to ‘Ghost Forests’ in which trees die from increases in salinity. Because data on the complex causes and impacts of tidal extension are limited, this project takes an integrated, large-scale approach to research and monitoring to expand our ability to model these processes and apply them to other coastal areas along the Atlantic Coast, Gulf Coast, and internationally. Results of this effort will provide critical data to guide future decisions regarding the fate of carbon, water quality, coastal resilience, wildlife and fisheries, and effective allocation of taxpayer dollars for ecosystem restoration.',\n",
       "  'Statement of Problem:\\xa0Throughout the southeastern United States, tidal freshwater forested wetlands (TFFW) found along the upper tidal estuary (where freshwater watersheds and estuaries meet), are converting to ‘Ghost Forests’ (dead trees) and then to oligohaline (low salinity) marsh. Nontidal floodplains are also converting into TFFW as tides extend farther upstream along rivers, and little scientific research has been directed toward describing or understanding the effects of this tidal extension.',\n",
       "  'Conversion is related to both changing sea level and associated salt-water intrusion and to human influences (e.g., land use change, coastal development, construction of dams, river dredging, etc). All these factors affect the resiliency and ecosystem services provided by wetlands of the upper estuary. Watershed inputs to the upper estuary (water, sediment, nutrients) also may determine the response of TFFW through complex interactions between the ecosystem, water, and landscape. Further, TFFW itself influences delivery of watershed sediment to lower tidal estuaries that are targets of restoration projects, making their function critical to ecosystem health.',\n",
       "  'Why this Research is Important:\\xa0This project fills a large gap in integrative ecosystem research and monitoring of different coastal areas which vary in land use activity, tidal range, relative sea-level rise, watershed characteristics and changes, and delivery of ecosystem services for taxpayers. A primary outcome thus far has been the value assigned to TFFW in terms of carbon storage, sequestration, and conveyance; adding them to the list of important \"blue carbon\" wetlands globally.',\n",
       "  '“Blue Carbon” refers to carbon that is stored in coastal or marine ecosystems. In addition to the monetary value of the ecosystem services, these coastal wetland areas are crucial for protecting the coast from storm surge, mitigating downstream eutrophication (when a water body becomes overly enriched in nutrients), and providing wildlife and fishery habitat. By improving understanding of the processes and services provided by both healthy TFFW and those transitioned to marshes, managers and coastal inhabitants (and taxpayers) will be able to anticipate how past and current changes will affect the ecosystems in the future.',\n",
       "  'The TFFW transition/conversion phenomenon is not unique to the southeast and mid-Atlantic United States, and expansion of the project enhances our ability to (1) apply this research to other domestic and international coastal areas undergoing similar changes, and (2) add to critical information gaps on carbon sequestration to better understand the role of coastal ecosystems in the cycling of carbon and nutrients).',\n",
       "  'Objective(s):\\xa0We hypothesize that ecosystem functions, sediment and nutrient processes, and carbon sequestration vary in predictable ways in different coastal environments as nontidal floodplains convert to TFFW and TFFW convert to marshes. Through extensive data collection over 15 years, we have quantified several processes and changes to sedimentation, nutrient uptake and release, forest condition, and balance of habitat. From analyses of these data we developed an understanding of emergent processes at multiple scales that are critical to future investigation, culminating in predictive models and advancements to ecological theory of coastal wetland change. In order to further expand our focus from TFFW stresses and degradation, our focus moves to the larger scale of the entire upper estuary landscape, guiding a series of new goals and objectives:',\n",
       "  'Methods:\\xa0We use a combination of complementary field, laboratory, modeling, and remote sensing techniques to achieve our objectives. Changes in vegetation, tidal extent, surface elevation, and microtopography are measured using field data collection. For example, plots of trees and herbaceous vegetation are surveyed over time to document a record of habitat change, while direct measurements of growth records inform contemporary and historic productivity.',\n",
       "  'Water level and salinity are measured continuously with logging devices. Surfaces of hummocks and hollows are measured annually to assess changes in elevation, accretion, subsidence and general resilience to sea-level rise.\\xa0 More complex methods that involve both field collection and laboratory analyses are required to measure soil biogeochemistry, nutrients, and carbon (C) storage and flux.',\n",
       "  'We collect and analyze shallow soil cores (up to 1-meter in depth), for total organic carbon and nitrogen, and date them using Pb-210 activity to determine rates of soil C burial. In analyses of deeper cores (up to 6-7 meters depth), macrofossils, pollen, C-14, and carbon and nutrient sequestration are used to develop historic baselines prior to rising sea-level conditions. Further soil and vegetation analyses assess ground carbon stocks, belowground net ecosystem productivity, greenhouse gas production (CO2, CH4, N2O) in soils of hummocks and hollows, and transpirational demand on soils in forests vs. newly converted marsh.',\n",
       "  'Finally, river channel bathymetry, longitudinal sampling of suspended and bed sediment, and analysis of sediment stable isotopes and metals to determine changes in mineral and organic sediment sources (“sediment fingerprinting”) are used together to determine how channels and floodplains have changed and have impacted sediment transport.\\xa0 We will use remote sensing techniques to gather information on the extent of tidal flooding along upper estuarine reaches in our study areas.\\xa0 Using our data, we will validate remote sensing results and apply it to regional and national mapping efforts, as well as model development. Synthesizing the various data streams is accomplished through development and application of both statistical and dynamic process models to generate new understanding and tools to predict ecosystem change in new coastal areas.']}"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_project = scrape_project(\"https://www.usgs.gov/programs/climate-research-and-development-program/science/impacts-coastal-and-watershed-changes\")\n",
    "sample_project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7026\n"
     ]
    }
   ],
   "source": [
    "long_text = \" \".join(sample_project['overview_texts'])\n",
    "print(len(long_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_concepts = llm_text(\n",
    "    long_text=long_text,\n",
    "    prompt=\"concepts\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'concise_summary': \"This project studies the conversion of tidal freshwater forested wetlands to 'Ghost Forests' caused by tidal extension and human influences, and its impact on the upper estuarine wetlands' resiliency and function, using an integrated approach to research and monitoring and scientific methods in different coastal areas. The results guide future decisions on ecosystem restoration, carbon, water quality, coastal resilience, wildlife and fisheries.\",\n",
       " 'geographic_places': ['Atlantic Coast',\n",
       "  'Gulf Coast',\n",
       "  'southeastern United States',\n",
       "  'international coastal areas'],\n",
       " 'other_places': ['watersheds',\n",
       "  'estuaries',\n",
       "  'oligohaline marsh',\n",
       "  'nontidal floodplains'],\n",
       " 'organizations': [{'name': 'National Science Foundation (NSF)',\n",
       "   'description': 'funding agency that supports the project through the Coastal SEES (Science, Engineering, and Education for Sustainability) Program'},\n",
       "  {'name': 'University of Maryland (UMD)',\n",
       "   'description': 'lead institution of researchers involved in the project'},\n",
       "  {'name': 'Smithsonian Environmental Research Center (SERC)',\n",
       "   'description': 'research team based organization involved in the project'}],\n",
       " 'scientific_disciplines': ['ecology',\n",
       "  'biogeochemistry',\n",
       "  'geology',\n",
       "  'remote sensing'],\n",
       " 'scientific_methods_and_techniques': ['field data collection',\n",
       "  'laboratory analyses',\n",
       "  'modeling',\n",
       "  'remote sensing'],\n",
       " 'scientific_data': ['vegetation data',\n",
       "  'growth records',\n",
       "  'water level data',\n",
       "  'salinity data',\n",
       "  'soil core data',\n",
       "  'sediment data',\n",
       "  'stable isotopes'],\n",
       " 'scientific_models': ['ecological models', 'dynamic process models'],\n",
       " 'geologic_formations': [],\n",
       " 'rock_formations': [],\n",
       " 'rock_types': [],\n",
       " 'geologic_time_periods': [],\n",
       " 'other_geologic_features': [],\n",
       " 'mineral_commodities': [],\n",
       " 'energy_commodities': [],\n",
       " 'mineral_species': [],\n",
       " 'biological_species': ['wildlife', 'fishery habitats'],\n",
       " 'objectives': ['quantify ecosystem functions, sediment and nutrient processes, and carbon sequestration in different coastal environments',\n",
       "  'understand emergent processes critical to future investigation and advancements to ecological theory of coastal wetland change',\n",
       "  'develop predictive models',\n",
       "  'gather information on the extent of tidal flooding along upper estuarine reaches in the study areas',\n",
       "  'validate remote sensing results',\n",
       "  'apply models to predict ecosystem change in new coastal areas']}"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extracted_concepts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Geo Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': 'Impacts of coastal and watershed changes on upper estuaries: causes and implications of wetland ecosystem transitions along the US Atlantic and Gulf Coasts',\n",
       " 'status': 'Active',\n",
       " 'org_owner': 'Climate Research and Development Program',\n",
       " 'org_owner_link': 'https://www.usgs.gov/programs/climate-research-and-development-program',\n",
       " 'article_date': '2020-02-26T00:00:00',\n",
       " 'contacts': [{'name': 'Ken Krauss, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/ken-krauss',\n",
       "   'email': 'kraussk@usgs.gov'},\n",
       "  {'name': 'Gregory Noe',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/gregory-noe',\n",
       "   'email': 'gnoe@usgs.gov'},\n",
       "  {'name': 'Camille LaFosse Stagg, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/camille-lafosse-stagg',\n",
       "   'email': 'staggc@usgs.gov'},\n",
       "  {'name': 'Hongqing Wang, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/hongqing-wang',\n",
       "   'email': 'wangh@usgs.gov'},\n",
       "  {'name': 'Eric J Ward, Ph.D.',\n",
       "   'link': 'https://www.usgs.gov/staff-profiles/eric-j-ward',\n",
       "   'email': 'eward@usgs.gov'}],\n",
       " 'keywords': [{'term': 'Biogeochemical Cycling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Biogeochemical Cycling'},\n",
       "  {'term': 'Climate Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Change'},\n",
       "  {'term': 'Drought and Floods',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Drought and Floods'},\n",
       "  {'term': 'Ecosystem Modelling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Modelling'},\n",
       "  {'term': 'Ecosystem Services',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Services'},\n",
       "  {'term': 'Forests', 'link': 'https://www.usgs.gov/search?keywords=Forests'},\n",
       "  {'term': 'Oceans and Estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Oceans and Estuaries'},\n",
       "  {'term': 'Sea Level Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Change'},\n",
       "  {'term': 'Soils', 'link': 'https://www.usgs.gov/search?keywords=Soils'},\n",
       "  {'term': 'Water Quality and Quantity',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Quality and Quantity'},\n",
       "  {'term': 'Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetlands'},\n",
       "  {'term': 'Adaptation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Adaptation'},\n",
       "  {'term': 'Climate Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Change'},\n",
       "  {'term': 'Coastal Ecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Coastal Ecology'},\n",
       "  {'term': 'Conservation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Conservation'},\n",
       "  {'term': 'Conservation Planning',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Conservation Planning'},\n",
       "  {'term': 'Drought', 'link': 'https://www.usgs.gov/search?keywords=Drought'},\n",
       "  {'term': 'Ecological Modeling',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecological Modeling'},\n",
       "  {'term': 'Eutrophication',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Eutrophication'},\n",
       "  {'term': 'Geospatial',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Geospatial'},\n",
       "  {'term': 'Gulf of Mexico',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Gulf of Mexico'},\n",
       "  {'term': 'Hurricane/Storms',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hurricane/Storms'},\n",
       "  {'term': 'Hydrological Alteration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrological Alteration'},\n",
       "  {'term': 'Hydrology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrology'},\n",
       "  {'term': 'Land Use / Land Cover Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Land Use / Land Cover Change'},\n",
       "  {'term': 'Plants', 'link': 'https://www.usgs.gov/search?keywords=Plants'},\n",
       "  {'term': 'Restoration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Restoration'},\n",
       "  {'term': 'Sea Level Rise',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Rise'},\n",
       "  {'term': 'Sedimentation',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sedimentation'},\n",
       "  {'term': 'Wetland Ecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetland Ecology'},\n",
       "  {'term': 'Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Wetlands'},\n",
       "  {'term': 'Brackish',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Brackish'},\n",
       "  {'term': 'Coastal', 'link': 'https://www.usgs.gov/search?keywords=Coastal'},\n",
       "  {'term': 'Estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Estuaries'},\n",
       "  {'term': 'Floodplains',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Floodplains'},\n",
       "  {'term': 'Forested Wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Forested Wetlands'},\n",
       "  {'term': 'Forests', 'link': 'https://www.usgs.gov/search?keywords=Forests'},\n",
       "  {'term': 'Freshwater',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Freshwater'},\n",
       "  {'term': 'GIS', 'link': 'https://www.usgs.gov/search?keywords=GIS'},\n",
       "  {'term': 'Habitat Mapping',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Habitat Mapping'},\n",
       "  {'term': 'Habitat Models',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Habitat Models'},\n",
       "  {'term': 'Marshes', 'link': 'https://www.usgs.gov/search?keywords=Marshes'},\n",
       "  {'term': 'Remote Sensing',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Remote Sensing'},\n",
       "  {'term': 'Climate Hazards',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Hazards'},\n",
       "  {'term': 'Climate Variability',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Climate Variability'},\n",
       "  {'term': 'Ecosystem Resilience',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystem Resilience'},\n",
       "  {'term': 'Sea Level',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level'},\n",
       "  {'term': 'Sea Level Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Sea Level Change'},\n",
       "  {'term': 'Water Resources',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Resources'},\n",
       "  {'term': 'Coastal and Wetland Ecosystems',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Coastal and Wetland Ecosystems'},\n",
       "  {'term': 'Cycling of Carbon and Nutrients',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Cycling of Carbon and Nutrients'},\n",
       "  {'term': 'Hydrologic Change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Hydrologic Change'},\n",
       "  {'term': 'Landscape Change and Impacts',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Landscape Change and Impacts'},\n",
       "  {'term': 'Paleoclimate',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Paleoclimate'},\n",
       "  {'term': 'Species Stressors',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Species Stressors'},\n",
       "  {'term': 'Priority Landscapes',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Priority Landscapes'},\n",
       "  {'term': 'Ecological Stressors',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecological Stressors'},\n",
       "  {'term': 'Management and Restoration',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Management and Restoration'},\n",
       "  {'term': 'Paleoclimate and Paleoecology',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Paleoclimate and Paleoecology'},\n",
       "  {'term': 'Landscape Science',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Landscape Science'},\n",
       "  {'term': 'Climate', 'link': 'https://www.usgs.gov/search?keywords=Climate'},\n",
       "  {'term': 'Coasts', 'link': 'https://www.usgs.gov/search?keywords=Coasts'},\n",
       "  {'term': 'Biology', 'link': 'https://www.usgs.gov/search?keywords=Biology'},\n",
       "  {'term': 'Ecosystems',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Ecosystems'},\n",
       "  {'term': 'estuaries',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=estuaries'},\n",
       "  {'term': 'tidal freshwater forested wetlands',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=tidal freshwater forested wetlands'},\n",
       "  {'term': 'sea level rise',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=sea level rise'},\n",
       "  {'term': 'Water Quality',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=Water Quality'},\n",
       "  {'term': 'land use change',\n",
       "   'link': 'https://www.usgs.gov/search?keywords=land use change'}],\n",
       " 'partners': [{'partner_name': 'Chesapeake Bay Program',\n",
       "   'partner_link': 'http://www.chesapeakebay.net/'},\n",
       "  {'partner_name': 'Chesapeake Bay-Virginia National Estuarine Research Reserve',\n",
       "   'partner_link': 'https://www.vims.edu/cbnerr/'},\n",
       "  {'partner_name': 'Clemson University',\n",
       "   'partner_link': 'http://www.clemson.edu/'}],\n",
       " 'overview_texts': ['Estuaries and their surrounding wetlands are coastal transition zones where freshwater rivers meet tidal seawater.\\xa0 As sea levels rise, tidal forces move saltier water farther upstream, extending into freshwater wetland areas. Human changes to the surrounding landscape may amplify the effects of this tidal extension, impacting the resiliency and function of the upper estuarine wetlands. One visible indicator is the rapid conversion of some Southeast and mid-Atlantic tidal freshwater forested wetlands to ‘Ghost Forests’ in which trees die from increases in salinity. Because data on the complex causes and impacts of tidal extension are limited, this project takes an integrated, large-scale approach to research and monitoring to expand our ability to model these processes and apply them to other coastal areas along the Atlantic Coast, Gulf Coast, and internationally. Results of this effort will provide critical data to guide future decisions regarding the fate of carbon, water quality, coastal resilience, wildlife and fisheries, and effective allocation of taxpayer dollars for ecosystem restoration.',\n",
       "  'Statement of Problem:\\xa0Throughout the southeastern United States, tidal freshwater forested wetlands (TFFW) found along the upper tidal estuary (where freshwater watersheds and estuaries meet), are converting to ‘Ghost Forests’ (dead trees) and then to oligohaline (low salinity) marsh. Nontidal floodplains are also converting into TFFW as tides extend farther upstream along rivers, and little scientific research has been directed toward describing or understanding the effects of this tidal extension.',\n",
       "  'Conversion is related to both changing sea level and associated salt-water intrusion and to human influences (e.g., land use change, coastal development, construction of dams, river dredging, etc). All these factors affect the resiliency and ecosystem services provided by wetlands of the upper estuary. Watershed inputs to the upper estuary (water, sediment, nutrients) also may determine the response of TFFW through complex interactions between the ecosystem, water, and landscape. Further, TFFW itself influences delivery of watershed sediment to lower tidal estuaries that are targets of restoration projects, making their function critical to ecosystem health.',\n",
       "  'Why this Research is Important:\\xa0This project fills a large gap in integrative ecosystem research and monitoring of different coastal areas which vary in land use activity, tidal range, relative sea-level rise, watershed characteristics and changes, and delivery of ecosystem services for taxpayers. A primary outcome thus far has been the value assigned to TFFW in terms of carbon storage, sequestration, and conveyance; adding them to the list of important \"blue carbon\" wetlands globally.',\n",
       "  '“Blue Carbon” refers to carbon that is stored in coastal or marine ecosystems. In addition to the monetary value of the ecosystem services, these coastal wetland areas are crucial for protecting the coast from storm surge, mitigating downstream eutrophication (when a water body becomes overly enriched in nutrients), and providing wildlife and fishery habitat. By improving understanding of the processes and services provided by both healthy TFFW and those transitioned to marshes, managers and coastal inhabitants (and taxpayers) will be able to anticipate how past and current changes will affect the ecosystems in the future.',\n",
       "  'The TFFW transition/conversion phenomenon is not unique to the southeast and mid-Atlantic United States, and expansion of the project enhances our ability to (1) apply this research to other domestic and international coastal areas undergoing similar changes, and (2) add to critical information gaps on carbon sequestration to better understand the role of coastal ecosystems in the cycling of carbon and nutrients).',\n",
       "  'Objective(s):\\xa0We hypothesize that ecosystem functions, sediment and nutrient processes, and carbon sequestration vary in predictable ways in different coastal environments as nontidal floodplains convert to TFFW and TFFW convert to marshes. Through extensive data collection over 15 years, we have quantified several processes and changes to sedimentation, nutrient uptake and release, forest condition, and balance of habitat. From analyses of these data we developed an understanding of emergent processes at multiple scales that are critical to future investigation, culminating in predictive models and advancements to ecological theory of coastal wetland change. In order to further expand our focus from TFFW stresses and degradation, our focus moves to the larger scale of the entire upper estuary landscape, guiding a series of new goals and objectives:',\n",
       "  'Methods:\\xa0We use a combination of complementary field, laboratory, modeling, and remote sensing techniques to achieve our objectives. Changes in vegetation, tidal extent, surface elevation, and microtopography are measured using field data collection. For example, plots of trees and herbaceous vegetation are surveyed over time to document a record of habitat change, while direct measurements of growth records inform contemporary and historic productivity.',\n",
       "  'Water level and salinity are measured continuously with logging devices. Surfaces of hummocks and hollows are measured annually to assess changes in elevation, accretion, subsidence and general resilience to sea-level rise.\\xa0 More complex methods that involve both field collection and laboratory analyses are required to measure soil biogeochemistry, nutrients, and carbon (C) storage and flux.',\n",
       "  'We collect and analyze shallow soil cores (up to 1-meter in depth), for total organic carbon and nitrogen, and date them using Pb-210 activity to determine rates of soil C burial. In analyses of deeper cores (up to 6-7 meters depth), macrofossils, pollen, C-14, and carbon and nutrient sequestration are used to develop historic baselines prior to rising sea-level conditions. Further soil and vegetation analyses assess ground carbon stocks, belowground net ecosystem productivity, greenhouse gas production (CO2, CH4, N2O) in soils of hummocks and hollows, and transpirational demand on soils in forests vs. newly converted marsh.',\n",
       "  'Finally, river channel bathymetry, longitudinal sampling of suspended and bed sediment, and analysis of sediment stable isotopes and metals to determine changes in mineral and organic sediment sources (“sediment fingerprinting”) are used together to determine how channels and floodplains have changed and have impacted sediment transport.\\xa0 We will use remote sensing techniques to gather information on the extent of tidal flooding along upper estuarine reaches in our study areas.\\xa0 Using our data, we will validate remote sensing results and apply it to regional and national mapping efforts, as well as model development. Synthesizing the various data streams is accomplished through development and application of both statistical and dynamic process models to generate new understanding and tools to predict ecosystem change in new coastal areas.']}"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geo_project_url = \"https://www.usgs.gov/centers/gmeg/science/concealed-rare-earth-element-ree-terranes-southern-basin-and-range-geologic\"\n",
    "geo_project_scrape = scrape_project(\"https://www.usgs.gov/programs/climate-research-and-development-program/science/impacts-coastal-and-watershed-changes\")\n",
    "geo_project_scrape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7026\n"
     ]
    }
   ],
   "source": [
    "geo_project_long_text = \" \".join(geo_project_scrape['overview_texts'])\n",
    "print(len(geo_project_long_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "geo_project_extracted_concepts = llm_text(\n",
    "    long_text=geo_project_long_text,\n",
    "    prompt=\"concepts\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'concise_summary': 'This project investigates the impacts of tidal extension on estuaries and wetlands due to rising sea levels and human activities, and aims to model the processes to guide ecosystem restoration and future decisions in coastal areas along the Atlantic Coast, Gulf Coast, and internationally.',\n",
       " 'study_objectives': ['To understand and describe the effects of tidal extension in freshwater wetlands and nontidal floodplains, and changes to sedimentation, nutrient uptake and release, and habitat balance',\n",
       "  'To quantify ecosystem functions, sediment and nutrient processes, and carbon sequestration and their variability in different coastal environments undergoing TFFW conversion and marsh transition',\n",
       "  'To develop advanced ecological theory of coastal wetland change by analyzing extensive field, laboratory, modeling, and remote sensing data over 15 years and using dynamic process models to predict ecosystem change in new coastal areas'],\n",
       " 'geographic_places': {'named_geographic_places': ['Southeast United States',\n",
       "   'Mid-Atlantic United States',\n",
       "   'Atlantic Coast',\n",
       "   'Gulf Coast',\n",
       "   'International coastal areas'],\n",
       "  'other_types_of_places': ['estuaries',\n",
       "   'wetlands',\n",
       "   'floodplains',\n",
       "   'rivers',\n",
       "   'coastline']},\n",
       " 'organizations': [{'name': 'Not Provided', 'description': 'Not Provided'}],\n",
       " 'scientific_disciplines': ['ecology',\n",
       "  'biogeochemistry',\n",
       "  'sedimentology',\n",
       "  'geology',\n",
       "  'remote sensing',\n",
       "  'Statistics',\n",
       "  'dynamic process modeling'],\n",
       " 'scientific_methods_and_techniques': ['field data collection',\n",
       "  'vegetation survey',\n",
       "  'growth record measurement',\n",
       "  'water level and salinity monitoring',\n",
       "  'topography measurement',\n",
       "  'soil biogeochemistry analysis',\n",
       "  'Pb-210 dating',\n",
       "  'macrofossil and pollen analysis',\n",
       "  'carbon and nutrient analysis',\n",
       "  'river channel bathymetry measurement',\n",
       "  'suspended and bed sediment sampling',\n",
       "  'sediment fingerprinting',\n",
       "  'remote sensing',\n",
       "  'dynamic process modeling'],\n",
       " 'scientific_data': ['field data',\n",
       "  'soil cores data',\n",
       "  'sediment stable isotopes and metals data',\n",
       "  'remote sensing data'],\n",
       " 'scientific_models': ['dynamic process models'],\n",
       " 'geologic_features': ['geologic formations',\n",
       "  'rock formations',\n",
       "  'rock types',\n",
       "  'geologic time periods'],\n",
       " 'mineral_and_energy_commodities': ['carbon', 'nitrogen', 'CO2', 'CH4', 'N2O'],\n",
       " 'mineral_species': [],\n",
       " 'biological_species': [],\n",
       " 'date_produced': 'Not Provided',\n",
       " 'AI_model_version': 'Not Provided'}"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geo_project_extracted_concepts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Following up on Definitions\n",
    "\n",
    "One key in this is to determine whether the AI model we are consulting and what it has been trained on agree with our own definition of concepts discovered. We can follow up by asking for definitions and reference sources on the concepts identified, work through those, add definition and linkages to our own knowledgebase, and iteratively improve both our ability to use what the models identify and feedback improvements to the models through training and fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "method_definitions = llm_definitions(\n",
    "    terms=geo_project_extracted_concepts['scientific_methods_and_techniques'],\n",
    "    context=\"scientific methods and techniques\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'term': 'Field data collection',\n",
       "  'definition': 'The process of collecting information and data about a physical environment or phenomenon in its natural setting using various methods such as observation, measurement, and surveying.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Field_research'},\n",
       " {'term': 'Vegetation survey',\n",
       "  'definition': 'A methodology used to determine the type, abundance, and distribution of the plant species and communities in a particular area, which can provide information about biodiversity, ecosystem health, and natural resource management.',\n",
       "  'source': 'https://www.fs.fed.us/biology/resources/pdfs/veg_protocol.pdf'},\n",
       " {'term': 'Growth record measurement',\n",
       "  'definition': 'A method of quantifying and analyzing the growth patterns of living organisms, such as trees, using various techniques including dendrochronology (tree-ring analysis) and stem increment measurements, which can provide information about climate, environmental conditions, and past events.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Dendrochronology'},\n",
       " {'term': 'Water level and salinity monitoring',\n",
       "  'definition': 'The process of measuring and tracking the water level and salinity in a particular waterbody using various instruments such as gauges, sensors, and sondes, which can provide information about water availability, quality, and ecosystem health.',\n",
       "  'source': 'https://www.usgs.gov/special-topic/water-science-school/science/water-levels-and-streamflow?qt-science_center_objects=0#qt-science_center_objects'},\n",
       " {'term': 'Topography measurement',\n",
       "  'definition': 'A process of defining and measuring the physical and natural features of a land surface, including elevation, slope, and relief, using various methods such as surveying, remote sensing, and mapping, which can provide information about the terrain, geology, and land use patterns.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Topography'},\n",
       " {'term': 'Soil biogeochemistry analysis',\n",
       "  'definition': 'A method of studying the biogeochemical processes and properties of soil, including nutrient cycling, carbon storage, and microbial activity, using various techniques such as chemical analysis, mineralogy, and microbiology, which can provide information about soil quality, productivity, and environmental impact.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Soil_biology'},\n",
       " {'term': 'Pb-210 dating',\n",
       "  'definition': 'A radiometric dating technique used to determine the age of sediment and soil samples based on the decay of the radioactive isotope lead-210, which can provide information about the sediment accumulation rates, erosion, and land-use changes.',\n",
       "  'source': 'https://www.nrc.gov/docs/ML1418/ML14181A987.pdf'},\n",
       " {'term': 'Macrofossil and pollen analysis',\n",
       "  'definition': 'A method of analyzing and identifying the organic remains of plants and animals, such as stems, leaves, pollen grains, and shells, which are preserved in sediment and soil samples, and can provide information about the vegetation history, climate, and environmental changes.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Paleobotany'},\n",
       " {'term': 'Carbon and nutrient analysis',\n",
       "  'definition': 'A method of analyzing and quantifying the organic and inorganic components of soil and sediment samples, including carbon, nitrogen, phosphorus, and other nutrients, using various chemical and physical techniques, which can provide information about soil fertility, nutrient cycling, and ecosystem processes.',\n",
       "  'source': 'https://www.soils.org/discover-soils/soil-basics/chemical-properties/organic-matter'},\n",
       " {'term': 'River channel bathymetry measurement',\n",
       "  'definition': 'A process of measuring and mapping the depth and morphology of a river or stream channel using various techniques such as sonar, GPS, and laser scanning, which can provide information about the hydraulic characteristics, sediment transport, and habitat quality of the waterbody.',\n",
       "  'source': 'https://riverscapes.xyz/methods/bathymetry-survey/'},\n",
       " {'term': 'Suspended and bed sediment sampling',\n",
       "  'definition': 'A method of collecting and analyzing sediment samples suspended in the water column or deposited on the bed of a waterbody using various instruments such as dredges, grabs, and corers, which can provide information about sediment texture, composition, and source.',\n",
       "  'source': 'https://www.epa.gov/wqc/susp-sediment-concentrations'},\n",
       " {'term': 'Sediment fingerprinting',\n",
       "  'definition': 'A technique of identifying and apportioning the sources of sediment in a waterbody using various tracers such as stable isotopes, geochemical markers, and radionuclides, which can provide information about the sediment budget, erosion, and land-use impact.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/Sediment_fingerprinting'},\n",
       " {'term': 'Remote sensing',\n",
       "  'definition': 'A technology of acquiring and analyzing spatial and spectral information about the surface of the earth using various platforms such as satellite, airborne, and ground-based sensors, which can provide information about the land use, vegetation, topography, and other environmental parameters.',\n",
       "  'source': 'https://www.nrcs.usda.gov/wps/portal/nrcs/main/national/technical/rca/'},\n",
       " {'term': 'Dynamic process modeling',\n",
       "  'definition': 'A method of simulating and predicting the behavior and response of natural and human systems to environmental changes and perturbations using various mathematical and computational tools such as equations, models, and algorithms, which can provide information about the complex and interdependent processes occurring in the system.',\n",
       "  'source': 'https://en.wikipedia.org/wiki/System_dynamics'}]"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.loads(method_definitions.split(\"\\n\\n\")[1].split('```')[1].replace('\\n', ''))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:geokb]",
   "language": "python",
   "name": "conda-env-geokb-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}