API_documentation_UNIXshell/Search_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search API\n",
    "This notebook explains how to use the API to search the database using either search terms or a set of bibcodes. Lists of query parameters and search fields are also given. Examples here will be shown using `curl` commands from the shell command line,  with the use of the [requests](http://docs.python-requests.org/en/master/) library, though the same work can be done with Python using the `requests` library (see the \"API documentation - Python\" folder in this same repository) or the [unofficial Python ADS library](https://ads.readthedocs.io/en/latest/).\n",
    "\n",
    "The base URL (denoted as {base_url} in the examples below) for queries is `https://api.adsabs.harvard.edu/v1/search`. \n",
    "\n",
    "All queries to the API must include an HTTP header that specifies your API access token, e.g.\n",
    "\n",
    "`curl -H 'Authorization: Bearer <token>' 'https://api.adsabs.harvard.edu/v1/search/query?q=star'`\n",
    "\n",
    "In the examples below, {token_header} is used to denote `-H 'Authorization: Bearer <token>'`, where `<token>` is your personal API access token as described in the top-level [README](https://github.com/adsabs/adsabs-dev-api/blob/master/README.md). All examples use the default HTTP method GET, unless otherwise specified.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contents\n",
    "- Search API\n",
    "- Query parameters\n",
    "- Fields\n",
    "- Examples\n",
    "- Advanced search syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get search results\n",
    "`curl {token_header} {base_url}/query?q=search+term&fl=field1,field2`\n",
    "\n",
    "To search, construct a URL using the base URL given above, along with your search parameters as detailed in the Query Parameters section below. Multiple search parameters may be added to the URL; separate parameters using an ampersand (`&`). All text must be UTF-8 and URL encoded. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "author%3Amart%C3%ADnez+neutron+star\n"
     ]
    }
   ],
   "source": [
    "from urllib.parse import urlencode\n",
    "\n",
    "# accented letters, special characters, and spaces need to be encoded\n",
    "# Python has a package for this, or you can do it by hand\n",
    "query = {\"author\":\"martínez neutron star\"}\n",
    "encoded_query = urlencode(query)\n",
    "print(encoded_query)\n",
    "\n",
    "# note that the colon (:) may be encoded, depending on the algorithm you use. Your curl request \n",
    "# should accept either the unencoded colon (:) or the encoded version (%3A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response from your `curl` statement will be JSON encoded. \n",
    "\n",
    "Also, note that the search API uses the same syntax as [Apache Solr](http://lucene.apache.org/solr/). For a full reference of query possibilities, please refer to the Solr documentation and [ADS Search Help](https://ui.adsabs.harvard.edu/help/search/). Later sections in this notebook present useful parameters and patterns for the vast majority of use cases, but are not meant to be exhaustive.\n",
    "\n",
    "Note: to print the JSON response in a more readable manner, add `| python -m json.tool` to the end of your `curl` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"response\": {\n",
      "        \"docs\": [\n",
      "            {\n",
      "                \"author\": [\n",
      "                    \"P\\u00e9rez Rojas, Hugo\",\n",
      "                    \"Mart\\u00ednez, Aurora P\\u00e9rez\",\n",
      "                    \"Mesquita, Alexandre\",\n",
      "                    \"Razeira, Mois\\u00e9s\",\n",
      "                    \"Gomes, Rosana O.\",\n",
      "                    \"Vasconcellos, C\\u00e9sar A. Z.\"\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"numFound\": 168,\n",
      "        \"start\": 0\n",
      "    },\n",
      "    \"responseHeader\": {\n",
      "        \"QTime\": 743,\n",
      "        \"params\": {\n",
      "            \"fl\": \"author\",\n",
      "            \"q\": \"author:mart\\u00ednez neutron star\",\n",
      "            \"rows\": \"1\",\n",
      "            \"start\": \"0\",\n",
      "            \"wt\": \"json\",\n",
      "            \"x-amzn-trace-id\": \"Root=1-5b354e2a-a5b6ddca2051f3311a0beb8f\"\n",
      "        },\n",
      "        \"status\": 0\n",
      "    }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   398  100   398    0     0    398      0  0:00:01  0:00:01 --:--:--   366\r",
      "100   398  100   398    0     0    398      0  0:00:01  0:00:01 --:--:--   366\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=author%3Amart%C3%ADnez+neutron+star&fl=author&rows=1\" \\\n",
    "    | python -m json.tool"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse a query\n",
    "`curl {token_header} {base_url}/qtree?q=termORother+term` \n",
    "\n",
    "Use the syntax above to return a query tree, also known as an Abstract Syntax Tree (AST), as understood by our query parser. This can be useful if you want to modify and/or enhance queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":3,\"params\":{\"q\":\"author:martínez neutron star\",\"x-amzn-trace-id\":\"Root=1-5b2d4f71-c818c6a8dfcab0d0fc622eba\",\"fl\":\"id\",\"start\":\"0\",\"rows\":\"10\",\"wt\":\"json\"}},\"qtree\":\"\\n{\\\"name\\\":\\\"OPERATOR\\\", \\\"label\\\":\\\"DEFOP\\\", \\\"children\\\": [\\n    {\\\"name\\\":\\\"MODIFIER\\\", \\\"label\\\":\\\"MODIFIER\\\", \\\"children\\\": [\\n        {\\\"name\\\":\\\"TMODIFIER\\\", \\\"label\\\":\\\"TMODIFIER\\\", \\\"children\\\": [\\n            {\\\"name\\\":\\\"FIELD\\\", \\\"label\\\":\\\"FIELD\\\", \\\"children\\\": [\\n                {\\\"name\\\":\\\"TERM_NORMAL\\\", \\\"input\\\":\\\"author\\\", \\\"start\\\":0, \\\"end\\\":5},\\n                {\\\"name\\\":\\\"QNORMAL\\\", \\\"label\\\":\\\"QNORMAL\\\", \\\"children\\\": [\\n                    {\\\"name\\\":\\\"TERM_NORMAL\\\", \\\"input\\\":\\\"martínez\\\", \\\"start\\\":7, \\\"end\\\":14}]\\n                }]\\n            }]\\n        }]\\n    },\\n    {\\\"name\\\":\\\"MODIFIER\\\", \\\"label\\\":\\\"MODIFIER\\\", \\\"children\\\": [\\n        {\\\"name\\\":\\\"TMODIFIER\\\", \\\"label\\\":\\\"TMODIFIER\\\", \\\"children\\\": [\\n            {\\\"name\\\":\\\"FIELD\\\", \\\"label\\\":\\\"FIELD\\\", \\\"children\\\": [\\n                {\\\"name\\\":\\\"QNORMAL\\\", \\\"label\\\":\\\"QNORMAL\\\", \\\"children\\\": [\\n                    {\\\"name\\\":\\\"TERM_NORMAL\\\", \\\"input\\\":\\\"neutron\\\", \\\"start\\\":16, \\\"end\\\":22}]\\n                }]\\n            }]\\n        }]\\n    },\\n    {\\\"name\\\":\\\"MODIFIER\\\", \\\"label\\\":\\\"MODIFIER\\\", \\\"children\\\": [\\n        {\\\"name\\\":\\\"TMODIFIER\\\", \\\"label\\\":\\\"TMODIFIER\\\", \\\"children\\\": [\\n            {\\\"name\\\":\\\"FIELD\\\", \\\"label\\\":\\\"FIELD\\\", \\\"children\\\": [\\n                {\\\"name\\\":\\\"QNORMAL\\\", \\\"label\\\":\\\"QNORMAL\\\", \\\"children\\\": [\\n                    {\\\"name\\\":\\\"TERM_NORMAL\\\", \\\"input\\\":\\\"star\\\", \\\"start\\\":24, \\\"end\\\":27}]\\n                }]\\n            }]\\n        }]\\n    }]\\n}\"}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  1705  100  1705    0     0   1705      0  0:00:01 --:--:--  0:00:01  7478\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/qtree?q=author%3Amart%C3%ADnez+neutron+star\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Post a large identifier query\n",
    "`method = POST`\n",
    "\n",
    "`curl -H \"Content-Type: big-query/csv\" {token_header} {base_url}/bigquery?{urlparams} -X POST -d {payload}`\n",
    "\n",
    "This type of query, typically called a \"big query\" for short, returns standard search results, but instead of the typical search terms, accepts as its input a list of bibcodes. There is a maximum of 2000 bibcodes allowed for each big query request. There are also limits on how often you can make a big query request; typically, only 100 requests per day per user are allowed.\n",
    "\n",
    "Name          | Type   | Description \n",
    ":-------------|:------:|:--------------\n",
    "`urlparams`   |`string`| **Required**. Query parameters must be encoded as in a standard query. The query parameter `q=` must be included even if no additional searching is desired; to prevent it from restricting the search results, enter `q=*:*`. It's also recommended to use the `fl=` parameter to specify which fields to return.\n",
    "`payload`     |`string`| **Required**. Newline (`\\n`) separated list of bibcodes to return. The first line specifies the field that will be used for searching, i.e. `bibcode` (for example, `bibcode\\n1907AN....174...59.\\n1908PA.....16..445.\\n1989LNP...334..242S`)\n",
    "\n",
    "Notes:\n",
    "- Currently, we only allow searching in the `bibcode` field. You can submit both canonical bibcodes (the ones that appear on a paper's abstract page, or in the URL pointing to the abstract page) as well as alternate bibcodes (e.g. bibcodes that pointed to a previous version of a paper, such as the arXiv version). The search automatically recognizes both types.\n",
    "- The list of bibcodes supplied in the `payload` is *only applied after the main search*, specified with the `q=` parameter. To return the entire list of supplied bibcodes, use `q=*:*`.\n",
    "- Currently, the default number of results returned by the big query is not controlled by the number of bibcodes requested, but rather is set to the default value of 10. To adjust this, set `&rows=` either to the number of bibcodes requested or to the maximum value of 2000.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"responseHeader\":{\n",
      "    \"status\":0,\n",
      "    \"QTime\":42,\n",
      "    \"params\":{\n",
      "      \"q\":\"*:*\",\n",
      "      \"fl\":\"bibcode,title\",\n",
      "      \"start\":\"0\",\n",
      "      \"internal_logging_params\":\"X-Amzn-Trace-Id=Root=1-63dbdd56-3fc0c53447bf9d6f59f5c36f\",\n",
      "      \"fq\":\"{!bitset}\",\n",
      "      \"rows\":\"2000\",\n",
      "      \"wt\":\"json\"}},\n",
      "  \"response\":{\"numFound\":3,\"start\":0,\"docs\":[\n",
      "      {\n",
      "        \"bibcode\":\"1989LNP...334..242S\",\n",
      "        \"title\":[\"The Optical and Radio Properties of X-Ray Selected Bl-Lacertae Objects\"]},\n",
      "      {\n",
      "        \"bibcode\":\"1907AN....174...59.\",\n",
      "        \"title\":[\"Kleine Mitteilungen\"]},\n",
      "      {\n",
      "        \"bibcode\":\"1908PA.....16..445.\",\n",
      "        \"title\":[\"Variable Stars\"]}]\n",
      "  }}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   729  100   662  100    67   3229    326 --:--:-- --:--:-- --:--:--  3573\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "curl -H \"Content-Type: big-query/csv\" -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/bigquery?q=*:*&fl=bibcode,title&rows=2000\" \\\n",
    "    -X POST -d $'bibcode\\n1907AN....174...59.\\n1908PA.....16..445.\\n1989LNP...334..242S'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query Parameters\n",
    "All query parameters passed in the search URL must be UTF-8, URL-encoded strings. Due to the requirements of the authentication library used for validating requests, most non-ASCII characters that appear in the URL need to be encoded; for example, the double quote character (\") should be encoded as %22. In most programming languages, such as Python, the libraries used to retrieve content from web services will do this encoding for you; however, if you're using `curl` requests in the shell, such as shown in the examples in this notebook, you'll need to perform this encoding yourself.\n",
    "\n",
    "#### q\n",
    "**Required** The search query. This should be a UTF-8, URL-encoded string of <=1000 characters. `q` accepts both fields (`title:exoplanets`) and unfielded (`exoplanets`) searches.\n",
    "\n",
    "#### rows \n",
    "The number of results to return. The default is 10 and the maximum is 2000.\n",
    "\n",
    "#### start\n",
    "The starting point for returned results, used for pagination. The default is 0. To return the next page of results, set `start` equal to the value of `start` from the previous request, plus the number of results returned in the previous request. For the default values, set `start=10` to return the second page of results.\n",
    "\n",
    "#### fl\n",
    "The list of fields to return. The value should be a comma separated list of field names, e.g. `fl=bibcode,author,title`. The default is the document id (`fl=id`). A non-exhaustive list of available fields is shown below.\n",
    "\n",
    "#### fq\n",
    "Filters the list of search results. The syntax is the same as that for the `q` parameter. Adding search parameters via the `fq` parameter can speed up search results, as it searches only the results returned by the search entered via the `q` parameter, not the entire index. This parameter may be used more than once in a single search URL.\n",
    "\n",
    "#### sort\n",
    "The sorting field and direction to be used when returning results. The format requires both the field to sort on (see the list below and the direction, either `asc` or `desc` (ascending or descending). For example, an appropriately formatted sort parameter is `sort=citation_count+desc`. The default sort method is the relevancy score as calculated by the search engine. Other useful fields to sort on may be `date`, `read_count`, `first_author`, or `bibcode`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fields\n",
    "This is a non-exhaustive list of fields available for searching via the API. A more comprehensive list is available in our [help pages](https://ui.adsabs.harvard.edu/help/search/comprehensive-solr-term-list). These fields can be used with the `fl` and `sort` parameters. \n",
    "\n",
    "Note: some fields, such as body, can be searched but not returned via `fl` or sorted on. Also, multivalued fields, such as author, cannot be used for sorting.\n",
    "\n",
    "* `abstract` - the abstract of the record\n",
    "* `ack` - the acknowledgements section of an article\n",
    "* `aff` - an array of the authors' affiliations\n",
    "* `alternate_bibcode` - list of alternate bibcodes for a single record; if a published record also has an arXiv version, the bibcode associated with the arXiv version is here\n",
    "* `alternate_title` - list of alternate titles for a single record (usually if they are in multiple languages)\n",
    "* `arxiv_class` - the arXiv class the pre-print was submitted to\n",
    "* `author` - an array of the author names associated with the record\n",
    "* `bibcode` - the canonical ADS bibcode identifier for this record\n",
    "* `bibgroup` - the bibliographic groups that the bibcode has been associated with\n",
    "* `bibstem` - the abbreviated name of the journal or publication, e.g., *ApJ*.\n",
    "* `body`\\* - the full text content of the article\n",
    "* `citation_count` - number of citations the item has received\n",
    "* `copyright` - the copyright applied to the article\n",
    "* `data` - the list of sources that have data related to this bibcode\n",
    "* `date` - the machine-readable version of `pubdate`, useful for sorting\n",
    "* `database` - database the record is associated with (astronomy, physics, or general). By default, all three databases are searched; to limit to astronomy articles, add `fq=database:astronomy` to the URL\n",
    "* `doi`-  the digital object identifier of the article\n",
    "* `doctype` - the type of document it is (see [here](https://ui.adsabs.harvard.edu/help/search/search-syntax) for a list of doctypes)\n",
    "* `first_author` - the first author of the article\n",
    "* `grant` - the list of grant IDs and agencies noted within an article\n",
    "* `id` - a **non-persistent** unique integer for this record, used for fast look-up of a document\n",
    "* `identifier` - an array of alternative identifiers for the record. May contain alternate bibcodes, DOIs and/or arxiv ids.\n",
    "* `indexstamp` - time at which the document was (last) indexed\n",
    "* `issue` - issue the record appeared in\n",
    "* `keyword` - an array of normalized and un-normalized keyword values associated with the record\n",
    "* `lang`\\* - the language of the article's title\n",
    "* `orcid_pub` - ORCiD iDs supplied by publishers\n",
    "* `orcid_user` - ORCiD iDs supplied by knonwn users in the ADS\n",
    "* `orcid_other` - ORCiD iDs supplied by anonymous users in the ADS\n",
    "* `page` - starting page\n",
    "* `property` - an array of miscellaneous flags associated with the record (see [here](https://ui.adsabs.harvard.edu/help/search/search-syntax) for a list of properties\n",
    "* `pub` - the canonical name of the publication the record appeared in\n",
    "* `pubdate` - publication date in the form YYYY-MM-DD (DD value will always be \"00\"). This field is stored as a human-readable string, so is useful for being returned via `fl`, but not for sorting (see `date`)\n",
    "* `read_count` - number of times the record has been viewed within in a 90-day windows (ads and arxiv)\n",
    "* `title` - the title of the record\n",
    "* `vizier` - the subject tags given to the article by VizieR\n",
    "* `volume` - volume the record appeared in\n",
    "* `year` - the year the article was published\n",
    "\n",
    "\\* *These fields are only indexed and so are only searchable - they are not stored or returned by the search end point, if requested*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "Example `curl` requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":6,\"params\":{\"q\":\"bibcode:2011ApJ...737..103S\",\"x-amzn-trace-id\":\"Root=1-5b2d4f77-eb5609cc57124c307500906c\",\"fl\":\"title\",\"start\":\"0\",\"rows\":\"10\",\"wt\":\"json\"}},\"response\":{\"numFound\":1,\"start\":0,\"docs\":[{\"title\":[\"Measuring Reddening with Sloan Digital Sky Survey Stellar Spectra and Recalibrating SFD\"]}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   344  100   344    0     0    344      0  0:00:01 --:--:--  0:00:01  1702\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# search by bibcode, return the title\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=bibcode:2011ApJ...737..103S&fl=title\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":977,\"params\":{\"q\":\"black holes\",\"x-amzn-trace-id\":\"Root=1-5b2d4f78-87a695a096d4ece4e36e76c6\",\"fl\":\"bibcode\",\"start\":\"0\",\"fq\":\"database:astronomy\",\"rows\":\"10\",\"wt\":\"json\"}},\"response\":{\"numFound\":60233,\"start\":0,\"docs\":[{\"bibcode\":\"2004IAUS..222..115T\"},{\"bibcode\":\"2006kbls.confE..15H\"},{\"bibcode\":\"1997PKAS...12..145P\"},{\"bibcode\":\"1992ianp.symp..225S\"},{\"bibcode\":\"2017APS..APR.E5009F\"},{\"bibcode\":\"2017nova.pres.2298K\"},{\"bibcode\":\"1986Sci...234..882T\"},{\"bibcode\":\"2002PhRvD..66f3505B\"},{\"bibcode\":\"1996PhRvL..77.5164K\"},{\"bibcode\":\"2017ApJ...840L..24F\"}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   600  100   600    0     0    600      0  0:00:01  0:00:01 --:--:--   528\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# search in all metadata fields for \"black holes\", restricted to the astronomy database; return a list of bibcodes\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=black+holes&fq=database:astronomy&fl=bibcode\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":223,\"params\":{\"q\":\"\\\"dark energy\\\"\",\"x-amzn-trace-id\":\"Root=1-5b2d4f7a-026b88865a0439add4cb3005\",\"fl\":\"bibcode,citation_count\",\"start\":\"0\",\"sort\":\"citation_count desc\",\"fq\":\"author:\\\"Spergel, D\\\"\",\"rows\":\"10\",\"wt\":\"json\"}},\"response\":{\"numFound\":62,\"start\":0,\"docs\":[{\"citation_count\":8728,\"bibcode\":\"2003ApJS..148..175S\"},{\"citation_count\":6436,\"bibcode\":\"2007ApJS..170..377S\"},{\"citation_count\":6397,\"bibcode\":\"2011ApJS..192...18K\"},{\"citation_count\":4484,\"bibcode\":\"2009ApJS..180..330K\"},{\"citation_count\":3976,\"bibcode\":\"2003ApJS..148....1B\"},{\"citation_count\":1581,\"bibcode\":\"2009ApJS..180..306D\"},{\"citation_count\":1383,\"bibcode\":\"2009ApJS..180..225H\"},{\"citation_count\":1209,\"bibcode\":\"2010MNRAS.401.2148P\"},{\"citation_count\":478,\"bibcode\":\"2010MNRAS.404...60R\"},{\"citation_count\":437,\"bibcode\":\"2011ApJS..192...17B\"}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   866  100   866    0     0    866      0  0:00:01 --:--:--  0:00:01  2086\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# search in all metadata fields for \"dark energy\", filter by author, return a list of bibcodes and citation counts, \n",
    "# sorted by citation count (the double quotes are encoded as %22)\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=%22dark+energy%22&fq=author:%22Spergel,+D%22&sort=citation_count+desc&fl=bibcode,citation_count\"\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":68,\"params\":{\"q\":\"author:\\\"Kurtz, M\\\"\",\"x-amzn-trace-id\":\"Root=1-5b2d4f7b-beaf5e3c449be466ef2fb630\",\"fl\":\"title\",\"start\":\"0\",\"fq\":\"property:refereed\",\"rows\":\"10\",\"wt\":\"json\"}},\"response\":{\"numFound\":116,\"start\":0,\"docs\":[{\"title\":[\"Correlated radon and CO<SUB>2</SUB> variations near the San Andreas Fault\"]},{\"title\":[\"Methanol synthesis over ZnO: A structure-sensitive reaction?\"]},{\"title\":[\"Effects of Hyperbaric Hypocapnic Ventilation with One Hundred Per Cent Oxygen and Tham on Anoxia by Tracheal Occlusion *\"]},{\"title\":[\"Multiple pass laser interferometer\"]},{\"title\":[\"Electron impact ionization in the vicinity of comets\"]},{\"title\":[\"Measuring the Dark Matter Scale of Local Group Dwarf Spheroidals\"]},{\"title\":[\"Zaragozic acids: a family of fungal metabolites that are picomolar competitive inhibitors of squalene synthase.\"]},{\"title\":[\"A cognitive system for astronomical image interpretation\"]},{\"title\":[\"Sphingoid Base 1-phosphate Phosphatase: A Key Regulator of Sphingolipid Metabolism and Stress Response\"]},{\"title\":[\"Effect of Experience on Retention and Elimination of Misconceptions about Molecular Structure and Bonding\"]}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  1189  100  1189    0     0   1189      0  0:00:01 --:--:--  0:00:01  3898\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# limit a search to only refereed articles\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=author:%22Kurtz,+M%22&fq=property:refereed&fl=title\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":137,\"params\":{\"q\":\"\\\"transiting exoplanets\\\" \",\"x-amzn-trace-id\":\"Root=1-5b2d4f7c-4920a2362a8bb94865f5483c\",\"fl\":\"bibcode\",\"start\":\"0\",\"sort\":\"bibcode desc\",\"rows\":\"5\",\"wt\":\"json\"}},\"response\":{\"numFound\":1631,\"start\":0,\"docs\":[{\"bibcode\":\"2018sptz.prop14131M\"},{\"bibcode\":\"2018sptz.prop14059B\"},{\"bibcode\":\"2018sptz.prop14047D\"},{\"bibcode\":\"2018Sci...359.1453C\"},{\"bibcode\":\"2018RNAAS...2b..40M\"}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   439  100   439    0     0    439      0  0:00:01 --:--:--  0:00:01   993\r",
      "100   439  100   439    0     0    439      0  0:00:01 --:--:--  0:00:01   993\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# search for the phrase \"transiting exoplanets\", fetch 5 rows\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=%22transiting+exoplanets%22+&rows=5&fl=bibcode&sort=bibcode+desc\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":142,\"params\":{\"q\":\"\\\"transiting exoplanets\\\"\",\"x-amzn-trace-id\":\"Root=1-5b2d503c-893f6dcad5341f72c5f8fa5a\",\"fl\":\"bibcode\",\"start\":\"5\",\"sort\":\"bibcode desc\",\"rows\":\"5\",\"wt\":\"json\"}},\"response\":{\"numFound\":1631,\"start\":5,\"docs\":[{\"bibcode\":\"2018RNAAS...2b..20E\"},{\"bibcode\":\"2018RNAAS...2a..47J\"},{\"bibcode\":\"2018RNAAS...2a..28L\"},{\"bibcode\":\"2018PASP..130g4401B\"},{\"bibcode\":\"2018PASP..130f4502T\"}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   438  100   438    0     0    438      0  0:00:01 --:--:--  0:00:01  1575\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# same, but return the next 5 rows\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=%22transiting+exoplanets%22&rows=5&start=5&fl=bibcode&sort=bibcode+desc\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Search Syntax\n",
    "The `q` parameter supports both fielded and unfielded searches:\n",
    "- `black holes`\n",
    "- `title:exoplanets`\n",
    "\n",
    "Use quotation marks to indicate phrase searching (remember to encode quotes and spaces in `curl` statements; double quotes are encoded as %22, while spaces can be encoded as `+`)\n",
    "- `\"black holes\"`\n",
    "- `title:\"dark energy\"`\n",
    "\n",
    "Prepend terms with `+` or `-` to indicate inclusion or exclusion (note that after encoding spaces in the search string, the URL may include `++` or `+-`):\n",
    "- `\"transiting exoplanets\" -Kepler`\n",
    "- `\"transiting exoplanets\" +JWST`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"responseHeader\":{\"status\":0,\"QTime\":120,\"params\":{\"q\":\"\\\"transiting exoplanets\\\"  JWST\",\"x-amzn-trace-id\":\"Root=1-5b2d4f81-06c03436981012dfca638861\",\"fl\":\"title\",\"start\":\"0\",\"rows\":\"3\",\"wt\":\"json\"}},\"response\":{\"numFound\":150,\"start\":0,\"docs\":[{\"title\":[\"Transiting Exoplanets with JWST\"]},{\"title\":[\"Spectroscopy of transiting exoplanets: From Spitzer to JWST\"]},{\"title\":[\"Characterizing Transiting Exoplanet Atmospheres with JWST\"]}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   441  100   441    0     0    441      0  0:00:01 --:--:--  0:00:01  1373\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    "    \"https://api.adsabs.harvard.edu/v1/search/query?q=%22transiting+exoplanets%22++JWST&fl=title&rows=3\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To filter by a publication date range, you can use either the `year` or the `pubdate` fields:\n",
    "- `pubdate:[2013-07-00 TO *]`\n",
    "- `pubdate:[2005-01 TO 2007-01]`\n",
    "- `pubdate:2013-02`\n",
    "- `year:2013`\n",
    "- `year:[2012 TO 2013]`\n",
    "\n",
    "The default search uses a boolean `AND` between terms, but you may use `OR` and `AND` in combination with `()` to create more complex queries.\n",
    "\n",
    "Prefix queries (wildcards, `*`) are supported for most fields."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Highlighting search terms\n",
    "Snippets of text, with search terms highlighted, may be returned by a search query. Limits are in place to prevent scraping of our full-text holdings, as noted below, but more highlights can be returned via these API calls than are displayed in the search results on the website.\n",
    "\n",
    "The following parameters control the highlights feature:\n",
    "- `hl=true` - (required) turn on the highlighting feature\n",
    "- `hl.fl` - (required) a comma-separated list of fields to return highlights for (e.g. title, abstract, or body)\n",
    "- `hl.snippets` - number of highlighted snippets to return; the maximum is 4\n",
    "- `hl.fragsize` - length of snippet to return; the maximum length is 100 characters. The snippet positioning is optimized to break on word and sentence boundaries, so the highlighted term will not always be centered in the snippet\n",
    "- `hl.maxAnalyzedChars` - this parameter is only needed if the highlighted field is very long and the frequency of the search term is low. By default, the first 51,200 characters of a field are used for highlights. If the search is returning fewer snippets than expected, try increasing this value to capture more of the field. However, increasing this value unnecessarily will slow searching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading PDFs\n",
    "PDFs may be downloaded using a GET request to the `link_gateway` endpoint. The URL to query consists of the base URL `https://ui.adsabs.harvard.edu/link_gateway/`, the bibcode to retrieve the PDF for, and an endpoint to indicate whether to download the arXiv PDF (`/EPRINT_PDF`) or the publisher PDF (`PUB_PDF`). In addition to the usual header that contains your API token, the following flags will also need to be set:\n",
    "* `-L` - this flag follows the redirect URL to the arXiv or to the publisher's site\n",
    "* `-o /full/path/to/output.pdf` - this flag and the required path parameter that follows will download the PDF to the specified location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r",
      "100   281  100   281    0     0    281      0  0:00:01  0:00:01 --:--:--   163\n",
      "\r",
      "100   305  100   305    0     0    305      0  0:00:01  0:00:01 --:--:--   305\n",
      "\r",
      "100   286  100   286    0     0    143      0  0:00:02  0:00:02 --:--:--   143\n",
      "\r",
      "  0  553k    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0\r",
      " 37  553k   37  208k    0     0  70997      0  0:00:07  0:00:03  0:00:04 1024k\r",
      "100  553k  100  553k    0     0   184k      0  0:00:03  0:00:03 --:--:-- 1641k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# download the arXiv PDF\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    " 'https://ui.adsabs.harvard.edu/link_gateway/1999ApJ...517..565P/EPRINT_PDF' \\\n",
    " -L -o '/full/path/to/output.pdf'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   295  100   295    0     0    295      0  0:00:01 --:--:--  0:00:01  1666\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   318    0   318    0     0    318      0 --:--:-- --:--:-- --:--:--   521\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r",
      "100  184k    0  184k    0     0  94585      0 --:--:--  0:00:02 --:--:--  173k\r",
      "100 2258k    0 2258k    0     0   752k      0 --:--:--  0:00:03 --:--:-- 1295k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "token=\"your-token-here\"\n",
    "# download the publisher PDF\n",
    "curl -H \"Authorization: Bearer $token\" \\\n",
    " 'https://ui.adsabs.harvard.edu/link_gateway/1998AJ....116.1009R/PUB_PDF' \\\n",
    " -L -o '/full/path/to/output.pdf'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}