cleanBib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "# Clean Bibliography\n",
    "\n",
    "To goal of this notebook is to clean your `.bib` file to ensure that it only contains the full first names of references that you have cited in your paper. The full first names will then be used to query the probabilistic gender classifier, [Gender API](https://gender-api.com). \n",
    "\n",
    "The only required file you need is your manuscript's bibliography in `.bib` format. __Your `.bib` must only contain references cited in the manuscript__. Otherwise, the estimated gender proportions will be inaccurate. \n",
    "\n",
    "If you are not using LaTeX, collect and organize only the references you have cited in your manuscript using your reference manager of choice (e.g. Mendeley, Zotero, EndNote, ReadCube, etc.) and export that selected bibliography as a `.bib` file. __Please export your .bib in an output style that uses full first names (rather than only first initials) and using the full author lists (rather than abbreviated author lists with \"et al.\").__\n",
    "\n",
    "   * [Export `.bib` from Mendeley](https://blog.mendeley.com/2011/10/25/howto-use-mendeley-to-create-citations-using-latex-and-bibtex/)\n",
    "   * [Export `.bib` from Zotero](https://libguides.mit.edu/ld.php?content_id=34248570)\n",
    "   * [Export `.bib` from EndNote](https://www.reed.edu/cis/help/LaTeX/EndNote.html). Note: Please export full first names by either [choosing an output style that does so by default (e.g. in MLA style)](https://canterbury.libguides.com/endnote/basics-output) or by [customizing an output style.](http://bibliotek.usn.no/cite-and-write/endnote/how-to-use/how-to-show-the-author-s-full-name-in-the-reference-list-article185897-28181.html)\n",
    "   * [Export `.bib` from Read Cube Papers](https://support.papersapp.com/support/solutions/articles/30000024634-how-can-i-export-references-from-readcube-papers-)\n",
    "\n",
    "For those working in LaTeX, we can use an optional `.aux` file to automatically filter your `.bib` to check that it only contains entries which are cited in your manuscript.\n",
    "\n",
    "| Input                 | Output                                                                                                                        |\n",
    "|-----------------------|-------------------------------------------------------------------------------------------------------------------------------|\n",
    "| `.bib` file(s)**(REQUIRED)**    | `cleanBib.csv`: table of author first names, titles, and .bib keys                                                            |\n",
    "| `.aux` file (OPTIONAL)| `Authors.csv`: table of author first names, estimated gender classification, and confidence                                   |\n",
    "| `.tex` file (OPTIONAL)| `yourTexFile_gendercolor.tex`: your `.tex` file modified to compile .pdf with in-line citations colored-coded by gender pairs |\n",
    "\n",
    "## Import libraries, set paths, check settings\n",
    "\n",
    "### Upload your `.bib` file(s) and optionally an `.aux` file generated from compiling your LaTeX manuscript and your `.tex` file\n",
    "\n",
    "![upload button](img/upload.png)\n",
    "\n",
    "![confirm upload button](img/confirmUpload.png)\n",
    "\n",
    "Then, run the code block below. (click to select the block and then press Ctrl+Enter; or click the block and press the Run button in the top menubar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "Python 3"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import bibtexparser\n",
    "from bibtexparser.bparser import BibTexParser\n",
    "import glob\n",
    "import subprocess\n",
    "import os\n",
    "from pybtex.database.input import bibtex\n",
    "import csv\n",
    "from pylatexenc.latex2text import LatexNodes2Text \n",
    "import unicodedata\n",
    "import re\n",
    "import pandas as pd\n",
    "from habanero import Crossref\n",
    "import string\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "def checkcites_output(aux_file):\n",
    "    '''take in aux file for tex document, return list of citation keys\n",
    "    that are in .bib file but not in document'''\n",
    "\n",
    "    result = subprocess.run(['texlua', 'checkcites.lua', aux_file[0]], stdout=subprocess.PIPE)\n",
    "    result = result.stdout.decode('utf-8')\n",
    "    unused_array_raw = result.split('\\n')\n",
    "    # process array of unused references + other output \n",
    "    unused_array_final = list()\n",
    "    for x in unused_array_raw:\n",
    "        if len(x) > 0: # if line is not empty\n",
    "            if x[0] == '-':  # and if first character is a '-', it's a citation key\n",
    "                unused_array_final.append(x[2:]) # truncate '- '            \n",
    "    if \"------------------------------------------------------------------------\" in unused_array_final:\n",
    "        return(result)\n",
    "    else:\n",
    "        return(unused_array_final)\n",
    "\n",
    "\n",
    "def removeMiddleName(line):\n",
    "    arr = line.split()\n",
    "    last = arr.pop()\n",
    "    n = len(arr)\n",
    "    if n == 4:\n",
    "        first, middle = ' '.join(arr[:2]), ' '.join(arr[2:])\n",
    "    elif n == 3:\n",
    "        first, middle = arr[0], ' '.join(arr[1:])\n",
    "    elif n == 2:\n",
    "        first, middle = arr\n",
    "    elif n==1:\n",
    "        return line\n",
    "    return(str(first + ' ' + middle))\n",
    "\n",
    "\n",
    "def returnFirstName(line):\n",
    "    arr = line.split()\n",
    "    n = len(arr)\n",
    "    if n == 4:\n",
    "        first, middle = ' '.join(arr[:2]), ' '.join(arr[2:])\n",
    "    elif n == 3:\n",
    "        first, middle = arr[0], ' '.join(arr[1:])\n",
    "    elif n == 2:\n",
    "        first, middle = arr\n",
    "    elif n==1:\n",
    "        return line\n",
    "    return(str(middle))\n",
    "\n",
    "\n",
    "def convertLatexSpecialChars(latex_text):\n",
    "    return LatexNodes2Text().latex_to_text(latex_text)\n",
    "\n",
    "\n",
    "def convertSpecialCharsToUTF8(text):\n",
    "    data = LatexNodes2Text().latex_to_text(text)\n",
    "    return unicodedata.normalize('NFD', data).encode('ascii', 'ignore').decode('utf-8')\n",
    "\n",
    "\n",
    "def namesFromXref(doi, title, authorPos):\n",
    "    '''Use DOI and article titles to query Crossref for author list'''\n",
    "    if authorPos == 'first':\n",
    "        idx = 0\n",
    "    elif authorPos == 'last':\n",
    "        idx = -1\n",
    "    # get cross ref data\n",
    "    authors = ['']\n",
    "    # first try DOI\n",
    "    if doi != \"\":\n",
    "        works = cr.works(query = title, select = [\"DOI\",\"author\"], limit=1, filter = {'doi': doi})\n",
    "        if works['message']['total-results'] > 0:\n",
    "            authors = works['message']['items'][0]['author']\n",
    "    elif title != '': \n",
    "        works = cr.works(query = f'title:\"{title}\"', select = [\"title\",\"author\"], limit=10)\n",
    "        cnt = 0\n",
    "        name = ''\n",
    "        # check that you grabbed the proper paper\n",
    "        if works['message']['items'][cnt]['title'][0].lower() == title.lower():\n",
    "            authors = works['message']['items'][0]['author']\n",
    "\n",
    "    # check the all fields are available\n",
    "    if not 'given' in authors[idx]:\n",
    "        name = ''\n",
    "    else:\n",
    "        # trim initials\n",
    "        name = authors[idx]['given'].replace('.',' ').split()[0]\n",
    "\n",
    "    return name\n",
    "\n",
    "\n",
    "cr = Crossref()\n",
    "homedir = '/home/jovyan/'\n",
    "bib_files = glob.glob(homedir + '*.bib')\n",
    "paper_aux_file = glob.glob(homedir + '*.aux')\n",
    "paper_bib_file = 'library_paper.bib'\n",
    "try:\n",
    "    tex_file = glob.glob(homedir + \"*.tex\")[0]\n",
    "except:\n",
    "    print('No optional .tex file found.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### Define the _first_ and _last_ author of your paper.\n",
    "\n",
    "For example: \n",
    "```\n",
    "yourFirstAuthor = 'Teich, Erin G.'\n",
    "yourLastAuthor = 'Bassett, Danielle S.'\n",
    "```\n",
    "\n",
    "And optionally, define any co-first or co-last author(s), making sure to keep the square brackets to define a list.\n",
    "\n",
    "For example:\n",
    "```\n",
    "optionalEqualContributors = ['Dworkin, Jordan', 'Stiso, Jennifer']\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "```\n",
    "optionalEqualContributors = ['Dworkin, Jordan']\n",
    "```\n",
    "\n",
    "Then, run the code block below. (click to select the block and then press Ctrl+Enter; or click the block and press the Run button in the top menubar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "Python 3"
   },
   "outputs": [],
   "source": [
    "yourFirstAuthor = 'LastName, FirstName OptionalMiddleInitial'\n",
    "yourLastAuthor = 'LastName, FirstName OptionalMiddleInitial'\n",
    "optionalEqualContributors = ['LastName, FirstName OptionalMiddleInitial', 'LastName, FirstName OptionalMiddleInitial']\n",
    "\n",
    "if (yourFirstAuthor == 'LastName, FirstName OptionalMiddleInitial') or (yourLastAuthor == 'LastName, FirstName OptionalMiddleInitial'):\n",
    "    raise ValueError(\"Please enter your manuscript's first and last author names\")\n",
    "\n",
    "if paper_aux_file:\n",
    "    if optionalEqualContributors == ['LastName, FirstName OptionalMiddleInitial', 'LastName, FirstName OptionalMiddleInitial']:\n",
    "        citing_authors = np.array([yourFirstAuthor, yourLastAuthor])\n",
    "    else:\n",
    "        citing_authors = np.array([yourFirstAuthor, yourLastAuthor, optionalEqualContributors])\n",
    "    print(checkcites_output(paper_aux_file))\n",
    "    unused_in_paper = checkcites_output(paper_aux_file) # get citations in library not used in paper\n",
    "    print(\"Unused citations: \", unused_in_paper.count('=>'))\n",
    "    \n",
    "    \n",
    "    parser = BibTexParser()\n",
    "    parser.ignore_nonstandard_types = False\n",
    "    parser.common_strings = True\n",
    "    \n",
    "    bib_data = None\n",
    "    for bib_file in bib_files:\n",
    "        with open(bib_file) as bibtex_file:\n",
    "            if bib_data is None:\n",
    "                bib_data = bibtexparser.bparser.BibTexParser(common_strings=True, ignore_nonstandard_types=False).parse_file(bibtex_file)\n",
    "            else:\n",
    "                bib_data_extra = bibtexparser.bparser.BibTexParser(common_strings=True, ignore_nonstandard_types=False).parse_file(bibtex_file)\n",
    "                bib_data.entries_dict.update(bib_data_extra.entries_dict)\n",
    "                bib_data.entries.extend(bib_data_extra.entries)\n",
    "    \n",
    "    all_library_citations = list(bib_data.entries_dict.keys())\n",
    "    print(\"All citations: \", len(all_library_citations))\n",
    "    \n",
    "    for k in all_library_citations:\n",
    "        if re.search('\\\\b'+ k + '\\\\b', unused_in_paper.replace('\\n',' ').replace('=>',' ')) != None:\n",
    "            del bib_data.entries_dict[k] # remove from entries dictionary if not in paper\n",
    "            \n",
    "    in_paper_mask = [re.search('\\\\b'+ bib_data.entries[x]['ID'] + '\\\\b', unused_in_paper.replace('\\n',' ').replace('=>',' ')) == None for x in range(len(bib_data.entries))]\n",
    "    bib_data.entries = [bib_data.entries[x] for x in np.where(in_paper_mask)[0]] # replace entries list with entries only in paper\n",
    "    del bib_data.comments\n",
    "    \n",
    "    duplicates = []\n",
    "    for key in bib_data.entries_dict.keys():\n",
    "        count = str(bib_data.entries).count(\"'ID\\': \\'\"+ key + \"\\'\")\n",
    "        if count > 1:\n",
    "            duplicates.append(key)\n",
    "            \n",
    "    if len(duplicates) > 0:\n",
    "        raise ValueError(\"In your .bib file, please remove duplicate entries or duplicate entry ID keys for:\", ' '.join(map(str, duplicates)))\n",
    "\n",
    "    if os.path.exists(paper_bib_file):\n",
    "        os.remove(paper_bib_file)\n",
    "    \n",
    "    with open(paper_bib_file, 'w') as bibtex_file:\n",
    "        bibtexparser.dump(bib_data, bibtex_file)\n",
    "    \n",
    "    # define first author and last author names of citing paper -- will exclude citations of these authors\n",
    "    # beware of latex symbols within author names\n",
    "    # in_paper_citations = list(bib_data.entries_dict.keys())\n",
    "    in_paper_citations = [bib_data.entries[x]['ID'] for x in range(len(bib_data.entries))] # get list of citation keys in paper\n",
    "    \n",
    "    # extract author list for every cited paper\n",
    "    cited_authors = [bib_data.entries_dict[x]['author'] for x in in_paper_citations]\n",
    "    # find citing authors in cited author list\n",
    "    # using nested list comprehension, make a citing author -by- citation array of inclusion\n",
    "    self_cite_mask = np.array([[str(citing_author) in authors for authors in cited_authors] for citing_author in citing_authors])\n",
    "    self_cite_mask = np.any(self_cite_mask,axis=0) # collapse across citing authors such that any coauthorship by either citing author -> exclusion\n",
    "    \n",
    "    print(\"Self-citations: \", [bib_data.entries[x]['ID'] for x in np.where(self_cite_mask)[0]]) # print self citations\n",
    "    for idx,k in enumerate(in_paper_citations):\n",
    "        if self_cite_mask[idx]:\n",
    "            del bib_data.entries_dict[k] # delete citation from dictionary if self citationi\n",
    "    bib_data.entries = [bib_data.entries[x] for x in np.where(np.invert(self_cite_mask))[0]] # replace entries list with entries that aren't self citations\n",
    "    \n",
    "    paper_bib_file_excl_sc = os.path.splitext(paper_bib_file)[0] + '_noselfcite.bib'\n",
    "    \n",
    "    if os.path.exists(paper_bib_file_excl_sc):\n",
    "        os.remove(paper_bib_file_excl_sc)\n",
    "    \n",
    "    with open(paper_bib_file_excl_sc, 'w') as bibtex_file:\n",
    "        bibtexparser.dump(bib_data, bibtex_file)\n",
    "        \n",
    "if os.path.exists('*_noselfcite.bib'):\n",
    "    ID = glob.glob(homedir + paper_bib_file_excl_sc)\n",
    "else:\n",
    "    ID = glob.glob(homedir + '*bib')\n",
    "    with open(ID[0]) as bibtex_file:\n",
    "        bib_data = bibtexparser.bparser.BibTexParser(common_strings=True, ignore_nonstandard_types=False).parse_file(bibtex_file)\n",
    "    duplicates = []\n",
    "    for key in bib_data.entries_dict.keys():\n",
    "        count = str(bib_data.entries).count(\"'ID\\': \\'\"+ key + \"\\'\")\n",
    "        if count > 1:\n",
    "            duplicates.append(key)\n",
    "            \n",
    "    if len(duplicates) > 0:\n",
    "        raise ValueError(\"In your .bib file, please remove duplicate entries or duplicate entry ID keys for:\", ' '.join(map(str, duplicates)))\n",
    "\n",
    "FA = []\n",
    "LA = []\n",
    "parser = bibtex.Parser()\n",
    "bib_data = parser.parse_file(ID[0])\n",
    "counter = 1\n",
    "nameCount = 0\n",
    "outPath = homedir + 'cleanedBib.csv'\n",
    "\n",
    "if os.path.exists(outPath):\n",
    "    os.remove(outPath)\n",
    "\n",
    "with open(outPath, 'w', newline='') as csvfile:\n",
    "    writer = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)\n",
    "    writer.writerow(['Article', 'FA', 'LA', 'Title', 'SelfCite', 'CitationKey'])\n",
    "\n",
    "for key in bib_data.entries.keys():\n",
    "    diversity_bib_titles = ['The extent and drivers of gender imbalance in neuroscience reference lists','The gender citation gap in international relations','Quantitative evaluation of gender bias in astronomical publications from citation counts', '\\# CommunicationSoWhite', '{Just Ideas? The Status and Future of Publication Ethics in Philosophy: A White Paper}','Gendered citation patterns across political science and social science methodology fields','Gender Diversity Statement and Code Notebook v1.0']\n",
    "    if bib_data.entries[key].fields['title'] in diversity_bib_titles:\n",
    "        continue\n",
    "        \n",
    "    try:\n",
    "        author = bib_data.entries[key].persons['author']\n",
    "    except:\n",
    "        author = bib_data.entries[key].persons['editor']\n",
    "    FA = author[0].rich_first_names\n",
    "    LA = author[-1].rich_first_names\n",
    "    FA = convertLatexSpecialChars(str(FA)[7:-3]).translate(str.maketrans('', '', string.punctuation)).replace('Protected',\"\").replace(\" \",'')\n",
    "    LA = convertLatexSpecialChars(str(LA)[7:-3]).translate(str.maketrans('', '', string.punctuation)).replace('Protected',\"\").replace(\" \",'')\n",
    "\n",
    "    # check that we got a name (not an initial) from the bib file, if not try using the title in the crossref API\n",
    "    try:\n",
    "        title = bib_data.entries[key].fields['title'].replace(',', '').replace(',', '').replace('{','').replace('}','')\n",
    "    except:\n",
    "        title = ''\n",
    "    try:\n",
    "        doi =  bib_data.entries[key].fields['doi']\n",
    "    except:\n",
    "        doi = ''\n",
    "    if FA == '' or len(FA.split('.')[0]) <= 1:\n",
    "        while True:\n",
    "            try:\n",
    "                FA = namesFromXref(doi, title, 'first')\n",
    "            except UnboundLocalError:\n",
    "                sleep(1)\n",
    "                continue\n",
    "            break\n",
    "    if LA == '' or len(LA.split('.')[0]) <= 1:\n",
    "        while True:\n",
    "            try:\n",
    "                LA = namesFromXref(doi, title, 'last')\n",
    "            except UnboundLocalError:\n",
    "                sleep(1)\n",
    "                continue\n",
    "            break\n",
    "\n",
    "    if (yourFirstAuthor!='LastName, FirstName OptionalMiddleInitial') and (yourLastAuthor!='LastName, FirstName OptionalMiddleInitial'):\n",
    "        selfCiteCheck1 = [s for s in author if removeMiddleName(yourLastAuthor) in str([convertLatexSpecialChars(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertLatexSpecialChars(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "        selfCiteCheck1a = [s for s in author if removeMiddleName(yourLastAuthor) in str([convertSpecialCharsToUTF8(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertSpecialCharsToUTF8(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "        selfCiteCheck1b = [s for s in author if removeMiddleName(yourLastAuthor) in str([convertSpecialCharsToUTF8(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), LA]).replace(\"'\", \"\")]\n",
    "\n",
    "        selfCiteCheck2 = [s for s in author if removeMiddleName(yourFirstAuthor) in str([convertLatexSpecialChars(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertLatexSpecialChars(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "        selfCiteCheck2a = [s for s in author if removeMiddleName(yourFirstAuthor) in str([convertSpecialCharsToUTF8(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertSpecialCharsToUTF8(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "        selfCiteCheck2b = [s for s in author if removeMiddleName(yourFirstAuthor) in str([convertSpecialCharsToUTF8(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), FA]).replace(\"'\", \"\")]\n",
    "\n",
    "        nameCount = 0\n",
    "        if optionalEqualContributors != ('LastName, FirstName OptionalMiddleInitial', 'LastName, FirstName OptionalMiddleInitial'):\n",
    "            for name in optionalEqualContributors:\n",
    "                selfCiteCheck3 = [s for s in author if removeMiddleName(name) in str([convertLatexSpecialChars(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertLatexSpecialChars(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "                selfCiteCheck3a = [s for s in author if removeMiddleName(name) in str([convertSpecialCharsToUTF8(str(s.rich_last_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\"), convertSpecialCharsToUTF8(str(s.rich_first_names)[7:-3]).replace(\"', Protected('\",\"\").replace(\"'), '\", \"\")]).replace(\"'\", \"\")]\n",
    "                if len(selfCiteCheck3)>0:\n",
    "                    nameCount += 1\n",
    "                if len(selfCiteCheck3a)>0:\n",
    "                    nameCount += 1\n",
    "        selfCiteChecks = [selfCiteCheck1, selfCiteCheck1a, selfCiteCheck1b, selfCiteCheck2, selfCiteCheck2a, selfCiteCheck2b]\n",
    "        if sum([len(check) for check in selfCiteChecks]) + nameCount > 0:\n",
    "            selfCite = 'Y'\n",
    "            if len(FA) < 2:\n",
    "                print(str(counter) + \": \" + key + \"\\t\\t  <-- self-citation <--  ***NAME MISSING OR POSSIBLY INCOMPLETE***\")\n",
    "            else:\n",
    "                print(str(counter) + \": \" + key + \"  <-- self-citation\")\n",
    "        else:\n",
    "            selfCite= 'N'\n",
    "            if len(FA) < 2:\n",
    "                print(str(counter) + \": \" + key + \"\\t\\t  <--  ***NAME MISSING OR POSSIBLY INCOMPLETE***\")\n",
    "            else:\n",
    "                print(str(counter) + \": \" + key)\n",
    "    else:\n",
    "        selfCite = 'NA'\n",
    "        \n",
    "    with open(outPath, 'a', newline='') as csvfile:\n",
    "        writer = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)\n",
    "        if selfCite=='N':\n",
    "            writer.writerow([counter, convertSpecialCharsToUTF8(FA), convertSpecialCharsToUTF8(LA), title, selfCite, key])\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## Estimate gender of authors from cleaned bibliography\n",
    "\n",
    "### Checkpoint for cleaned bibliography and using Gender API to estimate genders by first names\n",
    "After registering for a [gender-api](https://gender-api.com/) (free account available), use your 500 free monthly search credits by __pasting your API key in the code for the line indicated below__ (replace only YOUR ACCOUNT KEY HERE):\n",
    "\n",
    "```genderAPI_key <- '&key=YOUR ACCOUNT KEY HERE'```\n",
    "\n",
    "[You can find your key in your account's profile page.](https://gender-api.com/en/account/overview#my-api-key)\n",
    "\n",
    "__NOTE__: If any of your cleanBib.csv entries are incomplete or contain first initials, the code will not continue to the stage that will use your limited free credits. \n",
    "\n",
    "Please manually edit the cleanedBib.csv by downloading the file, modifying it, and re-uploading it. Alternatively, you can edit the file directly within the Binder environment by clicking the `Edit` button, making modifications, and saving the file.\n",
    "\n",
    "![edit button](img/manualEdit.png)\n",
    "\n",
    "Common issues include: \n",
    "\n",
    "* Bibliography entry did not include a last author because the author list was truncated by \"and Others\" or \"et al.\" \n",
    "* Some older journals articles only provide first initial and not full first names, in which case you will need to go digging via Google to identify that person. \n",
    "* In rare cases where the author cannot be identified even after searching by hand, replace the first name with \"UNKNOWNNAME\" so that the classifier will estimate the gender as unknown. \n",
    "\n",
    "__NOTE__: your free account has 500 queries per month. This box contains the code that will use your limited API credits/queries if it runs without error. Re-running all code repeatedly will repeatedly use these credits.\n",
    "\n",
    "Then, run the code blocks below. (click to select the block and then press Ctrl+Enter; or click the block and press the Run button in the top menubar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "R"
   },
   "outputs": [],
   "source": [
    "genderAPI_key <- '&key=YOUR ACCOUNT KEY HERE'\n",
    "\n",
    "names=read.csv(\"/home/jovyan/cleanedBib.csv\",stringsAsFactors=F)\n",
    "setwd('/home/jovyan/')\n",
    "\n",
    "require(rjson)\n",
    "gendFA=NULL;gendLA=NULL\n",
    "gendFA_conf=NULL;gendLA_conf=NULL\n",
    "\n",
    "namesIncompleteFA=NULL\n",
    "namesIncompleteLA=NULL\n",
    "incompleteKeys=list()\n",
    "incompleteRows=list()\n",
    "\n",
    "for(i in 1:nrow(names)){\n",
    "  if (nchar(names$FA[i])<2 || grepl(\"\\\\.\", names$FA[i])){\n",
    "    namesIncompleteFA[i] = i+1\n",
    "    incompleteKeys = c(incompleteKeys, names$CitationKey[i])\n",
    "    incompleteRows = c(incompleteRows, i+1)\n",
    "  }\n",
    "  namesIncompleteFA = namesIncompleteFA[!is.na(namesIncompleteFA)]\n",
    "    \n",
    "  if (nchar(names$LA[i])<2 || grepl(\"\\\\.\", names$LA[i])){\n",
    "    namesIncompleteLA[i] = i+1\n",
    "    incompleteKeys = c(incompleteKeys, names$CitationKey[i])\n",
    "    incompleteRows = c(incompleteRows, i+1)\n",
    "  }\n",
    "  namesIncompleteLA = namesIncompleteLA[!is.na(namesIncompleteLA)]\n",
    "}\n",
    "\n",
    "write.table(incompleteKeys[2:length(incompleteKeys)], \"incompleteKeys.csv\", sep=\",\",  col.names=FALSE)\n",
    "write.table(incompleteRows[2:length(incompleteRows)], \"incompleteRows.csv\", sep=\",\",  col.names=FALSE)\n",
    "\n",
    "if (length(namesIncompleteFA)>0 || length(namesIncompleteLA)>0){\n",
    "    print(paste(\"STOP: Please revise incomplete full first names or empty cells in these rows: \", paste(unique(c(namesIncompleteFA, namesIncompleteLA)))))\n",
    "    stop(\"Do not continue without revising the incomplete names on rows in cleanedBib.csv as indicated above.\")\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "Python 3"
   },
   "outputs": [],
   "source": [
    "if os.path.exists('incompleteRows.csv'):\n",
    "    nameCount = 0\n",
    "    df = pd.read_table('cleanedBib.csv', sep=',')\n",
    "    df = pd.DataFrame(df)\n",
    "    selfCite = []\n",
    "    selfKey = []\n",
    "    counter=0\n",
    "\n",
    "    with open('incompleteRows.csv', newline='') as f:\n",
    "        reader = csv.reader(f)\n",
    "        incompleteRows = [int(y) for x in list(reader) for y in x]\n",
    "        incompleteRows = incompleteRows[1:]\n",
    "        \n",
    "    with open('incompleteKeys.csv', newline='') as f:\n",
    "        reader = csv.reader(f)\n",
    "        incompleteKeys = [y for x in list(reader) for y in x]\n",
    "        incompleteKeys = incompleteKeys[1:]\n",
    "\n",
    "    for incompleteRow in incompleteRows:\n",
    "        FA = df.iloc[incompleteRow-2]['FA']\n",
    "        LA = df.iloc[incompleteRow-2]['LA']\n",
    "\n",
    "        if (yourFirstAuthor!='LastName, FirstName OptionalMiddleInitial') and (yourLastAuthor!='LastName, FirstName OptionalMiddleInitial'):\n",
    "            if FA in returnFirstName(removeMiddleName(yourFirstAuthor)) or FA in returnFirstName(removeMiddleName(convertSpecialCharsToUTF8(yourFirstAuthor))) or LA in returnFirstName(removeMiddleName(yourLastAuthor)) or LA in returnFirstName(removeMiddleName(convertSpecialCharsToUTF8(yourLastAuthor))):\n",
    "                nameCount += 1\n",
    "                selfCite.append(incompleteRow)\n",
    "                selfKey.append(incompleteKeys[counter])\n",
    "        if optionalEqualContributors != ('LastName, FirstName OptionalMiddleInitial', 'LastName, FirstName OptionalMiddleInitial'):\n",
    "            for name in optionalEqualContributors:\n",
    "                if FA in returnFirstName(removeMiddleName(name)) or FA in returnFirstName(removeMiddleName(convertSpecialCharsToUTF8(name))) or LA in returnFirstName(removeMiddleName(name)) or LA in returnFirstName(removeMiddleName(convertSpecialCharsToUTF8(name))):\n",
    "                    nameCount += 1\n",
    "                    selfCite.append(incompleteRow)\n",
    "                    selfKey.append(incompleteKeys[counter])\n",
    "        counter += 1\n",
    "    if nameCount > 0:\n",
    "        print(\"WARNING: Before continuing, please check and remove the manually modified citations that are self-citations for row(s): \" + str(selfCite) + \" for the citation key(s): \" + str(selfKey))\n",
    "    else:\n",
    "        print(\"Please proceed to the next code block.\")\n",
    "else:\n",
    "    print(\"Please proceed to the next code block.\")\n",
    "                    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "R"
   },
   "outputs": [],
   "source": [
    "for(i in 1:nrow(names)){\n",
    "  ### get probabilistic genders for the ith article from GenderAPI\n",
    "  tfa=names$FA[i]\n",
    "  tla=names$LA[i]\n",
    "  \n",
    "  json_file_fa=paste0(\"https://gender-api.com/get?name=\",tfa,\n",
    "                      genderAPI_key)\n",
    "  json_data_fa=fromJSON(file=json_file_fa)\n",
    "  \n",
    "  ### Only query the server once if the first/last authors are the same\n",
    "  if(tla!=tfa){\n",
    "    json_file_la=paste0(\"https://gender-api.com/get?name=\",tla,\n",
    "                        genderAPI_key)\n",
    "    json_data_la=fromJSON(file=json_file_la)\n",
    "  }else{\n",
    "    json_data_la=json_data_fa\n",
    "    json_file_la=json_data_fa\n",
    "  }\n",
    "  \n",
    "  ### Locate and save gender probabilities from json query\n",
    "  if(json_data_fa$accuracy>=70){\n",
    "    ### If probability is above 70%, assigned \"W\" or \"M\" to author\n",
    "    gendFA[i]=ifelse(json_data_fa$gender==\"female\",\"W\",\"M\")\n",
    "    gendFA_conf[i]=json_data_fa$accuracy\n",
    "  }else{\n",
    "    ### If not, assign \"U\" for unknown, and potentially fill these in manually\n",
    "    gendFA[i]=\"U\"\n",
    "    gendFA_conf[i]=json_data_fa$accuracy\n",
    "  }\n",
    "  ### Do the same for last authors\n",
    "  if(json_data_la$accuracy>=70){\n",
    "    gendLA[i]=ifelse(json_data_la$gender==\"female\",\"W\",\"M\")\n",
    "    gendLA_conf[i]=json_data_la$accuracy\n",
    "  }else{\n",
    "    gendLA[i]=\"U\"\n",
    "    gendLA_conf[i]=json_data_la$accuracy\n",
    "  }\n",
    "  \n",
    "  ### Take a quick break before sending the server another request\n",
    "  Sys.sleep(sample(1:2,1))\n",
    "  print(i)\n",
    "  flush.console()  \n",
    "}\n",
    "\n",
    "### Add new columns to data.frame to save for later use\n",
    "names$FA_bin=gendFA; names$FA_conf=gendFA_conf\n",
    "names$LA_bin=gendLA; names$LA_conf=gendLA_conf\n",
    "\n",
    "\n",
    "### Pull names that the query server wasn't sure about\n",
    "unknownFAs=names$FA[names$FA_bin==\"U\"]\n",
    "unknownLAs=names$LA[names$LA_bin==\"U\"]\n",
    "unknownFAs; unknownLAs\n",
    "\n",
    "### At this stage, you can manually enter the gender of any\n",
    "### if you can find pronouns or other signifiers online\n",
    "\n",
    "# e.g. names$FA_bin[names$FA_bin==\"Romy\"]=\"W\"\n",
    "\n",
    "\n",
    "### Create column of gender categories (i.e., MM, WM, MW, WW)\n",
    "names$GendCat=paste0(gendFA,gendLA)\n",
    "write.csv(names,\"Authors.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## Describe the proportions of genders in your reference list and compare it to published base rates in neuroscience.\n",
    "\n",
    "Describe the gender proportions breakdown for man/man, man/woman, woman/man, and woman/woman. The example template can be copy and pasted into your manuscript. We have included it in our methods or references section. If you are using LaTeX, [the bibliography file can be found here](https://github.com/dalejn/cleanBib/blob/master/diversityStatement/bibfile.bib).\n",
    "\n",
    "OPTIONALLY: At this stage, you can manually enter the gender of any authors if you can find pronouns or other signifiers online. Modify Authors.csv, re-upload your manually modified Authors.csv, and uncomment \n",
    "\n",
    "```#names<-read.csv('Authors.csv')```\n",
    "\n",
    "by deleting the # at the beginning of the corresponding line below. Re-run the code box. \n",
    "\n",
    "This box does NOT contain code that will use your limited API credits/queries.\n",
    "\n",
    "### Additional info about the neuroscience benchmark\n",
    "For the top 5 neuroscience journals (Nature Neuroscience, Neuron, Brain, Journal of Neuroscience, and Neuroimage), the expected gender proportions in reference lists as reported by [Dworkin et al.](https://www.biorxiv.org/content/10.1101/2020.01.03.894378v1.full.pdf) are 58.4% for man/man, 9.4% for man/woman, 25.5% for woman/man, and 6.7% for woman/woman. Expected proportions were calculated by randomly sampling papers from 28,505 articles in the 5 journals, estimating gender breakdowns using probabilistic name classification tools, and regressing for relevant article variables like publication date, journal, number of authors, review article or not, and first-/last-author seniority. See [Dworkin et al.](https://www.biorxiv.org/content/10.1101/2020.01.03.894378v1.full.pdf) for more details. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "R"
   },
   "outputs": [],
   "source": [
    "# load manually modified results (OPTIONAL)\n",
    "#names<-read.csv('Authors.csv')\n",
    "\n",
    "##########################\n",
    "# Tables and proportions #\n",
    "##########################\n",
    "\n",
    "#Get the overall counts and proportions for each category\n",
    "print(\"Number of citations\")\n",
    "table(names$GendCat)\n",
    "\n",
    "print(\"Percent of citations\")\n",
    "round(table(names$GendCat)/sum(table(names$GendCat)),3)*100\n",
    "\n",
    "# Output table will show the observed (your) reference proportions in the first row\n",
    "# The second row displays estimated expected proportions in neuroscience from:\n",
    "# https://www.biorxiv.org/content/10.1101/2020.01.03.894378v1.full.pdf\n",
    "\n",
    "# Get proportions without unknowns\n",
    "checkProportions <- round(table(names$GendCat, exclude=c(\"MU\", \"UM\", \"UU\", \"WU\", \"UW\")),3)/round(sum(table(names$GendCat, exclude=c(\"MU\", \"UM\", \"UU\", \"WU\", \"UW\"))),3)\n",
    "\n",
    "# Check gap between observed and expected\n",
    "# Expected proportions in neuroscience were 58.4% for MM, 25.5% for WM, 9.4% for MW, and 6.7% for WW\n",
    "checkProportions <- rbind(checkProportions, c(0.584, 0.094, 0.255, 0.067))\n",
    "print(\"Comparison of observed (row 1) and expected (row 2) citation percents, excluding unknowns\")\n",
    "checkProportions\n",
    "gap <- round((checkProportions[1,]-checkProportions[2,])*100/checkProportions[2,], 2)\n",
    "\n",
    "print(\"Percent difference between observed (row 1) and expected (row 2)\")\n",
    "gap\n",
    "\n",
    "tab1 <- round(table(names$GendCat)/sum(table(names$GendCat)),3)\n",
    "tab2 <- table(names$GendCat)/sum(table(names$GendCat))\n",
    "\n",
    "if(!(\"MM\" %in% names$GendCat))\n",
    "{\n",
    "  tab1[\"MM\"] <- 0\n",
    "}\n",
    "\n",
    "if(!(\"MW\" %in% names$GendCat))\n",
    "{\n",
    "  tab1[\"MW\"] <- 0\n",
    "}\n",
    "\n",
    "if(!(\"WM\" %in% names$GendCat))\n",
    "{\n",
    "  tab1[\"WM\"] <- 0\n",
    "}\n",
    "\n",
    "if(!(\"WW\" %in% names$GendCat))\n",
    "{\n",
    "  tab1[\"WW\"] <- 0\n",
    "}\n",
    "\n",
    "if(sum(tab2[\"MM\"],tab2[\"MW\"],tab2[\"WM\"],tab2[\"WW\"])>=1){\n",
    "    print(\"EXAMPLE TEMPLATE:\")\n",
    "    print(paste0(\"Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field [1, 2, 3, 4, 5]. Here we sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, and other factors. We obtained predicted gender of the first and last author of each reference by using databases that store the probability of a name being carried by a woman [5, 6]. By this measure (and excluding self-citations to the first and last authors of our current paper), our references contain \", \n",
    "            tab1[\"WW\"]*100, \"% woman(first)/woman(last), \", \n",
    "            tab1[\"MW\"]*100, \"% man/woman, \", \n",
    "            tab1[\"WM\"]*100, \"% woman/man, \", \n",
    "            tab1[\"MM\"]*100, \"% man/man, and \", \n",
    "            0, \n",
    "            \"% unknown categorization. This method is limited in that a) names, pronouns, and social media profiles used to construct the databases may not, in every case, be indicative of gender identity and b) it cannot account for intersex, non-binary, or transgender people. We look forward to future work that could help us to better understand how to support equitable practices in science.\"))\n",
    "    cat(\"\\n\")\n",
    "    print(\"EXAMPLE LATEX:\")\n",
    "    cat(paste0(\"Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field \\\\cite{mitchell2013gendered,dion2018gendered,caplar2017quantitative, maliniak2013gender, Dworkin2020.01.03.894378}. Here we sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, and other factors. We obtained predicted gender of the first and last author of each reference by using databases that store the probability of a name being carried by a woman \\\\cite{Dworkin2020.01.03.894378,zhou_dale_2020_3672110}. By this measure (and excluding self-citations to the first and last authors of our current paper), our references contain $\", \n",
    "            tab1[\"WW\"]*100, \"\\\\%$ woman(first)/woman(last), $\", \n",
    "            tab1[\"MW\"]*100, \"\\\\%$ man/woman, $\", \n",
    "            tab1[\"WM\"]*100, \"\\\\%$ woman/man, $\", \n",
    "            tab1[\"MM\"]*100, \"\\\\%$ man/man, and $\", \n",
    "            0, \n",
    "            \"%$ unknown categorization. This method is limited in that a) names, pronouns, and social media profiles used to construct the databases may not, in every case, be indicative of gender identity and b) it cannot account for intersex, non-binary, or transgender people. We look forward to future work that could help us to better understand how to support equitable practices in science.\"))\n",
    "}else{\n",
    "    print(\"EXAMPLE TEMPLATE:\")\n",
    "    print(paste0(\"Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field [1, 2, 3, 4, 5]. Here we sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, and other factors. We obtained predicted gender of the first and last author of each reference by using databases that store the probability of a name being carried by a woman [5, 6]. By this measure (and excluding self-citations to the first and last authors of our current paper), our references contain \", \n",
    "            tab1[\"WW\"]*100, \"% woman(first)/woman(last), \", \n",
    "            tab1[\"MW\"]*100, \"% man/woman, \", \n",
    "            tab1[\"WM\"]*100, \"% woman/man, \", \n",
    "            tab1[\"MM\"]*100, \"% man/man, and \", \n",
    "            round((1-(tab2[\"MM\"]+tab2[\"MW\"]+tab2[\"WM\"]+tab2[\"WW\"]))*100, 2), \n",
    "            \"% unknown categorization. This method is limited in that a) names, pronouns, and social media profiles used to construct the databases may not, in every case, be indicative of gender identity and b) it cannot account for intersex, non-binary, or transgender people. We look forward to future work that could help us to better understand how to support equitable practices in science.\"))\n",
    "    cat(\"\\n\")\n",
    "    print(\"EXAMPLE LATEX:\")\n",
    "    cat(paste0(\"Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field \\\\cite{mitchell2013gendered,dion2018gendered,caplar2017quantitative, maliniak2013gender, Dworkin2020.01.03.894378}. Here we sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, and other factors. We obtained predicted gender of the first and last author of each reference by using databases that store the probability of a name being carried by a woman \\\\cite{Dworkin2020.01.03.894378,zhou_dale_2020_3672110}. By this measure (and excluding self-citations to the first and last authors of our current paper), our references contain $\", \n",
    "            tab1[\"WW\"]*100, \"\\\\%$ woman(first)/woman(last), $\", \n",
    "            tab1[\"MW\"]*100, \"\\\\%$ man/woman, $\", \n",
    "            tab1[\"WM\"]*100, \"\\\\%$ woman/man, $\", \n",
    "            tab1[\"MM\"]*100, \"\\\\%$ man/man, and $\", \n",
    "            round((1-(tab2[\"MM\"]+tab2[\"MW\"]+tab2[\"WM\"]+tab2[\"WW\"]))*100, 2), \n",
    "            \"%$ unknown categorization. This method is limited in that a) names, pronouns, and social media profiles used to construct the databases may not, in every case, be indicative of gender identity and b) it cannot account for intersex, non-binary, or transgender people. We look forward to future work that could help us to better understand how to support equitable practices in science.\"))\n",
    "}\n",
    "\n",
    "# Write\n",
    "write.csv(names,\"Authors.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### (OPTIONAL) Color-code your .tex file using the estimated gender classifications\n",
    "\n",
    "Running this code-block will optionally output your uploaded `.tex` file with color-coding for gender pair classifications. You can find the [example below's pre-print here.](https://www.biorxiv.org/content/10.1101/664250v1)\n",
    "\n",
    "![Color-coded .tex file, Eli Cornblath](img/texColors.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "Python 3"
   },
   "outputs": [],
   "source": [
    "cite_gender = pd.read_csv(homedir+'Authors.csv') # output of getReferenceGends.ipynb\n",
    "cite_gender.index = cite_gender.CitationKey\n",
    "cite_gender['Color'] = '' # what color to make each gender category\n",
    "colors = {'MM':'red','MW':'blue','WW':'green','WM':'magenta','UU':'black',\n",
    "'MU':'black','UM':'black','UW':'black','WU':'black'}\n",
    "for idx in cite_gender.index: # loop through each citation key and set color\n",
    "    cite_gender.loc[idx,'Color'] = colors[cite_gender.loc[idx,'GendCat']]\n",
    "cite_gender.loc[cite_gender.index[cite_gender.SelfCite=='Y'],'Color'] = 'black' # make self citations black\n",
    "\n",
    "fin = open(homedir+tex_file)\n",
    "texdoc=fin.readlines()\n",
    "with open(homedir+tex_file[:-4]+'_gendercolor.tex','w') as fout:\n",
    "    for i in range(len(texdoc)):\n",
    "        s = texdoc[i]\n",
    "        cite_instances = re.findall('\\\\\\\\cite\\{.*?\\}',s)\n",
    "        cite_keys = re.findall('\\\\\\\\cite\\{(.*?)\\}',s)\n",
    "        cite_keys = [x.split(',') for x in cite_keys]\n",
    "        cite_keys_sub = [['\\\\textcolor{' + cite_gender.loc[x.strip(),'Color'] + '}{\\\\cite{'+x.strip()+'}}' for x in cite_instance] for cite_instance in cite_keys]\n",
    "        cite_keys_sub = ['\\\\textsuperscript{,}'.join(x) for x in cite_keys_sub]\n",
    "        for idx,cite_instance in enumerate(cite_instances):\n",
    "            s = s.replace(cite_instances[idx],cite_keys_sub[idx])\n",
    "        fout.write(s)\n",
    "        # place color key after abstract\n",
    "        if '\\\\section*{Introduction}\\n' in s:            \n",
    "            l = ['\\\\textcolor{' + colors[k] + '}{'+k+'}' for k in colors.keys()]\n",
    "            fout.write('\\tKey: '+ ', '.join(l)+'.\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SoS",
   "language": "sos",
   "name": "sos"
  },
  "language_info": {
   "codemirror_mode": "sos",
   "file_extension": ".sos",
   "mimetype": "text/x-sos",
   "name": "sos",
   "nbconvert_exporter": "sos_notebook.converter.SoS_Exporter",
   "pygments_lexer": "sos"
  },
  "sos": {
   "kernels": [
    [
     "Python 3",
     "python3",
     "python3",
     "",
     {
      "name": "ipython",
      "version": 3
     }
    ],
    [
     "R",
     "ir",
     "R",
     "",
     "r"
    ]
   ],
   "panel": {
    "displayed": true,
    "height": 0
   },
   "version": "0.21.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}