Even More Improvements for Getbib (Including Prior)

[emrakyz]

EDIT AFTER 6 MONTHS: The API changed its formatting to single line instead of as seen below. I improved the script and preserved the old output where we had entries on separate lines with tabs.

I have used and benefited from the "getbib" script and the instructions on LaTeX from Luke for a long time. So, I have put a lot of thought into this script, since I am very interested in academia. Hope you all like this.

Justifications for Improvements

This script stands out as a highly valuable (at least in my opinion) and efficient tool for managing and fetching BibTeX entries for DOIs found in PDF files or provided directly. The robust design and comprehensive functionality make it an indispensable asset for researchers. The main reasons for its superiority are as follows:

• Exceptional time-saving: By automating the process of extracting DOIs and fetching BibTeX entries, the script drastically reduces the manual effort involved in managing citations, thereby saving users an incredible amount of time and energy.
• Outstanding versatility: The script's ability to handle various input types, including directories containing PDF files, single PDF files, and DOIs, sets it apart from other solutions. This adaptability allows users to process numerous scenarios with ease, making it the go-to tool for all their citation needs.
• Unparalleled consistency: The script ensures that DOIs are uniformly processed and normalized, improving the consistency of the entries in the BibTeX file. This feature is crucial for maintaining a clean and professional bibliography that adheres to high academic standards. It inserts an empty line between entries inside the BIB_FILE, as well as, making the author name lower case. It also removes any special characters and the first 2 numbers of the year from the first line. So it is easier to read, maintain and easier to use inside a LaTeX document. Normalizing also helps to check for duplicate entries. It prevents some weird entries escaping from getting caught as a duplicate. Most of these can be seen in Luke's videos, that's why I implement them (He puts an empty line between entries, shorten the author's name by removing special characters and the half of the year and makes the author name lower case. He uses some vim magic for this though. It can also be preferable but not needed with this script).
• Remarkable duplicate prevention: The script's built-in functionality to check for duplicate entries before appending them to the BibTeX file demonstrates a keen attention to detail. This feature ensures that the bibliography remains free of redundancies, streamlining the citation management process.
• Modularity and Adaptability: The use of functions and modular design in the script makes the code highly readable, maintainable, and extendable. This strong foundation allows for seamless adaptation to future changes and requirements.
• You can provide the DOI address even in very wrong forms and get a correct output. You can even feed it a website URL such as: https://doi.org/10.1038/s41594-023-00968-y.
• All of the DOI handling is done by a highly improved single "sed" command.
• Robust notification system to learn more about the errors or other types of feedback.
• The "curl" output is in red in order to separate the output and the notification better and to improve readability.

From the terminal (I have corrected it to show the last 2 digits):

From the uni.bib file (I have corrected it to show the last 2 digits): As it can be seen, there is an empty line between characters and a format conversion from "Antonio_2020" to "antonio20"

Details

BIB_FILE: The path to the BibTeX file where entries will be saved.
CORRECTION_METHOD: A very powerful sed command to extract and correct the DOI from the input even in harsher cases.
get_doi_from_pdf function: Extracts a DOI from the provided PDF file using pdfinfo and pdftotext commands. If pdfinfo doesn't find a DOI, it uses pdftotext to extract it from the first page of the PDF.
normalize_doi function: Normalizes the DOI by converting it to lowercase. This also helps for duplicate prevention.
process_doi function: Fetches the BibTeX entry for the given DOI using the Crossref with a curl command. Prints the output of the curl command in red using ANSI escape codes. Checks if the fetched BibTeX entry is valid and not empty. If the fetched BibTeX entry is not in the BIB_FILE, it appends the entry to the file.
The script processes input arguments, which can be a directory, a PDF file, or a DOI:
a) If it's a directory, the script processes all PDF files in the directory.
b) If it's a PDF file, the script processes the single PDF file.
c) If it's a DOI, the script processes the DOI directly.

More details on the correction method (sed command), from my prior pull request

Very Detailed Explanation (I realized that escaped backslashes do not appear. There is a backslash if you see nothing.) (For people who wonder about it, or try to learn. It could take a tremendous amount of time to learn all of it without explanation, so it would be better to explain):

sed The sed command is a stream editor that can be used to perform basic text transformations on an input file or from a pipeline. You can see Luke uses it a lot in his videos. It can also modify files' content if you want for other purposes. That function is used a lot for bootstrapping scripts for changing config files automatically if necessary.

-n This option tells sed not to print lines by default. We'll only print lines when we specify the p command in the script.

-E This option enables the use of extended regular expressions, which allows for more readable and flexible regex patterns.

's/ This starts the sed script and defines the s command (substitute). It is used to find a regex pattern in the input and replace it with a specified string.

.* This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all characters before "doi" or "DOI".

( This paranthesis opens a capturing group, which allows us to refer back to the matched text later in the script.

(DOI|doi) This regex pattern matches either "DOI" or "doi". The | symbol is used as an OR operator in regular expressions.

( This next paranthesis opens another capturing group.

(.(org))? This regex pattern matches an optional ".org". The . is an escaped period, and (org) matches the string "org". The ? following the group makes it optional. Escaping is needed for most of non-alphanumeric characters. You can test and practice them on vim, trying to use the "substitute" function to change some text.

/? This regex pattern matches an optional "/", with the ? making it optional. The prior backslash is for escaping. Again, some characters need to be escaped to be able to used in commands. Escaped means they have backslashes before them. Spaces may be the most escaped characters.

| This symbol, later, also acts as an OR operator, indicating that the pattern before or after it can be matched.

:? * This regex pattern matches an optional colon (":") followed by zero or more spaces. The ? makes the colon optional, and * matches zero or more spaces.

) This closes the capturing group started earlier.

) This closes the outer capturing group.

([^: ]+[^ .]) This regex pattern matches any character except colons and spaces one or more times ([^: ]+) Plus symbol here shows one or more times. If it is a star then it means zero or more times. It is then followed by a single alphanumeric character ([^ .]) Single because there are no plus or star symbol next to it. This part as a whole ensures that the last character of the matched text is alphanumeric.

.* This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all remaining characters in the input line.

/ This delimiter separates the regex pattern from the replacement string in the s command. s command needs a separator that is a forward slash.

doi:\6 This is the replacement string. The text "doi:" is followed by the 6th captured group from the regex pattern, which contains the characters after "doi" or "DOI" and the colon, "/", or space(s).

/p This delimiter separates the replacement string from the p command, which tells sed to print the modified line if a substitution has been made. The substitution mentioned here is the change of ".org/" to ":". This helps turning URLs into doi addresses.

; This separates different commands within the sed script.

T This command branches to the end of the script if no substitution was made since the last input line was read or conditional branch was taken. In this case, it ensures that the q command is only executed if a matching line has been found and a substitution was made. This is one of the most important parts to get the doi address from the urls such as "https://doi.org/10.1038/s41594-023-00968-5". Because we don't always have URLs for doi addresses. In this way, this function only works when we work with URLs. So in this case it helps changing .org/ with : This makes the part of the doi address as this: "doi:" rather than this: "doi.org/".

q This command tells sed to quit processing after the first match, ensuring that only the first matching line in the file is processed. Otherwise, we would get all doi addresses in a scientific study because there are lots of doi addresses in them.

' This closes the other '

TL;DR:

Basically this whole command ensures that the output we get starts with "doi:", then it can have every type of character in it except spaces and ".org/" , then it will end with an alphanumeric character [A-Z, a-z or 0-9]. That ensures removing the trailing dots from some doi addresses that have them.

New and improved "sed" command's examples:

Here are the examples before and after:
argument1="doi.org/10.1038/s41594-023-00968-y."
argument2="doi.org/10.1038/s41594-023-00968-y"
argument3="https://doi.org/10.1038/s41594-023-00968-y."
argument4="https://doi.org/10.1038/s41594-023-00968-y"
argument5="doi: 10.1038/s41594-023-00968-y"
argument6="doi: 10.1038/s41594-023-00968-y."
argument7="doi:10.1038/s41594-023-00968-y."
argument8="doi.org/10.1038/s41594-023-00968-5."
argument9="doi.org/10.1038/s41594-023-00968-5"
argument10="https://doi.org/10.1038/s41594-023-00968-5."
argument11="https://doi.org/10.1038/s41594-023-00968-5"
argument12="doi: 10.1038/s41594-023-00968-8"
argument13="doi: 10.1038/s41594-023-00968-3"
argument14="doi: 10.1038/s41594-023-00968-3."
argument15="doi:10.1038/s41594-023-00968-3."
argument16=".doi:10.1038/s41594-023-00968-3."
argument17="adoi:10.1038/s41594-023-00968-3."
argument18="a: doi:10.1038/s41594-023-00968-3."
argument19="a doi:10.1038/s41594-023-00968-3."
argument20="doi: a0.1038/s41594-023-00968-3"
argument21="doi: b01038/s41594-023-009.68-3."
argument22="doi:c01038/s41594-023-00968-3."
argument23=".doi:d01038/s4.1594-023-00968-3."
argument24="adoi:e01.038/s41594-023-00968-3."
argument25="a: doi:f010.38/s41594-023-00968-3."
argument26="a doi:g0103.8/s41594-023-00968-3."

The Example Output After the Change to the New Command

(Every Output is correct.):

doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-y
doi:10.1038/s41594-023-00968-5
doi:10.1038/s41594-023-00968-5
doi:10.1038/s41594-023-00968-8
doi:10.1038/s41594-023-00968-5
doi:10.1038/s41594-023-00968-8
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:10.1038/s41594-023-00968-3
doi:a0.1038/s41594-023-00968-3
doi:d01038/s4.1594-023-00968-3
doi:c01038/s41594-023-00968-3
doi:d01038/s4.1594-023-00968-3
doi:e01.038/s41594-023-00968-3
doi:f010.38/s41594-023-00968-3
doi:g0103.8/s41594-023-00968-3

[v3natio]

Hi, I've got some improvements for the BIB_ENTRY variable. We can take the path as a second argument to the function, or default to a .bib file in the current working directory. Otherwise, the script detects if none of those two conditions are present and warns the user.

#!/bin/sh

correction_method() {
  sed -n -E 's/.*((DOI|doi)((\.(org))?\/?|:? *))([^: ]+[^ .]).*/\6/p; T; q'
}

get_doi_from_pdf() {
  pdf="${1}"
  doi="$(pdfinfo "${pdf}" 2> "/dev/null" | correction_method)"
  [ -z "${doi}" ] && doi="$(pdftotext -q -l "2" "${pdf}" - 2> "/dev/null" | correction_method)"
  [ -z "${doi}" ] && echo "No DOI found for PDF: ${pdf}" >&2 && return "1"
  echo "${doi}"
}

correct_names() {
  sed 's/\}, /\},\n\t/g
    s/, /,\n\t/
    s/ }/\n}/
    s/,\s*pages=/,\n\tpages=/' |
  sed '1s/^ *//
    1s/[0-9]*\([0-9]\{2\}\)/\1/
    1s/_//
    1s/.*/\L&/
    s/.*=/\L&/
    s/=/ = /'
}

process_doi() {
  doi="${1}"
  bib_file="${2}"
  bibtex_entry="$(curl -s "https://api.crossref.org/works/${doi}/transform/application/x-bibtex" | correct_names)"
  red_color='\033[0;31m'
  reset_color='\033[0m'
  printf "${red_color}%s${reset_color}\n" "${bibtex_entry}"
  [ -z "${bibtex_entry}" ] && [ "$(echo "${bibtex_entry}" | cut -c2)" != "@" ] && {
    echo "Failed to fetch bibtex entry for DOI: ${doi}"
    return "1"
  }

  grep -iFq "doi = {${doi}}" "${bib_file}" || {
    [ -s "${bib_file}" ] && echo "" >> "${bib_file}"
    echo "${bibtex_entry}" >> "${bib_file}"
    echo "Added bibtex entry for DOI: ${doi}"
    return "0"
  }

  echo "Bibtex entry for DOI: ${doi} already exists in the file."
}

# Check for arguments
if [ -z "${1}" ]; then
  echo "Give either a PDF file, a DOI, or a directory path that has PDFs as the first argument."
  exit "0"
fi

# Determine the BIB_FILE
if [ -z "${2}" ]; then
  bib_files_in_cwd=$(find . -maxdepth 1 -type f -name "*.bib")
  bib_file_count=$(echo "$bib_files_in_cwd" | wc -l)

  if [ "$bib_file_count" -eq 1 ]; then
    BIB_FILE=$(echo "$bib_files_in_cwd")
  else
    echo "No .bib file provided and none found in the current directory."
    exit "1"
  fi
else
  BIB_FILE="${2}"
fi

# Check if BIB_FILE is set and exists
if [ -z "${BIB_FILE}" ] || [ ! -f "${BIB_FILE}" ]; then
  echo "No valid .bib file found or given."
  exit "1"
fi

# Process input
if [ -d "${1}" ]; then
  for pdf in "${1}"/*.pdf; do
    doi="$(get_doi_from_pdf "${pdf}")"
    [ -n "${doi}" ] && process_doi "${doi}" "${BIB_FILE}"
  done
  exit "0"
fi

if [ -f "${1}" ] && [ "$(echo "${1}" | grep -c "\.pdf$")" -ne "0" ]; then
  doi="$(get_doi_from_pdf "${1}")"
  [ -n "${doi}" ] && {
    process_doi "${doi}" "${BIB_FILE}"
    exit "0"
  }
fi

doi="$(echo "${1}" | correction_method)"
[ -n "${doi}" ] && process_doi "${doi}" "${BIB_FILE}"

[emrakyz]

Hi @v3natio

Thanks. I'll consider. A similar approach can be implemented in a simpler, more concise way.

I used that location in the script because that was what Luke used on his videos and I also use that location for latex related files on my system.

I can add a logical control and I will also make the script more concise & minimal in general.

[emrakyz]

@v3natio

Hi again, I have added the feature in a very concise way.
Even though we added a new feature, the script is now 10 lines less, and around 200 characters less.

If we come to your issue, the script now:

1. Use "${HOME}/latex/uni.bib" by default.
2. If not present, use "${2}".
3. If neither is correct, find "*.bib" files in "${HOME}" without searching dotfiles.
4. If no bib file at the end, give an error and exit.

Here is the implementation:

BIB_FILE="${HOME}/latex/uni.bib"
[ -f "${BIB_FILE}" ] || BIB_FILE="${2:-$(find "${HOME}" -path "${HOME}/.*" \
	-prune -o -type "f" -name "*.bib" -print -quit)}"

{ [ -f "${BIB_FILE}" ] || [ "${2}" ]; } || {
	printf "%s\n" "Create a .bib file or provide as \$2." && exit "1"
}

[v3natio]

Hi @emrakyz, that's a better implementation indeed, for my use-case I'll be removing the use of ${HOME}/latex/uni.bib, and just default to looking for the .bib file in the current working directory if no argument is given.

BIB_FILE="${2:-$(find . -maxdepth 1 -type f -name '*.bib' -print -quit)}"
if [ -z "${BIB_FILE}" ]; then
  printf "%s\n" "No .bib file found in the current directory. Create a .bib file or provide one as the second argument." >&2
  exit 1
fi

But obviously I understand that these scripts are supposed to match Luke's setup, so your approach should be the one being merged.

Thanks for the help, cheers!