citeproc settings in YAML metadata block: Problems when using pandoc-zotxt.lua filter

[njbart]

Bug description

This report describes an issue with obtaining bibliographic data directly from Zotero, bypassing the need for exporting .bib, .json, or .yaml bibliographic data, by using the Zotero add-on zotxt and the associated pandoc filter pandoc-zotxt.lua.

Using Rmarkdown, it is possible to knit/render files such as the following from within RStudio (provided Zotero is running, and an entry @bard:2015introduction does exist):

---
output:
  html_document: 
    pandoc_args:
      - '--lua-filter'
      - pandoc-zotxt.lua
      - '--citeproc'
---

Blah blah [@bard:2015introduction, 67].

Result:

Blah blah (Bard 2015, 67).

Bard, Kathryn A. 2015. An Introduction to the Archaeology of Ancient Egypt. 2nd ed. Hoboken, NJ: Wiley-Blackwell.

Using quarto from the command line, with this file (test.qmd)

---
format: html
---

Blah blah [@bard:2015introduction, 67].

and the command

quarto render test.qmd -L /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua -C

produces the exact same expected result.

(Note that I had to provide the full path to the filter here, but that seems to be a different issue.)

For qmd from within RStudio, one would expect to be able to achieve the same with appropriate settings in the YAML metadata block; however, this does not seem to be possible, as using pandoc-args (or pandoc_args) has no effect, and does not seem to be supported anyway.

Example:

---
format:
  html: 
    pandoc-args:
      - '--lua-filter'
      - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
      - '--citeproc'
---

Blah blah [@bard:2015introduction, 67].

Result:

Blah blah [@bard:2015introduction, 67].

(i.e., citation not resolved)

Including filters and citeproc: TRUE in the header has no effect either:

---
filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
citeproc: TRUE
format: html
---

Blah blah [@bard:2015introduction, 67].

Result:

Blah blah [@bard:2015introduction, 67].

(i.e., citation not resolved)

What does work is including the instruction to use a bibliography file (dummy.json) which includes only one (dummy) entry, different from @bard:2015introduction. This seems sufficient to trigger the activation of citeproc.

---
filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
format: html
bibliography: dummy.json
---

Blah blah [@bard:2015introduction, 67].

Result: as expected (note that not even citeproc: TRUE is required here).

Possible solutions:

1. Allow the use of citeproc as a (pseudo-) filter (note that the the order of filters matters)

   filters:
     - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
     - citeproc
   

2. Include support for using pandoc-args (or pandoc_args) in a YAML metadata block.

3. Make the following work:

   filters:
     - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
   citeproc: TRUE
   

   A problem I see here, though, is that in this particular case pandoc-zotxt.lua has to be run before citeproc, but there might be other filters that have to be run after citeproc, so the option to force a specific order of filters (and pseudo-filters) which would be possible with 1. and 2. might be best.

---

RStudio 2022.07.1+554 "Spotted Wakerobin" Release (7872775ebddc40635780ca1ed238934c3345c5de, 2022-07-22) for macOS
Mozilla/5.0 (Macintosh; Intel Mac OS X 11_6_8) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.10 Chrome/69.0.3497.128 Safari/537.36

quarto -v
1.1.179

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[dragonstyle]

Thanks for reporting this.

It looks like we were disabling the citeproc filter unless we determine that there are references or a bibliography. You should now be able to specify:

filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
citeproc: true

By default the citeproc filter will be run after user provided filters, but you can control its position using the pseudo filter as you suggest:

filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
  - citeproc

This will only position the citeproc filter if it is enabled (because there are references, a bibliography, or citeproc: true is specified). (This doesn't for citeproc on, just positions it if it is present).

See below

[dragonstyle]

Correction (thanks @cderv!) - citeproc is always run after user provided filters no matter what (which allows filters to interact with citation before they are rendered).

[njbart]

Thanks for the update. This is not a problem for the pandoc-zotxt.lua use case described above, but it might be worth keeping in mind that there are others where filters need to be run before and after citeproc in a specific order (for one example, see here).

[cderv]

I don't know if that will help but I think it is worth mentioning that Pandoc 2.19.1 has now a pandoc.utils.citeproc that runs citeproc on the document. https://pandoc.org/releases.html#pandoc-2.19.1-2022-08-18

Could help running before and after using filters.

[njbart]

Thanks for pointing this out.

For future reference: This filter (taken from here), in a file citeproc.lua

-- Lua filter that behaves like `--citeproc`
function Pandoc (doc)
  return pandoc.utils.citeproc(doc)
end

combined with the following in a YAML metadata block

filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua
  - /Users/nb/.local/share/pandoc/filters/citeproc.lua

does run pandoc-zotxt.lua and then citeproc with the expected results.

I haven’t tried anything else so far, but it does seem it will allow you to run citeproc at any specific stage within an ordered sequence of filters and hence would solve all problems I can imagine in this context for now.

What doesn’t work as expected so far is that full paths to filters in the filter subdirectory in pandoc’s user data directory seem to be required, even though quarto pandoc -v does return User data directory: /Users/nb/.local/share/pandoc. I’ll open a separate issue for that.

[dragonstyle]

Thanks for the additional context regarding filter order (esp with citeproc). It may be that we have to re-introduce the concept of filter ordering or perhaps allow you to provide different classes of filters that target different positions in the AST rendering stack for just this type of reason. We're planning to do some work on our own LUA / filters and I'd expect we'll reconsider this as a part of that!

[bcdavasconcelos]

@dragonstyle, just curious if there were any updates on this. In case not, perhaps a simple new field would suffice.

filters-after-citeproc:
  - filter1.lua

[cscheid]

@bcdavasconcelos Thanks for the idea. We'll do something similar, but slightly different.

We're not going to be able to ship this in 1.3 because of regression risk (we're getting ready for a release).

A simple new field unfortunately does not suffice in the presence of our new Custom AST handlers, but this is a generally good idea. We'll do it in 1.4.

[njbart]

Unfortunately, the solution described above seems to have stopped working. A file, test.qmd:

---
format: html
---

Blah blah [@bard:2015introduction, 67].

processed from the command line with

quarto render test.qmd -L /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua -C

still produces the expected result (i.e., includes a resolved citation), whereas a second file:

---
format: html
filters:
  - /Users/nb/.local/share/pandoc/filters/pandoc-zotxt.lua-1.2.0/pandoc-zotxt.lua
  - /Users/nb/.local/share/pandoc/filters/citeproc.lua
---

Blah blah [@bard:2015introduction, 67].

processed from the command line with

quarto render test.qmd

which used to work well back in September 2022 (see above), now emits a warning:

[WARNING] Citeproc: citation bard:2015introduction not found

and the result lacks a resolved citation.

Rendering the second file from within RStudio produces the same unsatisfactory result.

Since including pandoc-zotxt.lua and citeproc as command line parameters does lead to the expected result, it seems unlikely that there’s anything wrong with the other elements in the toolchain, including Zotero, zotxt, pandoc-zotxt.lua or BBT, or quarto per se – but maybe something has been changed with regard quarto’s processing of filters listed in the YAML header?

---

• OS: MacOS 15.1.1 (24B91)
• IDE: Version 2024.09.1+394 (2024.09.1+394)
• quarto check

Quarto 1.5.57
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.57
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      Chromium: 869685
      TinyTeX: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /usr/local/texlive/2024/bin/universal-darwin
      Version: 2024

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.13.0
      Path: /opt/homebrew/opt/python@3.13/bin/python3.13
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.4.2
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
      knitr: 1.49
      rmarkdown: 2.29

[✓] Checking Knitr engine render......OK

• Zotero 7.0.9
• zotxt 7.0.0
• pandoc-zotxt.lua 1.2.0
• BetterBibTeX 6.7.253