Get list of references in Lua filter

[tarleb]

I'd like to get the list of references for a document, essentially by exposing the function T.P.Citeproc.getReferences to Lua. Here's what I think would be a good way to approach this. The basic idea is to improve JSON support in filters, so Reference values can be opaque objects for now.

1. A global function tojson is added to the pandoc.utils module (or maybe a separate pandoc.json module?). The function uses aeson to do the encoding.
2. Modeled on Lua's tostring function, the tojson function checks for a __tojson metamethod and returns the result of that function if it exists.
3. A __tojson metamethod is added to all AST elements that are instances of ToJSON.
4. A new Reference type is added to Lua; it has no properties except for the __tostring, and __tojson metamethods.
5. The getReferences function is exported as pandoc.utils.references.

This would give access to references without us having to decide which Reference properties to expose, and how to bring case-insensitive behavior to Lua. Also, having tojson seems like a generally useful feature.

Does it sound like a sensible course of action?

Edit: There'd also be a function fromjson which converts a JSON string to a Lua object. It could take an optional second parameter to identify the target type. E.g., fromjson(input, pandoc.ReaderOptions)

[jgm]

Could you clarify why you want this?/what you'd be able to do with it that you can't do now?

[tarleb]

I'm trying to generate Crossref XML output. The XML should include information on all citations in an article, which I'd like to generate in a filter (or maybe in a custom writer). I don't see a clean solution to do this right now.

[jgm]

getReferences returns a [Reference Inlines].
If you tojson that (using the aeson instance), then you'll get pandoc json for the Inlines content. That may be pretty inconvenient to work with in lua (even converted to a lua table).
Wouldn't it be more ergonomic to have getReferences return a Lua table in which the Inlines content is Lua Inlines objects?

[tarleb]

Quite likely, yes. I guess I had assumed that I'd do some hacky back-and-forth through JSON to get the output I need.

The main thing that gave me pause was that keys in the Reference variables map are case-insensitive, which doesn't fit into the HsLua object framework. But if we ignore this, then there would be only two other values that'd need to be marshaled, Val Inlines and Name. That would give me the basic info I'd need and would be much nicer to use.

[jgm]

Can you make keys case-insensitive using metatables, e.g. https://nachtimwald.com/2019/02/12/lua-case-insenstive-table/ ?

[tarleb]

The problem I saw is that this kind of table requires low-level code and doesn't fit the deftype mechanism we use for other types. I'm trying to stick to these pre-made structures to reduce the maintenance overhead.

However, I now think special handling is justified in this case: for one thing, it will enable the case-insensitive handling. Another point is that users may want to use the References list as metadata, which is easy to support if a Reference is pushed in a way such that it can be treated as a MetaValue.

Long story short: Using a normal table/metatable combination for Reference values is likely the best choice. Thanks for asking the right questions!

Maybe JSON support could still be interesting, but that's a different topic.