Lua - accessing div.content in empty div changes its content

[cscheid]

Explain the problem.

In a Lua filter, accessing the content field of an empty div element appears to introduce an empty Plain [] element into the content.

Here's a repro:

test.md

::: {}
:::

test.lua

function Div(div)
  print("Before accessing content", div)

  div:walk({
    Plain = function(plain)
      print("oh oh")
    end
  })

  print("Access content here: ", div.content)
  print("This still shows empty: ", div.content[1])
  print("... and this shows 0: ", #div.content)
  print("But this prints an element", div)

  print("... and this finds an element (?)")
  div:walk({
    Plain = function(plain)
      print("oh oh")
    end
  })
end

repro

$ pandoc -f markdown test.md -L test.lua -o /dev/null
Before accessing content	Div ("",[],[]) []
Access content here: 	[]
This still shows empty: 	nil
... and this shows 0: 	0
But this prints an element	Div ("",[],[]) [Plain []]
... and this finds an element (?)
oh oh

Pandoc version?

3.1.12.3, (ARM macOS).

This appears to not be a regression, and reproduces going back to at least 2.19.2 (any other versions I tried further back would fail from a lack of a :walk{} method.)

[jgm]

@tarleb would know about this.

[tarleb]

Just to add a bit to the weirdness:

local div = pandoc.Div({})
print(div)
x = div.content
print(div)
print(pandoc.Div(pandoc.Blocks{}))
div.content = pandoc.Blocks{}
print(div)

prints

Div ("",[],[]) []
Div ("",[],[]) [Plain []]
Div ("",[],[]) []
Div ("",[],[]) [Plain []]

Which makes peekBlock the prime suspect.

[tarleb]

Same issue with BlockQuote:

local bq = pandoc.BlockQuote({})
print(bq)
x = bq.content
print(bq)

BlockQuote []
BlockQuote [Plain []]

[tarleb]

I'm still in the dark. What I can say so far is that the empty list is pushed correctly, it's an empty list in Lua:

print(#div.content, rawlen(div.content))  -- 0   0

Also, the peekBlocksFuzzy function seems to be fine, too, as that's what we use in the pandoc.Div constructor:

print(pandoc.Div(div.content))   -- Div ("",[],[]) []

So the problem must be in the way that empty lists are read back from lazy properties.

Apparently the list is treated as if it was nested. That might be a hint.

print(pandoc.Div(pandoc.Blocks{{}}))  -- Div ("",[],[]) [Plain []]

[tarleb]

Also, the problem doesn't exist with Note, which also has a content property of type [Block]:

local note = pandoc.Note({})
print(note)
x = note.content
print(note)

Note []
Note []

[tarleb]

Found it. The issue was hidden in the setBlockContent function in Text.Pandoc.Lua.Marshal.Block. The function is a bit of a crutch that's required because content can have different types depending on the specific Block element. The case of an empty list wasn't handled correctly.

Patch incoming.