Supporting pandoc 2.11

[ickc]

See sergiocorreia/panflute#142

Edit:

Warning: after studying the new AST, the decision is to completely rewrite pantable. I'd advice against PR for now, as the current code base is going to be obsolete soon.

Links on new table AST

• PR
  • Feature request started here: Support for table column spans, table attributes in AST jgm/pandoc#1024, Improving tables jgm/pandoc-types#65
  • this is where the actual change of the AST happened: https://github.com/jgm/pandoc-types/pull/66/files?file-filters%5B%5D=.hs#diff-01f4ffe52cf097ab9ff89eebce394c86
  • pandoc 2.10 release note: https://github.com/jgm/pandoc/releases/tag/2.10
  • pandoc 2.11 release note (see HTML writer): https://github.com/jgm/pandoc/releases/tag/2.11
  • pandoc-types 1.21 changelog: https://github.com/jgm/pandoc-types/blob/master/changelog
  • panflute 2.0: Develop sergiocorreia/panflute#156, sergiocorreia/panflute@243af31
• def
  • https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
  • source: https://github.com/jgm/pandoc-types/blob/master/src/Text/Pandoc/Arbitrary.hs
  • doc on lua filters has some info on this: https://pandoc.org/lua-filters.html
  • See comments at https://github.com/jgm/pandoc-types/pull/66#issuecomment-611053332 and after on normalizing tables
• example ASTs:
  • https://github.com/jgm/pandoc-types/blob/master/test/test-pandoc-types.hs
  • https://github.com/jgm/pandoc/tree/master/test/tables

pandoc reader/writer support status

2.10: add new table AST

2.10.1: add LaTeX reader support: rowspan, colspan

2.11:

• DocBook reader: column span

• HTML writer

• OpenDocument writer

[alerque]

Do you have work-in-progress code for the rewrite in a branch somewhere?

[ickc]

No. It probably is quite simple to support panflute>=2 from the current code base. But I've bigger plan for it which basically requires major rewrite and the old code base will becomes obsolete very soon so I don't think it will be time well spent to do that.

That said if anyone want to have a PR for that it would be welcomed. pantable2csv would be a bit strange there but we could assume it was a simple table like in the past and it would at least passes current tests. But I don't know if anyone would spend time on such a PR that would have been overwritten by the major change very soon.

No promise on the time line—but I'd guess by the end of the year.

P.S. The "bigger plan", which is always the goal of pantable, is to be able to write a table with all features supported in pandoc.

[ickc]

C.f. jgm/pandoc-types#86. Because of the potential breaking change, this project will be put on hold indefinitely until a stable pandoc table AST is decided.

[alerque]

Ug 😞. I understand of course, but still ug.

[ickc]

Frankly I have quite a big plan for pantable and it has gotten into a state I quite like it. May be I'm overreacting but knowing the AST may change again really is a wake up call for me to pause a bit and not betting too much in the stability of the pandoc AST.

May be the goal of supporting every feature pandoc table support is too ambitious. (That requires pantable having a lossless representation of pandoc AST and therefore very sensitive to AST change.) But that's really my motivation for pantable from day 1.

[alerque]

While I get that, 0% coverage is a lot worse than 0% < 𝑥 < 100% coverage. For the vast majority of my production use cases I'm just converting CSV data to very basic tables that pose no particular challenges for the AST. I have to imagine the bulk of what other people do with this is in line with those same basic requirements.

[ickc]

I'm considering to use what I have written as is and just make the 2 filters as lossy conversion to my internal structure for now and just pause for my big plan for the moment.

[tzok]

I would like to share my current solution: https://gist.github.com/tzok/3d017b4b35601aa1330adee38e827403

Namely, I have a Docker image with pandoc 2.9.2.1 - the last one compatible with current pantable. There is also a Makefile to show how to use the image.

For simple tables this works just fine.

[ickc]

Unless I did something wrong, pip install pantable==0.13.0 should now supports pandoc 2.11.2+. This is a big release so please be careful on potential regression and open new issues for any that you find.