Catch the engine(s) up to TeXLive 2020.0 #666

pkgw · 2020-10-24T20:52:47Z

At long last, let's start the work to catch the Tectonic engine implementations up to TeXLive 2020.0.

CC @Mrmaxmeier @crlf0710 tectonic-typesetting/tectonic-staging#8

There is a lot going on here so I hope that I got it all right. There is new code relating to CMaps that alters the PDF output, so unfortunately it does not seem practical to try to make it so that the new code can pass the existing PDF-output test suite.

pkgw · 2020-10-24T20:53:28Z

Current status: new xdvipdmx passes the test suite on my Linux laptop. We'll see how it fares in the CI battery ...

codecov · 2020-10-24T21:06:54Z

Codecov Report

Merging #666 into master will decrease coverage by 0.00%.
The diff coverage is 22.91%.

@@            Coverage Diff             @@
##           master     #666      +/-   ##
==========================================
- Coverage   46.30%   46.29%   -0.01%     
==========================================
  Files         132      133       +1     
  Lines       59137    60331    +1194     
==========================================
+ Hits        27386    27933     +547     
- Misses      31751    32398     +647

Impacted Files	Coverage Δ
src/engines/mod.rs	`61.75% <0.00%> (-2.22%)`	⬇️
src/errors.rs	`41.66% <ø> (ø)`
src/io/filesystem.rs	`52.94% <0.00%> (-7.06%)`	⬇️
src/io/mod.rs	`72.63% <0.00%> (-2.37%)`	⬇️
src/io/stdstreams.rs	`84.90% <0.00%> (-5.10%)`	⬇️
src/lib.rs	`72.41% <0.00%> (ø)`
tectonic/core-bridge.c	`81.94% <0.00%> (-1.16%)`	⬇️
tectonic/dpx-bmpimage.c	`1.81% <0.00%> (-0.02%)`	⬇️
tectonic/dpx-cid.c	`0.31% <0.00%> (-0.92%)`	⬇️
tectonic/dpx-cidtype0.c	`0.00% <0.00%> (-0.11%)`	⬇️
... and 75 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e4f756...d8dddeb. Read the comment docs.

Mrmaxmeier · 2020-10-24T21:20:29Z

Hmm, this causes a bunch of SIGABRTs and SIGSEGVs:
https://tt.ente.ninja/#/compare/tectonic@0.1.17/d47454ba362b95bb9191835a27f6e555478e903d

I'll add the tectonic-on-arXiv bot to this repo to simplify tracking down these issues.
EDIT: GitHub doesn't allow me to add the bot. @pkgw You'll need to approve this somewhere 🙂

pkgw · 2020-10-24T21:34:08Z

Windows vcpkg builds are current failing due to an unrelated issue: microsoft/vcpkg#13286, addressed by microsoft/vcpkg#13298 .

edit: Looking more closely, the relevant PR seems to be stalled. Not clear what the solution is going to be since it seems that the available versions of Windows just don't support TLS 1.3, which msys2.org is now requiring.

pkgw · 2020-10-24T21:38:10Z

@Mrmaxmeier OK, I installed the app on this repo. Do I need to do any further configuration?

Compared to xdvipdfmx this is a *much* smaller delta. Just two small modifications. Yay.

Mrmaxmeier · 2020-10-24T21:46:52Z

@Mrmaxmeier OK, I installed the app on this repo. Do I need to do any further configuration?

No. It's supposed to just work ™️ after installing. The whole thing is pretty hacky at the moment though and I'll need to manually add each fork for it to pull the new commits.

EDIT: This doesn't work :/
https://github.com/Mrmaxmeier/tectonic-on-arXiv/blob/4ab2738fb279dcf521b511b8a218bc8b3b5418c9/github-ci/src/index.ts#L19

pkgw · 2020-10-24T21:51:47Z

@Mrmaxmeier Should I just rename this branch? That feels like it would be the most TeX-esque solution ...

Mrmaxmeier · 2020-10-24T21:57:39Z

I'll fix it in the bot. The GitHub API for custom checks has probably improved a bunch since I hacked it together.
It looks like the relevant information is now included in the web-hook payload, so I'll use that instead of searching the git tree locally 👍

Not much new here. Omitted the conversion of various #ifndef/#define header guards to `#pragma once` due to (entirely hypothetical) concerns about portability.

Very little new here: - Instead of exiting when buffers fill up, increase their sizes and keep on going. - Avoid adding redundant filename extensions in some outputs.

pkgw · 2020-10-24T23:04:14Z

Looks like it's running now, super cool!

As I recall, the concrete values of the enum options here need to stay in sync with other parts of the engine. I'm not 100%, but can't hurt to keep the values consistent with what they were before the catch-up.

This typo is upstream.

Transcription error.

pkgw · 2020-10-25T00:59:03Z

@Mrmaxmeier There was one big typo that caused the vast majority of the regressions. Now there are only a handful, but I can't reproduce them locally — my build of the CLI program handles them successfully. Do you have a sense of what the might be going on if your tool reports internal errors that don't reproduce in the CLI?

Mrmaxmeier · 2020-10-25T16:20:50Z

The CI bot runs release builds in a Ubuntu 20.10 docker. Reproducability issues are probably either different system libraries (It'd be nice statically link harfbuzz) or some form of UB.
I tried reproducing 1702.05081 locally and it segfaults due to an uninitialized(?) char-pointer:

gef➤  
#1  0x00005555557675f0 in save_font () at tectonic/dpx-mpost.c:159
159	    next->font_name = NEW(strlen(current->font_name)+1, char);
gef➤  p current->font_name
$5 = 0x4052000000000000 <error: Cannot access memory at address 0x4052000000000000>

- Tweaks to how font design sizes are mapped to TeX values - Instead of having a separate `prim_eqtb`, merge it into the main eqtb. - This forces us to bump the format file serial number. - New primitives, mostly unimplemented: - \creationdate - \elapsedtime - \expanded - \filedump - \filemoddate - \filesize - \normaldeviate - \randomseed - \resettimer - \setrandomseed - \uniformdeviate - Some uc_hyph tweak deep in the linebreaking algorithm - print_raw_char when using pseudo selector in print_char - New magic numbers used in PDF last x/y position accounting, instead of cur_[hv]_offset. - Don't print_raw_char in show_context with trick_buf - Back up cur_cs in scan_keyword - Handle XETEX_MATH_GIVEN in scan_something_internal - If encountering an unexpandable primitive in scan_something_internal, try to deal with it. - Ditto for scan_int. - Do something different with active chars in str_toks_cat - Rework how file names are scanned. - Defend against undefined eTeX registers in main_control - Back up cur_cs in compare_strings.

Note that the `trip` reference outputs do not need updating if the `print_char` and `show_context` changes are reverte.

pkgw · 2020-10-26T12:08:41Z

Almost there! We just need to fix the Arxiv regressions and actually implement the new I/O primitives.

Looking at the diff in the update to TeXLive 2020.0, I think the upstream changes introduced some bugs.

pkgw · 2020-10-26T23:13:46Z

@Mrmaxmeier looks like the crashes are now fixed!

The only one that needs implementing is \filemoddate, although these need actual testing as well.

This requires us to add a new feature to the I/O API and so involves touching a lot of code. In particular, I think that it is probably important for the memory I/O layer to produce meaningful-ish mtimes, which adds some complexity there. The `InputFeatures` trait has added a new method, `get_unix_mtime`, to support this primitive. It has a default implementation to return `Ok(None)`, which indicates that the input does not have a well-defined modification time. **BREAKING CHANGE** In order to implement file modtimes for the memory I/O provider, we need to add more data to its data structures, breaking the APIs for users of its `files` field. This is a very clear signal that this API should be cleaned up and made more future-prof. That being said, migration should be pretty easy: instead of `files` containing a bunch of `Vec<u8>`s, it now contains a bunch of `MemoryFileInfo` structs that contain a `Vec<u8>` field named `data`. So you just need to add some `.data` field accessors to existing code.

pkgw · 2020-10-27T20:08:24Z

@Mrmaxmeier The last few builds generated internal errors for the tectonic-on-arXiv bot :-( Did I break your service?

It looks like it is going to take a while for vcpkg to get its act together. Tracking issue created: tectonic-typesetting#668

pkgw · 2020-10-27T22:41:16Z

I believe that this is now ready to go! Modulo the fact that we should always have more test coverage and the Arxiv tester seems to be broken at the moment.

Discovered in the course of tracking down issue tectonic-typesetting#478. XeLaTeX works because it generates a "PK" font file on the fly. We can't do that. More research needed to understand if we should even try.

pkgw · 2020-10-28T01:18:12Z

Reran the TeXLive 2020 bundle tests with this version of the engine. Ten more classes compile now, and there are no new failures, so that's a good sign.

pkgw · 2020-10-28T13:55:51Z

Going to try to retrigger the tectonic-on-arXiv bot since it looks like it may have been fixed a few hours ago.

pkgw added 2 commits October 24, 2020 16:46

tests/tex-outputs: update PDF outputs for new xdvipdfmx behavior

d47454b

pkgw marked this pull request as draft October 24, 2020 20:52

pkgw mentioned this pull request Oct 24, 2020

Updates to SourceForge XeTeX tectonic-typesetting/tectonic-staging#8

Closed

pkgw added 2 commits October 24, 2020 17:10

Cargo.toml: update vcpkg in hopes of fixing Windows/vcpkg build

ed456ff

tectonic/xdvipdfmx: remove some inappropriate Win32-isms that snuck in

5c91b00

Update synctex to the TeXLive 2020.0 reference

08c4250

Compared to xdvipdfmx this is a *much* smaller delta. Just two small modifications. Yay.

pkgw added 2 commits October 24, 2020 18:13

Update TECKit to the TeXLive 2020.0 reference

7a158df

Not much new here. Omitted the conversion of various #ifndef/#define header guards to `#pragma once` due to (entirely hypothetical) concerns about portability.

Update bibtex to the TeXLive 2020.0 reference

dc34298

Very little new here: - Instead of exiting when buffers fill up, increase their sizes and keep on going. - Avoid adding redundant filename extensions in some outputs.

pkgw added 3 commits October 24, 2020 20:08

tectonic/dpx-pdfdoc.h: fix values for box type enum

1449a24

As I recall, the concrete values of the enum options here need to stay in sync with other parts of the engine. I'm not 100%, but can't hurt to keep the values consistent with what they were before the catch-up.

tectonic/dpx-pdfximage.c: fix uninitialized value

c718a6b

This typo is upstream.

tectonic/dpx-tt_cmap.c: fix mistaken variable name

05ea01f

Transcription error.

pkgw added 4 commits October 25, 2020 16:29

Add -Z continue-on-errors unstable option.

07c7bf3

Update test cases for the engine updates

b29c6dd

Note that the `trip` reference outputs do not need updating if the `print_char` and `show_context` changes are reverte.

tectonic/xetex-XeTeXLayoutInterface.cpp: fix new Harfbuzz include

01f2820

xdvipdfmx: attempt to fix new segfault in Arxiv regression suite

40c432b

Looking at the diff in the update to TeXLive 2020.0, I think the upstream changes introduced some bugs.

pkgw added 4 commits October 26, 2020 22:44

Wire in the \creationdate implementation and add a test

e6e81ca

tests/tex-outputs.rs: try reenabling the_letter_a on Windows

a4bcd56

tectonic/xetex-texmfmp.c: work on \filesize and \filedump

3f43d56

The only one that needs implementing is \filemoddate, although these need actual testing as well.

tests: add some basic tests for \filedump and \filesize

a0fdaa1

pkgw force-pushed the catchup-texlive-2020 branch from ce7bf30 to a0fdaa1 Compare October 27, 2020 13:56

pkgw added 3 commits October 27, 2020 18:13

Wire up \filemoddate and add a test

b318d91

dist/azure-ci-cd.yml: disable the Windows/vcpkg build for now

cf840ea

It looks like it is going to take a while for vcpkg to get its act together. Tracking issue created: tectonic-typesetting#668

reference_sources: we are now caught up to TeXLive 2020.0 (r54631)

0cb9e0b

Mrmaxmeier mentioned this pull request Oct 27, 2020

tectonic-on-arxiv doesn't autostart on oxidize branch crlf0710/tectonic#287

Closed

tectonic/dpx-pkfont.c: warn if we try to kpse_find_glyph

d8dddeb

Discovered in the course of tracking down issue tectonic-typesetting#478. XeLaTeX works because it generates a "PK" font file on the fly. We can't do that. More research needed to understand if we should even try.

pkgw mentioned this pull request Oct 28, 2020

Tectonic seems to loop infinitely on this input file #470

Closed

pkgw marked this pull request as ready for review October 28, 2020 13:51

pkgw closed this Oct 28, 2020

pkgw reopened this Oct 28, 2020

pkgw merged commit 045dbb5 into tectonic-typesetting:master Oct 28, 2020

pkgw deleted the catchup-texlive-2020 branch October 28, 2020 19:33

This was referenced Oct 29, 2020

PRs & commits we need to port from master crlf0710/tectonic#299

Open

Oxidize everything #459

Open

burrbull mentioned this pull request Jan 3, 2021

Xetex666 (xetex engine) crlf0710/tectonic#339

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Catch the engine(s) up to TeXLive 2020.0 #666

Catch the engine(s) up to TeXLive 2020.0 #666

pkgw commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020

codecov bot commented Oct 24, 2020 •

edited

Loading

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020

pkgw commented Oct 25, 2020

Mrmaxmeier commented Oct 25, 2020

pkgw commented Oct 26, 2020

pkgw commented Oct 26, 2020

pkgw commented Oct 27, 2020

pkgw commented Oct 27, 2020

pkgw commented Oct 28, 2020

pkgw commented Oct 28, 2020

Catch the engine(s) up to TeXLive 2020.0 #666

Catch the engine(s) up to TeXLive 2020.0 #666

Conversation

pkgw commented Oct 24, 2020 • edited Loading

pkgw commented Oct 24, 2020

codecov bot commented Oct 24, 2020 • edited Loading

Codecov Report

Mrmaxmeier commented Oct 24, 2020 • edited Loading

pkgw commented Oct 24, 2020 • edited Loading

pkgw commented Oct 24, 2020

Mrmaxmeier commented Oct 24, 2020 • edited Loading

pkgw commented Oct 24, 2020

Mrmaxmeier commented Oct 24, 2020 • edited Loading

pkgw commented Oct 24, 2020

pkgw commented Oct 25, 2020

Mrmaxmeier commented Oct 25, 2020

pkgw commented Oct 26, 2020

pkgw commented Oct 26, 2020

pkgw commented Oct 27, 2020

pkgw commented Oct 27, 2020

pkgw commented Oct 28, 2020

pkgw commented Oct 28, 2020

pkgw commented Oct 24, 2020 •

edited

Loading

codecov bot commented Oct 24, 2020 •

edited

Loading

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading

pkgw commented Oct 24, 2020 •

edited

Loading

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading

Mrmaxmeier commented Oct 24, 2020 •

edited

Loading