RFC: Introduce bundle serve command.

[let-def]

This pull request adds a new bundle serve command, an interactive version of bundle cat. It allows querying the contents of multiple files without spawning a new process each time.

Motivation

While working on TeXpresso, a real-time LaTeX file previewing tool, we've encountered a bottleneck in our current workflow. It is possible to use Tectonic as a TeX distribution via the bundle cat command, but querying the contents of multiple files spawns a new process each time and leads to performance issues. We propose introducing a new bundle serve command to address this problem.

Background

TeXpresso is an experimental tool that allows users to preview LaTeX files in real-time. It achieves using a patched XeTeX and a specialized viewer. You can see it in action with Vim, Emacs and VSCode. The viewer requires a TeX distribution, which is where Tectonic comes in. We started by using bundle cat, but we need a more efficient solution.

Proposal

The bundle serve command is an interactive version of bundle cat that allows querying the contents of multiple files without spawning a new process each time. To achieve this, we need to delimit the different requests.

We implemented a simple protocol to stream all contents over a single stream:

1. On standard input, bundle serve reads a list of file names, one per line (delimited by \n).
2. For each file name, in order, bundle serve responds with a 9-byte header followed by a payload:
   • The first byte indicates the type of response: C for contents, P for path, or E for errors.
   • The next 8 bytes represent the length of the payload as a little-endian 64-bit unsigned integer.
   • The payload contents depend on the first byte:
     • C indicates the contents of the requested file.
     • P indicates a path on the local file system where the requested file can be found.
     • E indicates an error message explaining why the request failed.

Alternative

If a more complex protocol is not desirable, a simpler line-based protocol reading file names and writing either paths or error messages would be sufficient. This approach is similar to kpsewhich -interactive (with improved error handling).

Conclusion

We believe introducing the bundle serve command and its accompanying protocol would significantly improve the performance of TeXpresso and, potentially, other tools that rely on bundle cat. We're open to feedback and alternative solutions.

[let-def]

Notes:

1. At the moment, TeXpresso bundles a fork of Tectonic. I am cleaning that up as I would like it to work with vanilla tectonic. This PR would allow us to get rid of the fork without impacting performance.

2. I don't know of a standard solution to this "file delimitation" problem. HTTP 1.1 solved a similar one with Chunked Transfer Encoding, though it is both more flexible and more complex. The proposed protocol can be implemented in a few lines of code. The alternative has the benefits of playing better with shell utils/scripts.

[codecov]

Codecov Report

Attention: Patch coverage is 6.89655% with 54 lines in your changes are missing coverage. Please review.

Project coverage is 46.96%. Comparing base (19654bf) to head (000e277).

Files	Patch %	Lines
src/bin/tectonic/v2cli/commands/bundle.rs	7.69%	48 Missing ⚠️
crates/bundles/src/cache.rs	0.00%	3 Missing ⚠️
crates/io_base/src/lib.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1195      +/-   ##
==========================================
- Coverage   46.99%   46.96%   -0.04%     
==========================================
  Files         176      176              
  Lines       65196    65255      +59     
==========================================
+ Hits        30639    30644       +5     
- Misses      34557    34611      +54     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.