Cache and change detection

[aginiewicz]

Hello, first of all - you have a very interesting project. I'm starting to work on a new textbook. My previous book was using SageTeX, I'm looking around now, because I need something for generic Python this time. Would love to have a change detection:

If something changes in Python (version of some library, API, format of returned value, etc), and the results are no longer like they were before, I'd like to see a warning that I have to rework part of already written book. That way, I will be able to catch changes early.

The way I imagine it could be implemented (purly on python side) is something like this:

• For new interpreter session, calculate SHA-1 of interpreter session name and set it as current HASH
• For every new executed code, calculate SHA-1 of previous HASH and source of current code, set it as current HASH
• Execute current code and get output.
• Check if there is some code stored under current HASH (for example in sqlite database), if not, store results there
• If there is, but it differs, emit LaTeX warning, that something changed and validation of document is needed.

This way we have a few separate strings of HASH values, starting at the start of new interpreter session, and they are equal if both current code and execution history within given session does not change. As a side product (although this is not my intention), a cache mode would be possible, where all output could be read from sqlite, if no input changed (might be useful for some longer Monte Carlo simulations that take a long time to run). The cache itself could be implemented in lua, which could make running python not needed at all, but change detection need to execute all code anyway for comparison, so this is why I imagined it like this.

This would be a killer feature for me. Without it, I'm sticking to other systems, which store temporary files, because then I can store temporary files in git and analyse changes that way. Anyway, having this integrated into single run and displayed directly in editor trough LSP, in my opinion would be really useful.

What do you think?

[tndrle]

Hi, thanks a lot for starting off this discussion. I find the idea of having a change detection quite appealing. As you already explain, there are various ways to realize a detection mechanism, e.g. as part of PyLuaTeX alone or via external tools like git. A first estimation from my side is: implementing it in PyLuaTeX alone involves either a lot of work or lots of hard coded decisions (how to hash, what to hash, when to reset) (or possibly both). Also, I want PyLuaTeX to be rather lean and flexible. This is, by the way, why I leave the acutual logic of code typesetting to the user.

For the change detection, I'd suggest something similar:
I'm thinking about providing a user callback that is called after any Python code execution. The callback handler receives all relevant information about the executed code and the output. The user can then create their own change detection and adapt it to their needs.

A possible (simple) implementation could be like this:
document.tex:

\documentclass{article}
\usepackage{pyluatex}

\directlua{
hash = ""

function hash_handler(info)
  text = info.session .. table.concat(info.code) .. table.concat(info.output)
  % update MD5 hash with executed Python code and output
  hash = md5.sumhexa(hash .. text)
end

pyluatex.register_event_handler("post_execute", "hash", hash_handler)
}
% write final hash to *.pyluatex_log file
\AtEndDocument{\directlua{io.savedata("\jobname.pyluatex_log", hash)}}

\begin{document}
\py{1 + 1}
\end{document}

Content of document.pyluatex_log after running the above TeX file:

2dd0f7a89a5d2b47ae3be3fc7b5b246a

In this example, the actual change detection would be done with git by checking in document.pyluatex_log for version control. The implementation can easily be extended to generated a hash per session. Detecting changes here directly during the TeX run is possible as well. However, this requires much more logic (maybe with SQLite as you suggest or only with simple files).

What do you think about this idea? Would that be helpful for you?

Merry Christmas :)

[aginiewicz]

Hi!

I believe that hooks are a good idea, it allows for more fitting custom implementations. I'm thinking only about external files, because results can depend on external files or store external files (images?). Do you maybe have any idea if we can track which files (file descriptors?) are accessed and in what modes? checksums for files written should also be part of "result" hash somehow to be most universal - or maybe this is best case for git?

Happy New Year!

[tndrle]

Hi, I don't know how accessed files can be tracked automatically. I guess you would need to write a wrapper for this on the Python side. For tracking checksums of written files, you could compute the checksum in Python and write it to the log using PyLuaTeX's tex.log(). I can make those log messages accessible in the callback (info.log_message). Then you can write those checksums to a file or include them in the overall hash.

This is some code I could imagine:

\documentclass{article}
\usepackage{pyluatex}

\directlua{
hash = ""

function hash_handler(info)
  % optionally, search info.log_message for 'FILE HASH: ...' to handle it separately

  %                                                          log output is included in overall hash
  %                                                                                   |
  text = info.session .. table.concat(info.code) .. table.concat(info.output) .. info.log_message
  % update MD5 hash with executed Python code and output
  hash = md5.sumhexa(hash .. text)
end

pyluatex.register_event_handler("post_execute", "hash", hash_handler)
}
% write final hash to *.pyluatex_log file
\AtEndDocument{\directlua{io.savedata("\jobname.pyluatex_log", hash)}}

\begin{document}
\begin{python}
from pathlib import Path
import hashlib

file = Path('foo.txt')
file.write_text('some content')
tex.log('FILE HASH:', file, hashlib.md5(file.read_bytes()).hexdigest())
\end{python}
\end{document}

Tracking external files in git is always an option of course.

Tracking changes (or only hashes) of external files read or written in Python can become quite complex, I think. I don't see a universal way at the moment. Offering such functionality is out of scope for PyLuaTeX right now. In contrast, adding only the callback functionality is relatively easy to do. I try to implement an experimental version in the next days.

[tndrle]

Hey, I created a pull request with a possible implementation of a callback. I also added an example TeX file that implements a simple change detection.