Increase compatibility between latexmk and pythontex by providing warnings

[muelli]

...es

The code comes from John Collins collins@phys.psu.edu, the author of latexmk.
The warnings apparently make latexmk aware of the missing files and then runs
pythontex.

Since 0.11 is released already, I took the liberty to created a new changelog
entry under a newer version.

I changed the .dtx file and ran pdflatex on .ins and the .dtx.

[gpoore]

Can you give me a little more information about how you are configuring latexmk to run PythonTeX? That might help me figure out the best way to proceed. Have you looked into latexmk's -recorder option? I don't use latexmk, but a quick look at the docs suggests it might be relevant and might even do much of what you want.

Apparently latexmk needs a message of the form File #1' not found` in the log, so that it can detect that a file is missing. Then, based on the fact that a file is missing, and that the warning came from the PythonTeX package, latexmk can run PythonTeX and then compile the document a second time.

So every time a file should be brought it, but doesn't exist, a message should be written to the log, using a macro like \InputFileOrMessage. That's a good idea for the file of macros and the file of Pygments content that PythonTeX generates.

But detecting in general when PythonTeX needs to be run is tricky, and just knowing if files are missing isn't necessarily enough.

• If PythonTeX has never been run, then checking for the file of macros and issuing a warning when it is not found will be sufficient.
• If PythonTeX has run before, then you could check for missing files. That wouldn't account for changed code, though, so you would also need to see if the hash of the .pytxcode file has changed (hashing the document would also work, but the .pytxcode is better, because it only contains the code, not document text). And even that wouldn't guarantee that PythonTeX doesn't need to be run, because if PythonTeX is tracking dependencies (pytex.add_dependencies() method), then the only way to make sure things are up-to-date is to run PythonTeX. Unless there's a way for PythonTeX to pass its dependencies on to latexmk.
• It's not possible to create warning messages for missing files that are brought in by autoprint. Those files may or may not exist, depending on whether anything was written to stdout. So the fact that a file doesn't exist may just mean that nothing was written to stdout. It would be possible to always create a file, and just have empty files when nothing was printed, but that would result in a lot of unnecessary files.

My guess is that you will need to run both latex and pythontex at each compile to guarantee that everything is updated, and then compile again based on whether the PythonTeX output has changed. Currently, PythonTeX isn't very efficient if nothing has changed, but I can look into adding a more efficient mode in the next release or two.

[muelli]

With the patched pythontex I can run latexmk with the following as latexmkrc:

add_cus_dep('pytxcode', 'tex', 0, 'pythontex');
sub pythontex {
return system("pythontex "$_[0]"") ;
}

Apparently, that teaches latexmk to run pythontex in some cases it is needed. Not in all cases, but I'm happy about that for now. If you can make it run reliably, that's awesome, but I even appreciate latexmk being able to compile my document at all.

[gpoore]

Am I correct in understanding that your add_cus_dep() only works if you use a patched pythontex? I would have guessed that the add_cus_dep() would have worked with the default pythontex without patching, but if not, that helps narrow down the changes that are needed.

Have you tried latexmk's -recorder option? That might improve the reliability of latexmk detecting pythontex changes.

[muelli]

1. Latexmk uses the -recorder option by default. But that doesn't
   help in the situation of an initial run of pdflatex, because the fls
   file generated by the use of the -recorder option doesn't contain the
   names of relevant files that don't yet exist.
2. There are two sets of dependent files: A) Those for the pdflatex
   run; these include files generated by pythontex.py. B) Those for
   pythontex.py; at least in the cases I have tried, the only such file
   is the pytxcode file.
3. The warning messages that I suggested are so that latexmk reliably
   determines the correct dependencies of the pdflatex run.
4. It is mentioned that a good way of determining whether to rerun
   pythontex is to compute whether a hash code of the pytxcode file has
   changed. That is in fact exactly how latexmk does it.
5. An issue about the autoprint files are mentioned. My tests show
   that the change I proposed to pythontex.sty is compatible with
   autoprint files. If no stdout file(s) is/are generated, there will be
   an essentially spurious warning message(s) in the log file, but that
   doesn't cause any problem.

[gpoore]

I have found a solution that doesn't require warning messages for missing files.

With your add_cus_dep(), latexmk will run pythontex any time the .pytxcode file changes. That doesn't quite cover all possible scenarios, but should be good enough for almost all cases. (For example, if rerun=always, then theoretically pythontex may need to be run during every compile...but you could just customize your latexmkrc for that case.)

The problem is that whenever pythontex creates new files for input, latexmk isn't aware of them, since the files didn't exist during the previous run and thus weren't detected. pythontex creates a file of macros .pytxmcr that is always brought into the document. latexmk automatically detects and tracks this file of macros via -recorder. I've added a hash of the names of all files created by pythontex to the beginning of this file. So any time that pythontex runs and creates new files, the .pytxmcr will have a different hash, and that will trigger another compile.

This feature will be in the next release, which should be out in the next day or two. I will probably close this issue at that point; please reopen the issue if you have further suggestions.

[gpoore]

I actually ended up implementing this a little differently, for efficiency. The .pytxmcr contains the time at which files were last created. This has the same effect; the hash of the .pytxmcr changes whenever there are new files.

Edit: Originally, I accidentally referred to .pytxcode instead of .pytxmcr. That is fixed now.