Refactor of labscript.py

[philipstarkey]

This is a first pass attempt at breaking up labscript.py into separate files. The idea here is that, while this might not be how we ultimately want things to be split up, at this point something is better than nothing. This should provide enough of a base that other people begin to feel comfortable moving things around and/or breaking things up logically. It also introduces some conceptual boundaries between output classes and the base device classes that handle timing/clock generation. My hope is that this split will open up the possibility of actually writing tests for some of this stuff. That's probably still a long way off (with several more refactors in between) but it's a step towards that goal!

Other changes:

• More modern context managers for warning suppression that can actually be used to enable/disable warnings, not just disable.
• config is moved into compiler which means it should get reset between labscript shots and not just globally changed until you reload the compiler subprocess in runmanager.
• Some better/more consistent formatting. I didn't run black/ruff but I think we should think about it soon.
• Updated error message string formatting to use f strings for increased code readability
• Fixed a mistake with trigger error detection that was never raised (and also the if condition around the unraised error was wrong anyway)
• Fixed a minor bug in an error message for shutters (had open/close the wrong way around)
• Probably some other small changes.

Suggestions for reviewing:

• Look at the changes commit by commit. I did in place changes first, then moved stuff, in separate commits (3 times). That should make it easier to see what functionality is actually being changes
• Running a h5diff on files produced from a few different complex experiments scripts (using both the base branch and this branch) would be a good idea. I tested example.py already (looks the same).
• I haven't replicated the from pylab import * in the new files. I believe I caught every case where that would have overridden a builtin function that we relied on, and updated the code to use it directly from numpy instead, but I may have missed one or two. I almost missed the use of divmod. Not sure how much this matters though. There is also a good chance that pylab is doing less overriding of inbuilt functions these days than when we initially wrote it. And since we have no unit tests, we won't have caught those historical changes over the last 13 years anyway 🤷
• I haven't really tested the warning suppression stuff which is maybe worth doing since I moved inside compiler

[dihm]

Thanks for doing this Phil, it is sorely needed! I've made a minor comment just to confirm a minor change was intentional. Otherwise, most of the work is formatting changes and moving things which should be transparent. I can run it against an experiment in the lab (next week since we are having a snow day today), though that experiment isn't really all that complex so I doubt it would catch any edge cases that have been introduced.

I've also added another comment about removing the pylab star import. Probably scope creep for this PR, but I would be interested in addressing it in the next one.

[dihm]

Speaking of scope creep, what are you thoughts about adding type hints to API portions of labscript (like core labscript here)? Cost/benefit analysis makes it tight. It's a big lift that can rot quickly when interacting with external libs. Another project I'm on uses them as another layer of documentation and we occasionally manually check them with mypy to catch bugs which is a lower barrier to entry.

I know the biggest problem is Device name passing. I'm pretty sure modern python has a reasonable path to not require it (structurally at least). But it may be hard to implement without breaking changes to how labscript scripts are currently done.

Anyway, type hinting the API portions of labscript is a pretty decent quality of life improvement (given modern editor auto-completions) that I would interested in pursuing in the near-ish term.

PS. I know this came up recently somewhere, but I can't find that conversation ATM.

[dihm]

Note to future me: we'll need to update the docs to point to all these new files before final merging.

[philipstarkey]

Speaking of scope creep, what are you thoughts about adding type hints to API portions of labscript (like core labscript here)?

I am keen to get this in too, although I'd prefer to have it separate to this PR. The smaller the PRs, the easier it is to bisect the changes is something goes wrong and a major bug rears it's head. At least until we get some real testing going :)

[philipstarkey]

I've updated the documentation and also fixed some other things up while I was there.

There is one small issue on the API table of contents page, which seems to have pulled a docstring for the compiler class into the module list as a description, and then when you visit the labscript.compiler module documentation, you get an empty page (rather than something that lists the labscript.compiler.Compiler class). Also, none of the other modules have descriptions. Not sure if that is something we can add in or not 🤷 Since you set up the autosummary stuff I figured you might know what is going on?

[dihm]

Well, module descriptions are supposed to be pulled from the docstring at the top of the module file (before imports, after comments if present). Not required but probably helpful here to clearly state the organizing principle.

I'll have to take a look at the compiler stuff tomorrow. It is probably some issue with the templating that shouldn't be hard for me to track down.

[philipstarkey]

Looks like it is a bug in sphinx, see sphinx-doc/sphinx#11362 . I think I'll just remove that module from the docs - it's very internal anyway.

I've added module level docstrings to the other module, and the copyright header (although really that's feeling a bit pointless these days given the license file in the repo root is sufficient). And I moved a utility function I found. Hopefully this is in a better state now!

[dihm]

This all looks good for me. I still need to actually try compiling some shots in lab, but I will actually do that this week. Once that is done, I say we go ahead and merge.

[dihm]

Speaking of random sphinx syntax things, if you wanted you can add a lot more info to the module-level docstrings without breaking the summary strings. Basic syntax (of all sphinx docstrings with autosummary) is to have the first line until a linebreak be the summary, then the whole docstring is shown on the associated docs page as usual. As a trivial example, the core module docstring could be

"""Core classes containing common device functionality

These are used in labscript-devices when adding support for a hardware device.
"""

The first line would be the summary string in the table. The entire thing would show up on the module docs page (with the linebreak).

[dihm]

Just realized that I forgot to report back about compiling an experiment in the lab. Everything worked without issue. Not the most complicated experiment to compile, but it uses a Pulseblaster, NI DAQs, and a bunch of custom devices. I say this is ready to merge.

[dihm]

@philipstarkey Unless you have anything else you'd like to add to this PR, I'll plan on merging tomorrow.