Execution of small or time-critical analysis scripts as part of running shot

[philipstarkey]

Original report (archived issue) by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

Description:

Add a feature that allows the execution of small or time-critical scripts within the context of an individual labscript shot. The intended application of this tool is for cases where in-shot or time-critical processing of data is required, for example when images or traces have been acquired that are used to feedback to the next shot to stabilize the magnetic field.

Proposed syntax:

#!python

# Initialize a script executing “device”.
# These can be launched locally or remotly (using the to be implemented methods to start
# devices on remote computers.

ScripterLocal = ScriptManager(start_order=-1, stop_order=1)
ScripterRemote = ScriptManager(host=192.168.2.5, start_order=-1, stop_order=2)

from Scripts import Script1, Script2, Script3

# These will be executed after the shot completes in order on the local computer
ScripterLocal.AttachScript(Script1)
ScripterLocal.AttachScript(Script2)

# this will be executed on a remote computer after the shot stops, on the remote computer
ScripterRemote(Script3)

Comments:

As “devices” these will have Blacs tabs each with suitable error reporting. start_order and stop_order are two new kwargs to be present in the lowest level device class. These dictate the order in which devices are sent to buffered mode (start_order) and then back to manuals (stop_order). These default to 0 (so all 0's are started together, all 1's together and so forth).

Script1,2,3 are callable python functions. With the call syntax TBD, for example each should expect to be given a path to the current h5 file (or to the group were it should save any data), and current variables (?) and it should expect some action to be taken from its return values (such as having them sent to runmanager or something).

Persistence can be achieved by assuming that the returns of the current cycle will be passed to the function next time. OR each ScripterLocal can maintain a persistent local cache [I like this one]

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

• Edited issue description* changed title from "While setting values for the NI PCI 6733 DAQ... #17: execution of small or time-critical analysis scripts as part of running shot" to "Execution of small or time-critical analysis scripts as part of running shot"

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

Hi Ian,

In general I support this proposal. I think this sort of framework may come in handy once we rewrite BIAS in Python (which I hope will be able to be implemented as a remote BLACS tab)

I do have a couple of questions about the details of the proposal:

1. Would making a BLACS tab run a single-shot lyse script solve the same problem? I believe lyse scripts can be run without the lyse GUI, as long as the path to the h5 file is passed in. I guess the concern is if doing this would be slow? But if it's feasible, I think it would be better than having 2 distinct ways of making analysis scripts (and also means you automatically get access to the lyse API for accessing the h5 file, which I think is a big plus).

2. Is there a reason for defining this in the labscript experiment logic file? I'm not necessarily opposed to it, but as a contrasting example, we don't define the lyse analysis scripts to run on a shot from within the labscript file (even though we could). Since it's a break from the current design philosophy (only things that effect the shot, should be defined in the experiment logic), I think it would be prudent to discuss this in detail.

I think it would also be good for us to flesh out how we ensure h5 files remain a complete record of the experiment given that conceivably results of analysis will effect either the current shot or the next (or subsequent) shot(s). If the change is just through runmanager globals, I think everything will be fine, but if there are other mechanisms (which I think ChrisB was discussing with me once?) then we should decide on a standard way to record this so the shots are as reproducible as possible.

[philipstarkey]

Original comment by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

The design goal of this is to perform computations that are an essential part of the execution of the script, for example providing information that must be ready for the desired execution of the next script.

We have a tool like this in place already and we use it to stabilize the magnet field, but ChrisB rightly objects that it is a bit of a hack.

2. Yes. These scripts are intended to be used when they are an essential part of the experiment logic, and therefor should be included in the script defining the execution of the experiment. They should not be used for "analysis;" in fact my attitude (that ChrisB does not subscribe to) is that "everything" about the execution of your sequence should work 100% without lyse running at all. to me, it is an analysis and rendering tool, not part of the closed loop operation of the sequencer.

3. I don't care strongly. My opinion is that the desired functionality should not require the programmer know anything about lyse, but rather assume that they just know numpy scipy and enough about h5py to get the desired data out of the h5 file. The minimal functionality should be provided by the python return statement, and our vision is to somehow get this information back to runmanager globals [I do very much think that these functions should have their results dumped into the h5 file because they are being treated as a device, and this would allow lyse to plot the results later on to track performance or whatever.]. ChrisB and I are still in discussions about how this looks in practice. All of this is not to say that the lyse machinery could not be leveraged, but I do not think it should be required.

We already have a number of camera devices implemented in 100% python (they are indeed just devices that are given a tab of some sort in Blacs), and part of pushing the JQI changes back to the main branch will get these there for everybody. You will shortly see a feature request appearing regarding giving Blacs the ability to launch worker processes on remote computers, which is required for this camera functionality.

[philipstarkey]

Original comment by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

• Edited issue description

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Pull request #12 has been backed out due to regressions, so we'll think of a different way to do that!

Basically the transision order needs to be defined only for Devices that are actually are programmed by BLACS, which in labscript are identified by having a BLACS_connection attribute. But inspecting that attribute isn't very compatible with using set_passed_properties. I think in labscript 3, BLACS programmable devices will be identified possibly by inheriting from a mixin that adds the desired properties, but I'm happy for something hackier in the meantime.

One quick fix could be to make the transition order a connection table property instead of a device property - this would make BLACS not see a device group for the device and so would ignore it. It is still kinda silly though for devices that are not being transitioned to buffered mode to even have the attribute though. And that way it wouldn't be able to vary on a per-shot basis.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Here's a checklist of what needs to be done to get this working:

labscript:

• The ability to specify an order in which devices transition to static and to buffered mode. Addressed in labscript pull request 37

zprocess:

• Allow an event to connect to a remote broker by specifying a host and port numbers. Implemented in zprocess 2.6

BLACS:

• Server should have a method to give BLACS version number (so that future dependencies on communication with the server can be tracked), and another to give broker
  ports to allow subscribers to subscribe to its events.
• Emit a zprocess event (which is a wrapper around a zmq.PUB message) whenever the queue decreases in size saying how many shots are left in the queue (including zero)
• Implement ordered transition to static and transition to buffered based on settings
  saved by labscript as above. Backward compatible if the settings don't exist.

runmanager:

• A "remote API", runmanager.remote for interacting with a running instance of runmanager and setting globals, getting globals, clicking 'engage', and more to be added later.

• A tab showing the compilation queue, including variables that differ between different shots in the queue. Support re-ordering and deletion, and make it a TreeView grouping by sequence. Show compilation progress bars.

• A checkbox for "just-in-time" compilation vs ahead of time compilation. The 'set
  global' remote command can have a 'where' argument that says whether to set it
  in the globals interface, or the queued shots, or both.

• A setting for when the shots should be compiled in "just-in-time" mode - when BLACS has an empty queue, or when it has one remaining shot, or ten? This is optional - as a first implementation this should be just "compile them when BLACS has an empty queue"

• depend on the version of zprocess allowing connection to remote broker, and listen for events telling you when to submit a shot.

• There needs to be an option to keep the apparatus warm if the compilation queue runs dry. This could probably be implemented at first as 'repeat all/repeat last' options to the compilation queue, identical to those in the BLACS queue, which is the mechanism currently used to keep experiments warm in ahead-of-time-compiled shots. In the longer run, a notion of a "default shot", could be useful, in which runmanager has a 'current value' and a 'default value' for globals. the 'default value' could be used to submit 'keep warm' shots. A diff between default globals and current globals would also be super useful for quickly inspecting what has changed about the globals. But this is a separate feature.

labscript_devices:

• A "script_runner" device that either stores functions or whole scripts to execute
  at the end of a shot, specified at compile time. Leaning toward letting these
  be functions rather than actually being runnable lyse routines - I think they are distinct, though there's
  no reason why the lyse API can't be used to save results. The script runner may need
  to interact with the lyse API to specify the name of the group to which any results
  are saved, since it cannot be introspected as it usually is in a lyse routine.

[philipstarkey]

Original comment by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

For cases where we want to perform feedback, would the running script directly send the updated variables to runmanager using the new interface?

I think that functions are the at least "a" way to go, if not "the" way to go. What kind of call signature were you expecting for these functions? It could be as simple as,

def mini_function(h5file):
    """
    h5file is an open reference to the current shot file in its current state.  
    It has already been locked by h5lock and will be closed when the script
    is done
    """

    # do stuff

    pass

If these functions return anything, what should that mean? If we want access to the current variables are they to be extracted from the h5 file, or provided as an additional argument to the function?

If we want persistence of information how should that be managed? It could be 100% in the runmanager variables that are sent to run-manager, or ...

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

For cases where we want to perform feedback, would the running script directly send the updated variables to runmanager using the new interface?

Yes, this is what I'm imagining at present. Though, the script runner class could easily provide syntactic sugar for this, such as interpreting the return values of the function as variables to be updated for the next shot as in your current implementation. But extra layers of syntactic sugar are not preferable compared to making the underlying implementation easy to use in the first place, so I will try to make the the runmanager remote API as simple as possible, perhaps making wrappers around them more trouble than they're worth.

I think that functions are the at least "a" way to go, if not "the" way to go. What kind of call signature were you expecting for these functions? It could be as simple as,

I'm inclined to agree. That function signature seems fine, and I think I'll go with it for the time being unless any issues reveal themselves.

If these functions return anything, what should that mean?

As a first implementation, maybe nothing. I can imagine making it so that they can return true or false as to whether they were successful or not, or things like that...but trying to anticipate something now seems premature. Would prefer to indicate failure via an exception, anyhow - and if an exception is to be ignored, the calling code can print the traceback, save a flag of some sort indicating failure, and keep running.

If we want access to the current variables are they to be extracted from the h5 file, or provided as an additional argument to the function?

Perhaps the h5 file provided should be not open already - that way you can instantiate a lyse.Run(h5_filename) object, or call lyse.data(h5_filename) on it, which would be how you could get shot globals and save analysis results. If this interface is unsatisfactory (Phil has recently been pointing out the obtuseness of some of the interface and proposed that the globals be an attribute of the Run() object instead of requiring a separate call to lyse.data(), that sort of thing), then I would prefer to fix it there rather than introduce another interface.

If we want persistence of information how should that be managed? It could be 100% in the runmanager variables that are sent to run-manager, or ...

If using the lyse API, there is a persistent storage mechanism, or you could save state to disk, or the script_runner worker process could provide a dictionary or something as a dumping ground for persistence. I would lean toward the latter, though unless the data is on disk, it wouldn't persist across "device" restarts. If the user wants to store data on disk, probably we should leave them on their own, there are plenty of ways to do this that would be complicated by an additional wrapper, and should be just used directly. But probably a dictionary per function/scriptlet/whatever you want to call it that will persist from one shot to the next would be good, and would probably sensibly be provided as a keyword argument to the user function.

[philipstarkey]

Original comment by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

Great, but lets follow up on

"""
Perhaps the h5 file provided should be not open already - that way you can instantiate a lyse.Run(h5_filename) object, or call lyse.data(h5_filename) on it, which would be how you could get shot globals and save analysis results. If this interface is unsatisfactory (Phil has recently been pointing out the obtuseness of some of the interface and proposed that the globals be an attribute of the Run() object instead of requiring a separate call to lyse.data(), that sort of thing), then I would prefer to fix it there rather than introduce another interface.
"""

I would suggest two things:

1. Lyse should be able to deal with open references to the h5file, which is an easy check anyway
2. I am totally happy with the option to allow people to use the lyse infrastructure, but I think that labscript might benefit for a more streamlined interface to pick out globals that does not involve instantiating a large object. Such as:

#!python

one_global = ShotGlobals(h5file, global_name) # for a global
globals = ShotGlobals(h5file, [global1, global2, ...]) # for some globals
globals = ShotGlobals(h5file) # for all globals

the idea is that these would just be fast function calls that get data from an h5 file or a reference

One more thought comes to mind that one would want to define somehow: in labscript, devices generally store data to places in the h5 file related to the name of the device. Since the same script might be run by several different "devices" (why, I don't know, but it could). As a result, the device should be involved in informing the script where its data "should" go. It could be as simple as having a call signature that provides two references, one read-only reference (is this even possible?) to the whole h5 file (so you can access the images you want to analyze, for example) and a second to r/w reference to the data storage location.

[philipstarkey]

Original comment by Ian B. Spielman (Bitbucket: Ian Spielman, GitHub: ispielma).

---

I have been thinking about persistent variables that you want to retain from shot to shot, and I have a different suggestion than the above. I suggest a mechanism whereby there is a provided function such as:

SetPersistentVariable(name, value)

where these persistent variables are in reality stored in the next h5 file in the device group for this specific device and the function

var = GetPersistantVariable(h5file, name)

will get that specific persistent variable from the current h5 file. I guess this could equally well be done using the future ability to updated globals to runmanager, in which case the GetPersistantVariable function is not needed.

This suggests a useful syntax would be:

SetRunMangerVariable(name, value)

My reasoning for this suggestion is that it is preferable that this state information be retained in the history of shot files.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

I think that would be a bit tricky! Ensuring the data gets to the next HDF5 file would require it be stored somewhere in the meantime, since the next HDF5 file may not exist yet. And if the next shot file does exist because shots have been pre-compiled, then the notion of 'next' is a bit fuzzy since shots can be re-ordered before execution. For just-in-time-compiled shots, if the data is stored in runmanager ready to be added to the next shot file, then runmanager will need to store it to disk, since runmanager can be restarted and the data would be lost.

We could do that, but it's many hoops to jump though considering that there are a number of ways to do disk persistence in Python, from manually using h5py to using the shelve standard library module, or np.save, or pickle (shelve is probably the best for this kind of purpose), I suspect users using these things directly for disk persistence is the best solution (and when we get around to documenting this functionality, pointing users in that direction would be good). For non-disk-persistent data, some object that attributes can just be saved to as in lyse would be adequate. We could instantiate a shelf for the user so they don't have to choose a filename, but other than that there isn't much functionality I think we could add that isn't best served by the user using the standard library themselves.