-
Notifications
You must be signed in to change notification settings - Fork 2
BatchModeOperation
(Please have an up-to-date system. This info is current as of r7379.)
To run MeqTrees in batch mode, you need to have three things:
Main script
:
a TDL script that you actually want to run non-interactively. Normally this is exactly the same script that you would load into the browser (e.g. Timba/PyApps/test/pipeline_test.py, Siamese/example-sim.py, Calico/calico-wsrt.py, etc.) No changes are required.
Config file
:
a config file containing script options. This can be derived from .tdl.conf, but can have a more descriptive name (see e.g. Timba/PyApps/test/pipeline.tdl.conf, Siamese/batch-sim-example.tdl.conf.)
Control script
:
A control script that will actually implement batch-mode operations (see e.g. Timba/PyApps/test/run_pipeline_test.py, Siamese/batch-sim-example.py.)
NB: in older examples, we used to merge the main script and the control script into one, by placing control code into an if __name__ == "__main__" clause at the end of the main script. This can cause trouble with importing of sub-modules in frameworks, so I'm discouraging it from now on. Use a separate control script, it's neater anyway!
I have placed an educational example into PyApps/test. See the files pipeline_test.py, pipeline.tdl.conf, and run_pipeline_test.py, which I'll go through here.
The main script is the same script you would run in the browser, which is nice and convenient. The pipeline_test.py example defines a very simple tree for evaluating a*t+b, you can try it in the browser. Values of 'a' and 'b' are compile-time options, while the desired time interval is a runtime option.
pipeline.tdl.conf looks as follows:
- {{{[pipeline_test] a = 2 b = 1 t0 = 1 t1 = 2
[test_b] a = 1 b = 1 t0 = 0 t1 = 10.0 pipeline_test_constants.c0 = 2 pipeline_test_constants.c1 = 3
[test_a] a = 2 b = 0 t0 = 0 t1 = 2.0 pipeline_test_constants.c0 = 5 pipeline_test_constants.c1 = 6 }}}
This is a typical config file. You can have multiple section for different scripts, or for different operations with the same script -- the naming of sections is up to you anyway. Each section must contain a complete script configuration, listing all the options.
To get a complete starting configuration for your main script, run it in the browser and look at .tdl.conf. This will contain a complete configuration section under "[your script name]", which you can simply cut-and-paste into your batch config file. Alternatively, you can cut-and-paste from the meqtree.log file. Every time you run a script in the browser, an entry is added to meqtree.log (in that script's directory) containing the complete active configuration. (NB: due to a bug in pre-r7379 MeqTrees, the contents of meqtree.log were not entirely compatible with config files. This has now been fixed. If you must use the config from an older log file, replace "= False" with "= 0", and "= True" with "= 1".)
Let's go through our example control script (Timba/PyApps/test/run_pipeline_test.py) line-by-line. First there's a standard opening:
#!/usr/bin/pythonif name == 'main': from Timba.Apps import meqserver from Timba.TDL import Compile from Timba.TDL import TDLOptions``` Then we start a meqserver. The extra arguments are passed to the meqserver as is. You may want to read them from you control script's command line:
-
mqs = meqserver.default_mqs(wait_init=10,extra=["-mt","2"]); try:```
Once we're connected to a server, some cleanup is required before we can exit the script. Since we want to perform this cleanup regardless of whether the script ran to completion or was interrupted by an exception midway through, we use a try...finally block, with the cleanup code placed after a finally statement at the end of the script.
Then we attach to a config file and compile the main script:
-
TDLOptions.config.read("pipeline.tdl.conf"); script = "pipeline_test.py"; mod,ns,msg = Compile.compile_file(mqs,script);```
compile_file() returns a tuple of three values, of which we only really need the first -- a reference to the compiled module object. Why do we need the module object? Because we now want to run a TDL job, which is usually a function defined within the compiled module. The wait=True argument causes the function to not return to us until the job (inside meqserver) has been completed:
-
mod._test_forest(mqs,None,wait=True);```
If you're using a Calico or Siamese script, you may not know what the job function is. For this we have a provision for finding jobs by name or ID.
But first, some variations with the config file. The default behaviour is to use the config section "[script name]" (sans directory and suffix), which in this case is "[pipeline_test]". It is easy to tell compile_file() to use a different config section:
-
mod,ns,msg = Compile.compile_file(mqs,script,config="test_a"); TDLOptions.get_job_func('job1')(mqs,None,wait=True);```
Note how we use get_job_func() here to find the appropriate "job function" by its ID (if you look inside pipeline_test.py, you'll see how it defines job functions with IDs.) This is what you do when you don't know the function to call directly, as would be the case with somebody else's script (e.g. a Calico script.) How do you know the job ID? Run the job in the browser, and look inside meqtree.log, it'll tell you the name and ID of the job you just ran. You can also find jobs by their long name, as demonstrated here:
-
mod,ns,msg = Compile.compile_file(mqs,script,config="test_b"); TDLOptions.get_job_func('Run job 2')(mqs,None,wait=True);```
Finally, you can change config settings on-the-fly. This requires a little care, as compile_file() normally rereads the config file from disk. To get around this, we can manage configuration manually. First, we load initial values from a config section (save=False tells it to NOT save subsequent changes to .tdl.conf):
-
TDLOptions.init_options("test_b",save=False);```
then we make some config adjustments:
-
TDLOptions.set_option("a",5); TDLOptions.set_option("t0",-1); TDLOptions.set_option("t1",1);```
and now we recompile the script. config=None tells it to NOT reload the config file, but to use whatever config we have set up above:
-
mod,ns,msg = Compile.compile_file(mqs,script,config=None); mod._test_forest(mqs,None,wait=True);```
Runtime options can naturally be changed on-the-fly, no recompilation is necessary:
-
TDLOptions.set_option("t0",-5); TDLOptions.set_option("t1",5); mod._test_forest(mqs,None,wait=True);```
Finally, we need to stop and exit the meqserver. This is done as follows. Note that the finally keyword is part of our try...finally block explained above:
-
finally: # this halts the meqserver meqserver.stop_default_mqs(); print "Bye!";```
A more specific working example can be found under Siamese/batch-sim-example.py.