How to use the progress reporting functions

Reminder

• Start Fiji.
• Start the HPC Workflow Manager. (Plugins > HPC-ParallelTools > HPC Workflow Manager).
• Create a new directory on your local machine called "progress".
• Create an empty macro script called "progress.ijm" inside the empty directory you created.
• Create a new job with one (1) compute node selecting the macro script. (Right-click inside the table of the "HPC Workflow Manager" window and click the "Create a new job" context menu item. Select the macro script. Press the "Create" button.)

The macro code

Let's assume that we have recorded a macro that consists of three very slow image operations.

imageOperation1();
imageOperation2();
imageOperation3(); 

In order to guaranty that the "image operations" take a long time in any system we are going to use the wait(); function. Create imageOperation1, imageOperation2 and imageOperation3 like this:

function imageOperation1() {
  wait(5000); // wait 5 seconds
}

Often image operations need to be performed on a set of images. Let's assume we need to apply these image operations to 120 images.

images = 120;
for (i = 0; i < images; i++) {
  imageOperation1();
  imageOperation2();
  imageOperation3();
}

120 images * 3 operations * 5 seconds/operation = 1800 seconds = 30 minutes

As you can see macro scripts can take a lot of time to finish.

We need a way to report progress.

Fortunately, Parallel Macro makes this possible.

Now we will split the operations into tasks. The user determines the split as desired.

First we need to set descriptions to the tasks.

Add the following lines after the the function declarations to define the three tasks:

parAddTask("First op");
parAddTask("Second op");
parAddTask("Third op");

You must report all the tasks before you start changing their progress. To do this you need to add parReportTasks() after all desired tasks have been added and before you change the progress of any task.

Add calls to parReportProgress(); for each task to change the progress. Tasks have an auto-incremented id starting from zero (0).

The macro should now look like this:

function imageOperation1() {
  wait(5000); // wait 5 seconds
}

function imageOperation2() {
  wait(5000); // wait 5 seconds
}

function imageOperation3() {
  wait(5000); // wait 5 seconds
}

parAddTask("First op");
parAddTask("Second op");
parAddTask("Third op");
parReportTasks();

images = 120;
for (i = 0; i < images; i++) {
    imageOperation1();
    parReportProgress(0, (i+1)/images * 100);
    imageOperation2();
    parReportProgress(1, (i+1)/images * 100);
    imageOperation3();
    parReportProgress(2, (i+1)/images * 100);
}

To make the program more easily readable store the id of the tasks into variables like this:

task1 = parAddTask("First op");
task2 = parAddTask("Second op");
task3 = parAddTask("Third op");
parReportProgress(task1, (i+1)/images * 100);
parReportProgress(task2, (i+1)/images * 100);
parReportProgress(task3, (i+1)/images * 100);

Because in macro numbers are always of double type and the progress is integer you may want to set the progress to 100 once it is done to prevent setting the progress to 99 which can be confusing as it is not possible to tell if it was a rounding error or the task finished successfully.

function imageOperation1() {
  wait(5000); // wait 5 seconds
}

function imageOperation2() {
  wait(5000); // wait 5 seconds
}

function imageOperation3() {
  wait(5000); // wait 5 seconds
}

task1 = parAddTask("First op");
task2 = parAddTask("Second op");
task3 = parAddTask("Third op");
parReportTasks();

images = 120;
for (i = 0; i < images; i++) {
    imageOperation1();
    parReportProgress(task1, (i+1)/images * 100);
    imageOperation2();
    parReportProgress(task2, (i+1)/images * 100);
    imageOperation3();
    parReportProgress(task3, (i+1)/images * 100);
}
parReportProgress(task1, 100);
parReportProgress(task2, 100);
parReportProgress(task3, 100);

Reminder

• Save the final macro, or copy-paste the last code snippet and save the macro.
• Upload the data before starting the job. (Right-click the row of the job you created in the "HPC Workflow Manager" window and click the "Upload data" context menu item.)
• Start the job.

For future reference a table with all of the progress reporting functions can be found here.

video version 📺

main hub:house: previous 👈 next 👉