Implement #@import directive

[imagejan]

As discussed with @ctrueden, this should allow scripts to import the output objects of other scripts.

Given a script:

#@script(name="coolutils")
#@output imagefunctions
imagefunctions = {
  tile(a) {
  }
}

This should be importable like this:

#@import("coolutils")
imagefunctions.tile(a)

or with a local scope:

#@import("coolutils", scope="cool")
cool.imagefunctions.tile(a)

Subtasks:

• ImportProcessor implements ScriptProcessor
  • Harvest the import statements and stash them in the ScriptInfo as a property via info.set(String, String)
  • Probably stash them in the ScriptService in a map
• ScriptImportPreprocessor implements PreprocessorPlugin
  • If the module being preprocessed is not a ScriptModule, stop.
  • Otherwise:
    • for importName in ((ScriptInfo) module.getInfo()).getProperty("imports"):
      • moduleService.run(moduleService.getModuleByName(importName)
      • ((ScriptModule) module).getEngine() .getBindings(ScriptContext.ENGINE_SCOPE).putAll(outputs)
• Needed sub tasks:
  • ModuleService#getModuleByName: getModules().stream().filter(m -> name.equals(m.getName()).limit(1)...
  • ScriptInfo.getProperty(String) / ScriptInfo.setProperty(String, Object)

[imagejan]

I took a first attempt at implementing this on the script-imports-jan branch.

@ctrueden some open questions:

• I now followed your suggestion implementing ScriptInfo.setProperty(String, Object) to be able to set the "imports" property to another Map object that maps import modules to their desired scope. Maybe this is not necessary and we can use a simple property (as in #@script(name="foo", imports="myutils, barutils"))?? But then how to define the target scope?

• When I tried running a script from the script editor:

  #@script(name="myutils")
  #@output myfun
  
  myfun = {
    println 42
  }

  it currently still complains:
  [WARNING] [] Ignoring invalid parameter: script(name="myutils")

  Is this a matter of wrong priorities of the different script processors?

[ctrueden]

set the "imports" property to another Map object that maps import modules to their desired scope.

Perfect. Yes, this is what we should do, because as you say, we must attach the scope.

One thing you could do if you want to be more future-proof would be to invent a ScriptImport object that has a scope() attribute, and make the "imports" key point to a TreeMap<String, ScriptImport> rather than TreeMap<String, String>. If we do that, and then later support some additional attribute(s) in the #@import annotation, adding them to the ScriptImport class would be very easy without breaking the previous (admittedly implicit) contract.

Is this a matter of wrong priorities of the different script processors?

I believe so. I also noticed this problem but did not investigate yet. You could try putting priority = Priority.HIGH on the ScriptDirectiveScriptProcessor and see whether that fixes it.

[imagejan]

@ctrueden wrote:

One thing you could do if you want to be more future-proof would be to invent a ScriptImport object that has a scope() attribute, and make the "imports" key point to a TreeMap<String, ScriptImport> rather than TreeMap<String, String>.

Good point, I will do that.

You could try putting priority = Priority.HIGH on the ScriptDirectiveScriptProcessor and see whether that fixes it.

Indeed, see d48cc75. I also added a moduleService.addModule(info()); there, but we should agree how to handle the situation when the same module gets added twice, e.g. by running a #@script-annotated script from the script editor several times. I guess it would be best to update the registered module, i.e. remove and re-add it?!
And what about the case when different modules are registered with the same name? Should we only warn, or disallow/replace??

[ctrueden]

@imagejan Where I said TreeMap, I meant LinkedHashMap. The TreeMap keeps its keys in natural/sorted order. The LinkedHashMap keeps them in the order they were inserted, as desired here.

[imagejan]

Further progress on the script-imports-jan branch.

I can now run the following Groovy script:

#@script(name="utils", menuPath="")
#@output myfun

myfun = {
	println 42
}

and then run it from another one:

#@import("utils")

myfun() // will print: 42

This latter script will also work when run in Javascript, but not currently in Beanshell or Python, where it throws an exception, e.g. from Python:

TypeError: 'Script28$_run_closure1' object is not callable

---

Likewise, I can do with a Python script:

#@script(name="pyutil", menuPath="")
#@output myfun

def myfun():
	print 42

and run:

#@import("pyutil")

myfun()

which in turn doesn't work in Groovy (the myfun object is of class org.python.core.PyFunction, so I can run myfun.__call__() sucessfully).

---

So now I'd need your input again, @ctrueden:

• Can cross-language importability be achieved by wrapping those objects into Java Function objects? Would this be worth the effort?
• How can I implement the scope=() annotation? Creating a Java object that contains the imported objects?
• How do we gracefully handle multiple registration of the same ScriptInfo (as I asked in my post above)?

[ctrueden]

Can cross-language importability be achieved by wrapping those objects into Java Function objects? Would this be worth the effort?

I think so. Certainly we can tackle each case by case. For PyFunction specifically, the following code does the job:

import java.util.Arrays;
import java.util.function.Function;

import org.python.core.Py;
import org.python.core.PyObject;
import org.scijava.Context;
import org.scijava.script.ScriptModule;
import org.scijava.script.ScriptService;

public class PyFunctionAdapter {
	public static void main(final String... args) throws Throwable {
		final Context ctx = new Context();

		// Define a Python function as a script output.
		final String script = "#@output Object hello\n" + //
			"\n" + //
			"def hello(name):\n" + //
			"\treturn \"Hello, \" + str(name) + \"!\"\n" + //
			"\n";

		// Execute the script.
		final ScriptModule m = ctx.service(ScriptService.class).run("func.py",
			script, false).get();

		// Extract the Python function object.
		final Object hello = m.getOutput("hello");
		final PyObject pyFunc = (PyObject) hello;
		if (!pyFunc.isCallable()) {
			throw new IllegalStateException("expected callable Python object");
		}

		// Convert the Python function to a Java function.
		final Function<Object, Object> func = t -> pyFunc.__call__(
			t instanceof Object[] ? pyArgs((Object[]) t) : pyArgs(t));

		// Try out our shiny new Java function.
		System.out.println(func.apply("Curtis"));
	}

	private static PyObject[] pyArgs(final Object... o) {
		if (o == null) return null;
		return Arrays.stream(o).map(Py::java2py).toArray(PyObject[]::new);
	}
}

Some of the code above should become a case in the decode method of JythonScriptLanguage.

Getting late and I need sleep, so I'll try to tackle your other two questions tomorrow/soon.

[ctrueden]

How do we gracefully handle multiple registration of the same ScriptInfo (as I asked in my post above)?

Let's update the ScriptInfo#getIdentifier() method to use the script's name (via #getName()) when available. Right now, I think all scripts built from the Script Editor will be script:<inline>. We may also need to enhance it to use a hash code of the script for the inline stuff—e.g. script:<ab65c6e> or whatever. That way, two distinct iterations of a script in the Script Editor will be considered unique (because they are).

Once we do that, the ModuleService#addModule(ModuleInfo) method can be improved to always overwrite when the given ModuleInfo has the same ID as one already registered. Because that service is backed by a ModuleIndex, it could (but probably won't) be enough to override the #equals(Object) (and hashCode(), since they always go hand in hand) in all base ModuleInfo implementations to return true iff the getIdentifier() strings are equal. The reason I say it probably won't be enough is because I think the ObjectIndex being fundamentally List-driven means you can add duplicate elements by default—we'd have to decide which layer is the most appropriate to suppress that behavior.

I am out of time for today, but wanted to post this still-incomplete response. I will investigate making ModuleService#add work as desired as time allows, and also respond to your question about scopes next time.

[ctrueden]

I started working on the update to getIdentifier() behavior of ScriptInfo. See script-ids branch for current work. It is not yet fully working.

How can I implement the scope=() annotation? Creating a Java object that contains the imported objects?

Unfortunately, I still didn't have time to dig hard into this. However, my 2-minute idea is to use java.lang.reflect.Proxy and/or java.lang.reflect.InvocationHandler. More later.

[ctrueden]

Using Proxy won't work, because you have to know a priori which interface(s) you want it to proxy. It doesn't actually bake a new class with its own methods and/or fields. However, using Javassist like the following could work:

	private static long containerID;

	public static <T, R> Object createContainingObject(final Object object,
		final String fieldName) throws Exception
	{
		final ClassPool pool = ClassPool.getDefault();
		final CtClass ctc = pool.makeClass("ObjectContainer" + containerID++);
		ctc.addField( CtField.make("public Object " + fieldName + ";", ctc));
		final Class<?> c = ctc.toClass();
		
		final Object container = c.newInstance();
		c.getField(fieldName).set(container, object);
		return container;
	}

It would be nice not to drag in a javassist dependency, though. More importantly, the above won't solve the cross-language issue; it simply assigns an object into a field of another object instance.

The PyFunction-to-Function code I posted above is not enough to make the function usable as-is in other languages. E.g., if you take a Java Function and stuff it into a Python script engine, then try to call it by name, Jython is not smart enough to figure out what you mean. I.e.: you have to write myFunc.apply(...) rather than just myFunc(...). So I think using Java Function as our common data structure will not save us here.

My next idea is two invent two new script-language-agnostic classes: org.scijava.script.ScriptFunction and org.scijava.script.ScriptObject. The former would be the decoded version of a script language's function construct (e.g. in Python: a decoded PyFunction) while the latter would be the decoded version of a script language's container object construct (e.g. in Python: a decoded PyDerivedObject). Every language would need to be responsible for (re)encoding these two data structures into its own associated constructs. I think in practice this approach might work, but would be quite challenging to implement: in a few minutes of searching, for example, I saw no obvious way to extract the member functions and/or fields of a PyDerivedObject. And even if there was, you'd need to do some type conversion when a member function is called from a different language—the passed arguments would each need to be converted on the fly to the other language, then fed to the underlying function, for it to have a chance of working as expected.

So cross-language support for functions is much nastier than I thought. Let's stick to same-language usage for now. But cross-language could be a hackathon topic.

Finally, for the sake of completeness, and so my explorations just now are not lost, here is the code I was playing around with along several of the above lines:

import java.lang.reflect.Proxy;
import java.util.concurrent.ExecutionException;
import java.util.function.Function;

import org.python.core.PyObjectDerived;
import org.scijava.Context;
import org.scijava.script.ScriptService;

import javassist.ClassPool;
import javassist.CtClass;
import javassist.CtField;

public class Sandbox {

	private static long containerID;

	public static <T, R> Object createContainingObject(final Object object,
		final String fieldName) throws Exception
	{
		final ClassPool pool = ClassPool.getDefault();
		final CtClass ctc = pool.makeClass("ObjectContainer" + containerID++);
		ctc.addField( CtField.make("public Object " + fieldName + ";", ctc));
		final Class<?> c = ctc.toClass();
		
		final Object container = c.newInstance();
		c.getField(fieldName).set(container, object);
		return container;
	}

	public static void demo() throws Exception {
		final Context ctx = new Context();
		final ScriptService ss = ctx.service(ScriptService.class);

		// Create a Proxy object and see if we can use it in a script.
		ClassLoader loader = Thread.currentThread().getContextClassLoader();
		Class<?>[] interfaces = {java.util.List.class};
		Object proxyObject = Proxy.newProxyInstance(loader, interfaces, (proxy, method, args) -> {
			return "Result from " + method.getName() + " with " + args.length + " args";
		});
		final String proxyScript = "" + //
			"#@ Object functions\n" +
			"#@output String result\n" +
			"result = functions.get(5)\n"; // .get(x) works, but .gget(x) fails; not a List method
		final Object proxyResult = captureOutput(ss, "proxyTest.py", proxyScript, "result", "functions", proxyObject);
		System.out.println("proxy result = " + proxyResult);

		// Define a script that produces an object with functions.
		final String producingScript = "" + //
			"#@output Object functions\n" + //
			"class functions(object):\n" + //
			"    def greet(__self__, name):\n" + //
			"        return 'hello, ' + str(name)\n" + //
			"    def wave(__self__):\n" + //
			"        return '*waves goodbye*'\n" + //
			"functions = functions()";

		final Object functions = //
			captureOutput(ss, "producer.py", producingScript, "functions");

		// NB: Python object type is PyObjectDerived. If you leave
		// off the "functions = functions()" then it will be PyType.
		System.out.println("Python object type = " + functions.getClass());

		// Define and execute a Python script which uses the functions object.
		final String pythonScript = "" + //
			"#@input Object greeter\n" + //
			"#@output Object greeting\n" + //
			"greeting = greeter.greet('Chuckles')\n";
		final Object pythonGreeting = //
			captureOutput(ss, "script.py", pythonScript, "greeting", //
				"greeter", functions);
		System.out.println("Python greeting = " + pythonGreeting);

		// Where is the function inside the object? The dictionary is empty.
		System.out.println("Dict = " + ((PyObjectDerived) functions).getDict());

		// Define and execute a Groovy script which uses the functions object.
		final String groovyScript = "" + //
			"#@input Object greeter\n" + //
			"#@output Object greeting\n" + //
			"greeting = greeter.greet('Chuckles')\n";
		final Object groovyGreeting = //
			captureOutput(ss, "script.groovy", groovyScript, "greeting", //
				"greeter", functions);
		System.out.println("Groovy greeting = " + groovyGreeting);

		if (true) return;

		///////////////////////////////////////////////////////
		final Function<String, String> f1 = new Function<String, String>() {
			@Override
			public String apply(final String name) {
				return "[1]Hello, " + name;
			}
		};
		final Function<String, String> f2 = (String name) -> "[2]Hello, " + name;
		final Function<String, String> f3 = name -> "[3]Hello, " + name;

		final Object container1 = createContainingObject(f1, "greet");
		final Object container2 = createContainingObject(f2, "greet");
		final Object container3 = createContainingObject(f3, "greet");
	}

	private static Object captureOutput(final ScriptService ss, final String path,
		final String script, final String outputName, Object... inputs)
		throws InterruptedException, ExecutionException
	{
		return ss.run(path, script, false, inputs).get().getOutput(outputName);
	}

	public static void main(final String... args) throws Exception {
		demo();
	}

}

[oeway]

This features seems very cool! I don't fully understand. In the context of supporting SciJava script in ImJoy, I guess this can potentially enable the following:

1. take a python module, wrap it as a java object (myPythonModule) using the method mentioned in Convert Python function to Java Function scyjava#17 (question: we can potentially use a hashmap to contain member functions of the python class, right? is there better way to do this such that the python object will appear like a java object, i.e. support dot notion to retrieve properties and methods?)
2. pass the wrapped java object myPythonModule to a script execution context for running the script
3. in the scijava script, we can then do #@import("myPythonModule") to import the python module.

Am I correct? @ctrueden @imagejan

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/groovy-scripting-advices/46621/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/is-it-possible-to-supress-the-display-of-output-parameters/87991/4