Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quilt native Groovy #18

Closed
drernie opened this issue Oct 26, 2022 · 16 comments
Closed

Quilt native Groovy #18

drernie opened this issue Oct 26, 2022 · 16 comments
Assignees

Comments

@drernie
Copy link
Member

drernie commented Oct 26, 2022

Call Quilt from Groovy: bypass quilt3 command-line tool

Right now the plugin uses the command-line tool from the quilt3 pip package, specifically the 'install' and 'push' actions.
This is both an added dependency, and a 'blunt instrument'. For example, the only way to get a list of all the files in a package is to first install the package then ls the install directory.

Alternatives include:

  • Using "JavaEmbeddedPython" to call the Python API directly (which we tried initially, but failed due to threading issues)
  • Building a Java (Groovy) client for reading and writing Quilt manifests
@drernie drernie mentioned this issue Jan 20, 2023
@drernie drernie changed the title Bypass quilt3 command-line tool Call Quilt from Groovy: bypass quilt3 command-line tool Mar 28, 2023
@drernie drernie changed the title Call Quilt from Groovy: bypass quilt3 command-line tool Call Quilt from Groovy Jun 26, 2023
@drernie drernie changed the title Call Quilt from Groovy Quilt native Groovy Jun 26, 2023
@drernie
Copy link
Member Author

drernie commented Jun 26, 2023

  1. @dimaryaz will create a quiltcore-java
  2. nf-quilt will call that instead of quilt3
  3. Install
  4. Push
  5. List

@drernie
Copy link
Member Author

drernie commented Jun 26, 2023

Proposed Integration Plan

https://github.com/quiltdata/nf-quilt/blob/main/plugins/nf-quilt/src/main/nextflow/quilt/jep/QuiltPackage.groovy

End Result:

  1. First MIRROR the quilt3 calls with stub native calls
  2. Implement GET/INSTALL replacement
  3. Implementing PUT/PUSH replacement (*PATCH)
  4. Optional: Replace install+directory_walk with actual LIST operations

@drernie
Copy link
Member Author

drernie commented Jun 26, 2023

Implementation Proposal

  1. Ernie implements quiltcore.py using cross-platform abstractions
  2. Rigorously document that via a YAML file
  3. Dima leverages those to implement quiltcore.java

@drernie
Copy link
Member Author

drernie commented Jun 26, 2023

Things you can steal from nf-quilt:

  • build.gradle (sort of)
  • .github actions
  • groovy config

@drernie
Copy link
Member Author

drernie commented Jul 8, 2023

@drernie
Copy link
Member Author

drernie commented Jul 8, 2023

Almost:

22:19:43.595 [Test worker] DEBUG nextflow.plugin.PluginUpdater – Starting plugin nf-amazon version: 1.16.2
22:19:43.595 [Test worker] INFO  org.pf4j.AbstractPluginManager – Start plugin 'nf-amazon@1.16.2'
22:19:43.595 [Test worker] DEBUG org.pf4j.DefaultPluginFactory – Create instance for plugin 'nextflow.cloud.aws.AmazonPlugin'
22:19:43.599 [Test worker] ERROR org.pf4j.DefaultPluginFactory – nextflow.cloud.aws.AmazonPlugin
java.lang.ClassNotFoundException: nextflow.cloud.aws.AmazonPlugin

java.lang.NullPointerException: Cannot invoke "org.pf4j.Plugin.start()" because the return value of "org.pf4j.PluginWrapper.getPlugin()" is null
at org.pf4j.AbstractPluginManager.startPlugin(AbstractPluginManager.java:392)
at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:154)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:328)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:333)
at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:232)
at nextflow.plugin.PluginsFacade.setup(PluginsFacade.groovy:226)
at nextflow.quilt.quiltcore.S3Test.should create paths for s3(S3Test.groovy:42)

@drernie
Copy link
Member Author

drernie commented Jul 8, 2023

Different error if 'prod' versus 'dev'. Oddly:

22:27:45.027 [Test worker] DEBUG nextflow.plugin.PluginUpdater – Installing plugin nf-amazon version: 1.16.2
22:27:45.030 [Test worker] WARN  nextflow.file.FileHelper – Unable to start plugin 'nf-amazon' required by s3://quilt-example/.quilt
java.lang.UnsupportedOperationException: Install is not supported on dev mode - Missing plugin nf-amazon 1.16.2

java.lang.IllegalStateException: Missing plugin 'nf-amazon' required to read file: s3://quilt-example/.quilt
at nextflow.file.FileHelper.asPath(FileHelper.groovy:319)
at nextflow.file.FileHelper.asPath(FileHelper.groovy:273)
at nextflow.quilt.quiltcore.S3Test.should create paths for s3(S3Test.groovy:43)

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

Same problem as before:

18:18:09.310 [Test worker] INFO  org.pf4j.AbstractPluginManager – Start plugin 'nf-amazon@1.16.2'
18:18:09.310 [Test worker] DEBUG org.pf4j.DefaultPluginFactory – Create instance for plugin 'nextflow.cloud.aws.AmazonPlugin'
18:18:09.384 [Test worker] ERROR org.pf4j.DefaultPluginFactory – nextflow.cloud.aws.AmazonPlugin
java.lang.ClassNotFoundException: nextflow.cloud.aws.AmazonPlugin
	at org.pf4j.PluginClassLoader.loadClass(PluginClassLoader.java:144)
	at org.pf4j.DefaultPluginFactory.create(DefaultPluginFactory.java:46)
	at org.pf4j.PluginWrapper.getPlugin(PluginWrapper.java:81)
	at org.pf4j.AbstractPluginManager.startPlugin(AbstractPluginManager.java:392)
	at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:154)

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

api 'org.pf4j:pf4j:3.4.1'
api 'org.pf4j:pf4j-update:2.3.0'

https://github.com/pf4j/pf4j/blob/master/pf4j/src/main/java/org/pf4j/AbstractPluginManager.java

        log.info("Start plugin '{}'", getPluginLabel(pluginDescriptor));
        pluginWrapper.getPlugin().start();

https://github.com/pf4j/pf4j/blob/master/pf4j/src/main/java/org/pf4j/PluginWrapper.java

            plugin = pluginFactory.create(this);
   // Plugin create(PluginWrapper pluginWrapper);

https://github.com/pf4j/pf4j/blob/master/pf4j/src/main/java/org/pf4j/PluginClassLoader.java

            for (ClassLoadingStrategy.Source classLoadingSource : classLoadingStrategy.getSources()) {
                Class<?> c = null;
                try {
                    switch (classLoadingSource) {
                        case APPLICATION:
                            c = super.loadClass(className);
                            break;
                        case PLUGIN:
                            c = findClass(className);
                            break;
                        case DEPENDENCIES:
                            c = loadClassFromDependencies(className);
                            break;
                    }
                } catch (ClassNotFoundException ignored) {}

                if (c != null) {
                    log.trace("Found class '{}' in {} classpath", className, classLoadingSource);
                    return c;
                } else {
                    log.trace("Couldn't find class '{}' in {} classpath", className, classLoadingSource);
                }
            }

            throw new ClassNotFoundException(className);
private ClassLoadingStrategy classLoadingStrategy;

https://github.com/pf4j/pf4j/blob/master/pf4j/src/main/java/org/pf4j/ClassLoadingStrategy.java
https://www.includehelp.com/java/classloader-findclass-method-with-example.aspx

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

https://github.com/pf4j/pf4j/blob/master/pf4j/src/main/java/org/pf4j/PluginClassLoader.java

/**
 * One instance of this class should be created by plugin manager for every available plug-in.
 * By default, this class loader is a Parent Last ClassLoader - it loads the classes from the plugin's jars
 * before delegating to the parent class loader.
 * Use {@link #classLoadingStrategy} to change the loading strategy.
 *
 * @author Decebal Suiu
 */
 public class PluginClassLoader extends URLClassLoader {
    public PluginClassLoader(PluginManager pluginManager, PluginDescriptor pluginDescriptor, ClassLoader parent, ClassLoadingStrategy classLoadingStrategy) {
        super(new URL[0], parent);

        this.pluginManager = pluginManager;
        this.pluginDescriptor = pluginDescriptor;
        this.classLoadingStrategy = classLoadingStrategy;
    }
import java.net.URLClassLoader

https://github.com/pf4j/pf4j/blob/860b79710de3b30f2fb9b67a05d5a1aa4926debd/pf4j/src/main/java/org/pf4j/AbstractPluginManager.java#L227

    public String loadPlugin(Path pluginPath) {
        if ((pluginPath == null) || Files.notExists(pluginPath)) {
            throw new IllegalArgumentException(String.format("Specified plugin %s does not exist!", pluginPath));
        }

        log.debug("Loading plugin from '{}'", pluginPath);

        PluginWrapper pluginWrapper = loadPluginFromPath(pluginPath);

        // try to resolve  the loaded plugin together with other possible plugins that depend on this plugin
        resolvePlugins();

        return pluginWrapper.getDescriptor().getPluginId();
    }

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

                        case APPLICATION:
                            c = super.loadClass(className);
                            break;
                        case PLUGIN:
                            c = findClass(className);
                            break;
                        case DEPENDENCIES:
                            c = loadClassFromDependencies(className);
                            break;

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

Manually loading plugin finally failed on:

class AmazonPlugin extends BasePlugin {

    AmazonPlugin(PluginWrapper wrapper) {
        super(wrapper)
    }

    @Override
    void start() {
        super.start()
        FileHelper.getOrInstallProvider(S3FileSystemProvider)
    }
}

with:

java.lang.reflect.InaccessibleObjectException: Unable to make field private static volatile java.util.List java.nio.file.spi.FileSystemProvider.installedProviders accessible: module java.base does not "opens java.nio.file.spi" to unnamed module @29428716
	at nextflow.file.FileHelper.getOrInstallProvider(FileHelper.groovy:639)

The simplest solution might be to just catch the error and re-implement FileHelper.

@drernie
Copy link
Member Author

drernie commented Jul 9, 2023

Hmm. Others seem to have had this issue:
nextflow-io/nextflow#1963
nextflow-io/nextflow#1964

Installation != plugin requirement/activation. Installation means only downloading, unzipping and copy the plugin content into the HOME/.nextflow/plugins folder.

But, their solution still seems to (obviously) require running nextflow. Hmm.

@drernie
Copy link
Member Author

drernie commented Oct 14, 2023

Completed via quiltcore-java.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants