Improve support for resource registration #3315

jovanstevanovic · 2021-03-29T12:15:40Z

Task and high-level overview (#3326)

PROGRESS:

First step toward creating a custom filesystem with a TODO list is committed.

graalvmbot · 2021-03-29T12:20:06Z

Hello jovanstevanovic, thanks for contributing a PR to our project!

We use the Oracle Contributor Agreement to make the copyright of contributions clear. We don't have a record of you having signed this yet, based on your email address jovan -(dot)- stevanovic1 -(at)- outlook -(dot)- com. You can sign it at that link.

If you think you've already signed it, please comment below and we'll check.

christianwimmer

So this PR removes some substitutions that are not necessary because the JDK code eventually calls a method that remains substituted. I'm curious - where is the actual "root" in the JDK that remains substituted?

christianwimmer · 2021-03-29T19:36:21Z

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/JavaLangSubstitutions.java

-    private static URL findResource(String mn, String name) {
-        return ClassLoader.getSystemClassLoader().getResource(name);
-    }
+    @AnnotateOriginal


What is the AnnotateOriginal for? You are not adding any annotations, so this is not going to do anything

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/ResourceAttributes.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/ResourceAttributesView.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/ResourceFileSystemProvider.java

graalvmbot · 2021-04-15T12:00:22Z

Hello Jovan Stevanovic, thanks for contributing a PR to our project!

We use the Oracle Contributor Agreement to make the copyright of contributions clear. We don't have a record of you having signed this yet, based on your email address 44313413+jovanstevanovic -(at)- users -(dot)- noreply -(dot)- github -(dot)- com. You can sign it at that link.

If you think you've already signed it, please comment below and we'll check.

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/FileSystemProviderSupport.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/ResourceAttributes.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/ResourceStorageEntry.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/ByteArrayChannel.java

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

...c/com.oracle.svm.test/src/com/oracle/svm/test/NativeImageResourceFileSystemProviderTest.java

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ResourcesFeature.java

vjovanov · 2021-05-25T17:18:53Z

...evm/src/com.oracle.svm.core.jdk8/src/com/oracle/svm/core/jdk/resources/ByteArrayChannel.java

+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class ByteArrayChannel implements SeekableByteChannel {


Can we simply ignore the JDK 8 implementation? Would it still work for the set of programs that worked before this PR?

vjovanov · 2021-05-25T17:19:25Z

...m/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/JDKVersionSpecificResourceBuilder.java

+import java.net.URL;
+import java.net.URLConnection;
+
+public class JDKVersionSpecificResourceBuilder {


Then we could skip all of this as well.

gradinac · 2021-06-09T09:02:40Z

...tevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/ResourceStorageEntry.java

+        return isDirectory;
+    }
+
+    public void setDirectory(boolean directory) {


This and setData don't seem to be used - I would remove them

gradinac · 2021-06-09T09:17:01Z

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ResourcesFeature.java

+ * </p>
+ *
+ * <p>
+ * ResourceFileSystemProvider exposes methods for creating a new file system, getting existing one


I would tweak this comment a bit:

The {@link NativeImageResourceFileSystemProvider} provides most of the functionality of a {@link FileSystem}. It is an in-memory file system that upon creation contains a copy of the resources included in the native-image. Note that changes to files do not affect actual resources returned by resource manipulation methods like `Class.getResource`. Upon being closed, all changes are discarded.

gradinac · 2021-06-09T09:22:05Z

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ResourcesFeature.java

+ * </p>
+ *
+ * <p>
+ * Most of operations mentioned above aren't accessed directly i.e. by calling methods from this


I would tweak this one as well:

As with other file system providers, these methods provide a low-level interface and are not meant for direct usage - see {@link java.nio.file.Files}

gradinac · 2021-06-09T09:24:17Z

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

+                    new HashSet<>(Arrays.asList("basic", "resource")));
+
+    private final Set<InputStream> streams = Collections.synchronizedSet(new HashSet<>());
+    private final Set<Path> tmpPaths = Collections.synchronizedSet(new HashSet<Path>());


<Path> is redundant here -> <>

gradinac · 2021-06-09T09:25:09Z

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

+ * <p>
+ * Some parts of code from this class are copied from jdk.nio.zipfs.ZipFileSystem with small tweaks.
+ * The main reason why we cannot reuse this class (without any code copying) is that we cannot
+ * extend jdk.nio.zipfs.ZipPath which is central class in Zip file system.


is central class in Zip file system -> is the central class in the Zip file system

gradinac · 2021-06-09T10:10:09Z

.../com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileStore.java

+        private final long size;
+
+        ResourceFileStoreAttributes(NativeImageResourceFileStore fileStore) throws IOException {
+            Path path = FileSystems.getDefault().getPath(fileStore.name());


If I understood correctly, a NativeImageResourceFileStore is bound to a path inside of the native-image file system. However, here we seem to be resolving that path in the default file system - I may have missed something, but it seems like this file isn't connected to the native-image file system and the information here, provided that such a path exists, most likely isn't what we're looking to return

gradinac · 2021-06-09T10:25:37Z

...ratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/Target_java_lang_ClassLoader.java

    private static native void initSystemClassLoader();

-    @Substitute
-    private URL getResource(String name) {


To make sure I understand these correctly, we've pushed these substitutions to the appropriate class loaders - we now respect the class loader hierarchy when looking up resources

Correct. That was the idea at some point, but due to a lack of module support, I was forced to return almost all substitutions. Once we get full module support, we will surely push substitutions to appropriate class loaders.

gradinac · 2021-06-09T10:38:01Z

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

+            } else {
+                checkParents(dst);
+            }
+            Entry target = new Entry(eSrc, Entry.COPY);  // Copy eSrc entry.


When we do a copy of the entry, we may either create a copy of the original entry's bytes, or, if the previous entry is getting deleted, take the original entry's bytes directly. In both cases, however, copyOnWrite remains false - if we attempt to write data to this entry (using getBytes), it seems like we will accidentally reload the entry - should we also flip the copyOnWrite flag to true here?

Nice catch. That could cause a bug in the future.

gradinac · 2021-06-09T10:43:55Z

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

+        if (!isTrue(env, "create")) {
+            throw new FileSystemNotFoundException(resourcePath.toString());
+        }
+        this.useTempFile = isTrue(env, "useTempFile");


If I understood it correctly, this option allows us to create temporary files when modifying a file system entry?
Would such a flag be required? This seems to only be used when attempting to create new output streams, but it may help reduce complexity if we remove it and keep everything in memory

Correct. No, it's not required, we can get rid of this flag and simplify the operation of creating a new output stream and keep everything in memory.

gradinac · 2021-06-09T10:45:46Z

...com.oracle.svm.core/src/com/oracle/svm/core/jdk/resources/NativeImageResourceFileSystem.java

+        } else {
+            os = new ByteArrayOutputStream((e.size > 0) ? e.size : 8192);
+        }
+        return new EntryOutputStream(e, os);


If we keep everything in memory, would it make sense to also close the output streams when closing the file system?

Agree. I overlooked it. We are keeping all information about opened input streams, so we should do the same thing for output streams.

…ystem.

christianwimmer reviewed Mar 29, 2021

View reviewed changes

jovanstevanovic mentioned this pull request Apr 8, 2021

Improve support for resource registration #3326

Closed