Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provides built-in GraalVM Reachability Metadata for core features of ShardingSphere JDBC and adds nativeTest-related unit test subsets #29000

Merged
merged 3 commits into from
Nov 16, 2023

Conversation

linghengqian
Copy link
Member

@linghengqian linghengqian commented Nov 10, 2023

Fixes #21347.

Changes proposed in this pull request:

  • This PR introduces the submodule org.apache.shardingsphere:shardingsphere-infra-reachability-metadata for hosting the GraalVM Reachability Metadata of ShardingSphere itself and some third-party libraries. This involves me personally not having much time in the central repository of GraalVM Reachability Metadata to handle PRs requested by the GraalVM Committer.
  • org.apache.shardingsphere:shardingsphere-infra-reachability-metadata will become an internal dependency of org.apache.shardingsphere:shardingsphere-jdbc-core to be passed out.
  • In order to check the usability under GraalVM Native Image, we need a subset of unit tests that avoids using third-party libraries such as Mockito that do not support use under Native Image to execute nativetest of GraalVM Native Build Tools. This subset of unit tests will be located at org.apache.shardingsphere:shardingsphere-infra-nativetest.
  • The unit test subset of org.apache.shardingsphere:shardingsphere-infra-nativetest is adapted from the ShardingSphere Example. Use YAML instead of Java API to define ShardingSphere's DataSource because this is closest to the actual application scenario - this is the actual meaning of native Test. We only focus on core features in these unit test.
  • The processing of org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest seems to have hit a bug in ReadWrite Splitting. I will discuss the phenomenon I saw in a separate issue later.
  • We introduce two Maven Profiles for this purpose, namely generateMetadata and nativeTestInShardingSphere. They both only manage pluginManagement. Submodules that need to use the relevant Profile will automatically introduce the relevant plugin. This involves best practices defined by GraalVM Native Build Tools. Refer to Maven Native plugin fails on POM packaging projects graalvm/native-build-tools#490 .
  • Introducing new documentation for using ShardingSphere JDBC with Native Image.
  • Introducing a nightly CI to check nativeTest. It seems inappropriate to put such checks in every PR, not all contributors are using GraalVM CE as the default JDK.

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error Log
[2/8] Performing analysis...  [********]                                                               (148.8s @ 2.93GB)
  29,136 (81.15%) of 35,902 types reachable
  49,445 (57.37%) of 86,186 fields reachable
 149,418 (58.31%) of 256,235 methods reachable
   7,787 types,   979 fields, and 6,877 methods registered for reflection
      68 types,    82 fields, and    57 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (16.4s @ 3.28GB)
[4/8] Parsing methods...      [***]                                                                      (9.3s @ 2.32GB)
[5/8] Inlining methods...     [****]                                                                     (4.0s @ 2.39GB)
[6/8] Compiling methods...    [********]                                                                (67.8s @ 2.69GB)
[7/8] Layouting methods...    [***]                                                                     (10.5s @ 2.88GB)
[8/8] Creating image...       [****]                                                                    (15.9s @ 3.58GB)
  73.72MB (54.04%) for code area:    94,894 compilation units
  57.31MB (42.01%) for image heap:  576,269 objects and 277 resources
   5.38MB ( 3.95%) for other data
 136.41MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  13.33MB calcite-core-1.35.0.jar                             16.03MB byte[] for code metadata
  13.17MB java.base                                            7.42MB byte[] for java.lang.String
   5.79MB h2-2.2.224.jar                                       7.18MB java.lang.Class
   5.66MB java.xml                                             5.80MB java.lang.String
   5.29MB groovy-4.0.10.jar                                    4.52MB byte[] for general heap data
   3.89MB org.apache.shardingsphere.sql.parser.autogen         2.45MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.96MB java.desktop                                         2.12MB byte[] for embedded resources
   1.91MB jackson-databind-2.14.0.jar                          1.74MB byte[] for reflection metadata
   1.54MB avatica-core-1.23.0.jar                              1.22MB java.lang.String[]
   1.43MB janino-3.1.9.jar                                     1.05MB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
  18.06MB for 727 more packages                                7.56MB for 5238 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 AWT:  Use the tracing agent to collect metadata for AWT.
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                       36.5s (12.0% of total time) in 181 GCs | Peak RSS: 5.27GB | CPU load: 4.20
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_headless.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_xawt.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libfontmanager.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjava.so (jdk_library_shim)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjavajpeg.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjvm.so (jdk_library_shim)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/liblcms.so (jdk_library)
 /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests (executable)
========================================================================================================================
Finished generating 'native-tests' in 5m 1s.
[INFO] Executing: /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests --xml-output-dir /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-test-reports -Djunit.platform.listeners.uid.tracking.output.dir=/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/test-ids
JUnit Platform on Native Image - report
----------------------------------------

18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - This is logback-classic version 1.3.11
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - No custom configurators were discovered as a service.
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 0 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.xml]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Found resource [logback.xml] at [resource:/logback.xml]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs multiple times on the classpath.
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml#1]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml]
18:08:36,098 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@3f7d7df1 - URL [resource:/logback.xml] is not of type file
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - <level> element is deprecated. Near [level] on line 31
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please use "level" attribute within <logger> or <root> elements instead.
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.ContextNameModelHandler - Setting logger context name as [shardingsphere-infra-nativetest]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.LevelModelHandler - ROOT level set to INFO
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@12459199 - End of configuration.
18:08:36,099 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@85c50cc - Registering current configuration as safe fallback point
18:08:36,099 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 1 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY

org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest > testShardingInLocalTransactions() FAILED

org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest > testReadWriteSplittingInLocalTransactions() FAILED

org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest > testEncryptInLocalTransactions() FAILED

org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest > testMaskInLocalTransactions() FAILED

org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest > testShadowInLocalTransactions() FAILED


Failures (5):
  JUnit Jupiter:ShardingTest:testShardingInLocalTransactions()
    MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest', methodName = 'testShardingInLocalTransactions', methodParameterTypes = '']
    => java.nio.file.FileSystemNotFoundException
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
       java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
       java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
       org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
       [...]
  JUnit Jupiter:ReadWriteSplittingTest:testReadWriteSplittingInLocalTransactions()
    MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest', methodName = 'testReadWriteSplittingInLocalTransactions', methodParameterTypes = '']
    => java.lang.NullPointerException
       java.base@17.0.9/java.io.Reader.<init>(Reader.java:168)
       java.base@17.0.9/java.io.InputStreamReader.<init>(InputStreamReader.java:112)
       org.apache.shardingsphere.infra.nativetest.FileTestUtils.readInputStream(FileTestUtils.java:52)
       org.apache.shardingsphere.infra.nativetest.FileTestUtils.readFromFileURLString(FileTestUtils.java:46)
       org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest.testReadWriteSplittingInLocalTransactions(ReadWriteSplittingTest.java:53)
       java.base@17.0.9/java.lang.reflect.Method.invoke(Method.java:568)
       java.base@17.0.9/java.util.ArrayList.forEach(ArrayList.java:1511)
       java.base@17.0.9/java.util.ArrayList.forEach(ArrayList.java:1511)
  JUnit Jupiter:EncryptTest:testEncryptInLocalTransactions()
    MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest', methodName = 'testEncryptInLocalTransactions', methodParameterTypes = '']
    => java.nio.file.FileSystemNotFoundException
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
       java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
       java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
       org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
       [...]
  JUnit Jupiter:MaskTest:testMaskInLocalTransactions()
    MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest', methodName = 'testMaskInLocalTransactions', methodParameterTypes = '']
    => java.nio.file.FileSystemNotFoundException
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
       java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
       java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
       org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
       [...]
  JUnit Jupiter:ShadowTest:testShadowInLocalTransactions()
    MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest', methodName = 'testShadowInLocalTransactions', methodParameterTypes = '']
    => java.nio.file.FileSystemNotFoundException
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
       org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
       java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
       java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
       org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
       org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
       org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
       [...]

Test run finished after 98 ms
[         6 containers found      ]
[         0 containers skipped    ]
[         6 containers started    ]
[         0 containers aborted    ]
[         6 containers successful ]
[         0 containers failed     ]
[         5 tests found           ]
[         0 tests skipped         ]
[         5 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         5 tests failed          ]

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…ShardingSphere JDBC and adds nativeTest-related unit test subsets
@strongduanmu strongduanmu added this to the 5.4.2 milestone Nov 14, 2023
Copy link
Member

@strongduanmu strongduanmu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your perfect pr. @linghengqian Merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make ShardingSphere Proxy in GraalVM Native Image form available
3 participants