[GR-56101] JSON Metadata Versioning, Backwards Compatibility, and Evolution

[vjovanov]

TL;DR

The reachability metadata JSON files must evolve to completely cover the Java semantics. The most important upcoming features
are: type for proxy and lambda classes #7476, new conditions such as typeReached #7480, and the new resource-config.json format #7487.

This ticket aims to make the transition to the new features in JSON files smooth. This smooth transition will be achieved with the following goals.

Goals

1. Provide a JSON schema for all JSON-file versions starting from GraalVM for Java 17. The JSON schema will be versioned according to the GraalVM version.
2. Guarantee that the latest JSON schema is compatible (adheres to) all previous schema versions (backwards compatibility).
3. Guarantee that latest release of all actively maintained branches (e.g., 23.0.x, 23.1.x, 24.0.x) are compatible with the schema provided by the latest GraalVM (forward compatibility). This still means that users might need to update to the latest minor version. The build tools can be used to provide a reminder that this is necessary.
4. Evolve metadata in such way that forward compatibility is kept even beyond the latest release of all actively maintained branches. Old versions will interpret elements in a limited form. On example of "condition" elements, all conditions unknown by previous versions will be interpreted as true so that the element will be present in the image.
5. Introduce a test suite that will test the JSON versions according to backward compatibility and forward compatibility guarantees.
6. During the Native Image build, compare JSON files to the known versions; if the JSON schema is outdated suggest to the user to migrate to latest GraalVM versions.

Non-Goals

1. Require the community to maintain multiple versions of JSON files.
2. Break code that uses older versions of the reachability metadata JSON files.

Implementation

Introduce Backward and Forward Compatibility Policy for Metadata Files

Native Image will be backwards compatible with previous metadata-file versions as long as there are cornerstone libraries in the community that use the previous JSON versions. This is likely going to be a long period, if not forever.

Native Image will be forward compatible wrt. to metadata files with latest release of all actively maintained branches to the current version. The forward compatibility will also be semantic whenever possible: the new JSON elements will also be functional in the previous supported major CE versions.

Implement a Test Suite That Checks Metadata From all Previous Supported Major CE Versions

To assure that backward and forward compatibility is preserved we will introduce a test suite that contains all possible metadata elements. This test suite will be executed according to the compatibility guarantees. On the example of GraalVM for JDK 23 the test suite would test:

1. Metadata from JDK 17, 18, 19, 20, 21, 22 against the JDK 23
2. Metadata from JDK 23 with JDK 21 and JDK 17.

Versioning and Warning Messages

JSON format versioning is needed to warn users of previous GraalVM versions about using unsupported elements defined in future versions. Since JSON files are human written, adding a version field to the JSON files has only informative value: users can make mistakes and confuse the Native Image builder, they need to bump the version every time they use the new GraalVM, etc.

This proposal does not introduce the version field as the minimum-required version is defined by the contents of a JSON file. Native Image will match the consumed metadata files to all supported JSON schemas. If no known schemas match the file, and the file can still be parsed, a warning message for upgrade will be presented:

The 'app.jar!/META-INF/native-image/app/reflect-config.json' file does not conform to the know schema for the current GraalVM version. Please consider updating to the latest GraalVM version: <link-to-download>. 

Due to forward compatibility guarantees, the latest release of all actively maintained branches should never report
the message above.

If the version is supported, but it uses deprecated JSON elements, Native Image (or the build tools) will suggest to use the migration tool to update files to the latest version.

The Metadata Migration Tool

native-image-configure will support updating the input JSON files to the latest version:

native-image-configure --upgrade-metadata <path_to_file_or_folder> [--output-dir <path-to-output-dir>]

Schema Version in the Shared Metadata Repository

The metadata repository will move to the latest schema version with a graceful period after the latest release of all actively maintained branches. This will give time to the users to bump the GraalVM version.

Since this metadata is supported by the relevant releases, this should not negatively affect the community. If a project is using unsupported versions of GraalVM and it will have to update to the supported versions to be guaranteed.