WG - Restructuring Quarkus Core Repository

[cescoffier]

Hello,

I would like to start a discussion and maybe even a working group about restructuring/reorganizing Quarkus Core, subject team members have discussed on and off for at least four years.

I won't tackle the core of the core (well, a little bit), but primarily extensions.

Current Context and Problem Statement

An extension in Quarkus Core is located in ./extensions and comprises at least two modules: deployment (the build time part) and runtime (the runtime part).

We currently have 170 extensions in the core (a rough estimate; we probably have more). Some were added in Quarkus Core because we didn't have the Quarkiverse then, and some were added here because they required changes in Quarkus to work.

With 170 extensions, it's a LOT of build items. Another rough estimation gave me 454 (again, rough estimate). There is no real distinction between build items intended only to be used in the extension itself and build items designed to be used by other extensions. Occasionally, we created deployment-spi modules containing the public build items. In most cases, it was not done to separate public from private build items but to allow the extension to contribute to another extension without forcing a dependency on that extension.

Finally, another issue extension writers often have is importing the whole Quarkus project to work on a specific extension. You can load just one extension in your IDE; it works well. But when you have integration tests, they're in another location, quite far from the extension itself.

So, this discussion is meant to improve these pain points. As I said, it's been discussed for a long time, but we have yet to act on it. With Quarkus gaining maturity, it's also time to do some cleanup and reorganization to ease the development.

Some ideas

These ideas are not exhaustive or exclusive. These are just points that have been mentioned and discussed several times.

Moving extensions to Quarkiverse

Several extensions from Quarkus Core were contributed before the Quarkiverse existed. The Quarkiverse is the best place to develop and maintain Quarkus extensions. It has an agile release process (allowing quick fixes), automatic testing against multiple Quarkus versions (catching up regression and increasing compatibility), and its own documentation mechanism.
Quarkiverse extensions can still be listed on code.quarkus.io, provide code start, etc.

Moving an extension to Quarkiverse does not "break" users' use of the extension. Thanks to relocations, we can avoid disruptions.

We should encourage extension owners to consider that move. It's only possible for extensions not used by another extension in Quarkus. We absolutely need to avoid having the Quarkus Core repository depend on artifacts hosted on Quarkiverse.

Restructuring extensions

As mentioned, most extensions contain two modules (deployment and runtime). However, this does not allow distinguishing private build items from public build items.

We should consider adding a deployment-spi module to all extensions. This module would contain a public build item and provide build items that other extensions can produce or consume.

Thus, we would have something like this:

• runtime
• deployment: contains the processors and the private build item. This module depends on the deployment-spi module
• deployment-spi: contains the public build items and associated data structures.
• spi: an optional module containing the runtime public API and SPI if needed

It also "breaks" the dependency between two extensions. It can be prevented by keeping the dependency on the deployment module (which depends on the deployment-spi module anyway).

This change would also improve our "all build items" page by listing only the public ones from the SPI modules, avoiding the trap of using build items not intended for use outside the extension.

This change can be done in two phases to avoid breaking existing extension:

1. First, create new build items in the SPI module and modify the extension to support the old build item (from the deployment module) and the new one.
2. Deprecate the old build items with a plan for removal.
3. Remove the old one after enough baking time.

Note that the deployment and deployment-spi modules must not share the same packages (split packages are bad). For example, the deployment-spi module should contain a package name: io.quarkus.<extension>.deployment.spi. The deployment module should contain packages under io.quarkus.<extension>.deployment (but not the spi sub-package).

While talking about packages, in the runtime module, the io.quarkus.<extension> package should contain the public API and data structures. The implementation code should be under io.quarkus.<extension>.runtime.

Distinguishing public from private API in Quarkus Core

I know I said above that this discussion was only about extensions. Still, I was reminded (hint: it was Guillaume) that our core APIs do not distinguish between public and private (including public and private build items).

We should start restructuring these APIs to provide a more precise separation. Separating public APIs, public SPIs (public build items), and private content would help avoid misusages and allow us to document the public ones more carefully. In practice, it would require creating a core/deployment-spi module to store the public build items. In addition, a core/core-api module would also be created to store the public APIs currently in core/runtime.

We have developed many “utils” methods over the last five years. They are spread across the whole project and often duplicated in multiple extensions. Creating a new module (or multiples) offering these utility methods would help us minimize the number of duplicated code.

Moving some annotations into plain Jars, which do not depend on anything from Quarkus, could also allow external projects to improve their integration with Quarkus. For example, if we had a plain (no dependency) jar providing the @RegisterForReflection, it would allow projects to use it without having a hard dependency on Quarkus itself.

Moving integration tests closer to the extension

First, I'd like to clarify that this is not about moving ALL the integration tests to their respective extensions. It's only for new and existing integration tests testing a single extension.

The advantage of integrating tests is that they can be run in JVM and native mode. Thus, they can quickly detect issues with native compilation or differences in behavior. Integrating tests closer to the extension provides a better developer experience for the occasional extension maintainer.

In practice, a new integration-tests module in the extension directory with a specific parent would be required to configure the integration test cohesively. This integration-tests module can be a multi-module. Typically, Quarkus REST (aka RESTEasy Reactive) or gRPC have many integration tests in different modules.

That being said, this proposal has a significant drawback for the CI build. Right now, the CI has specific jobs for native integration tests. This would make the configuration of that CI job more complicated. We will have to discuss this, and the pros/cons highlighted. We should start with a feasibility study with a few extensions (one with a single module and one with multiple modules) to better understand the consequences.

Exploding the Quarkus Core repository

This idea comes from the very early days of Quarkus. At that time, we decided to use a single repo to simplify maintenance, refactoring, and release of Quarkus.

However, the idea of exploding the mono-repository into domain-specific repositories (like data or messaging) was discussed multiple times.

I'm still determining if we reached the maturity required to implement this move, but I wanted to mention it here.

Expected outcomes

Of course, the expected outcome is a better Quarkus core repository, but also:

• a clearer distinction between public and private API and build items - thus, improving our documentation
• an easier way to develop extensions from Quarkus Core
• a set of guidelines (probably an ADR) explaining the package names and structures

I probably forgot about other proposals. Feel free to mention them. In addition, let's discuss what you think is doable, what makes sense, and what does not.

[cescoffier]

CC @dmlloyd, @gsmet , @geoand, @maxandersen, @holly-cummins

[holly-cummins]

One benefit of moving some of our core extensions out to the Quarkiverse is that it would force us to sort out a few things, like a clearer policy on the criteria for when we mention quarkiverse extensions on quarkus.io guide pages, and a stronger set of opinions and tooling on branching policies, cross-version testing, and tooling. It would also be a good trigger for us to make the distinction between platform and core clearer (assuming that some of the extensions we move out of core are platform ones and stay as platform ones).

I like the idea of public and private build items, too. I'd be sad if we made too many of the current ones private, and so ended up forcing extension authors to roll their own equivalents; but given how many we currently list in https://quarkus.io/guides/all-builditems, the risk of that is low.

[maxandersen]

Just so we don't forget - things does not necessarily need to move to quarkiverse. Could also just be staying in quarkusio org when that makes most sense.

The issue emmanuel mention about docs missing quarkiverse content is not really limited to quarkiverse. It's a concesquence of Quarkus docs publishing being tied purely to Quarkus core repo - where one could argue it would make more sense we made documentation tied to platform(s).

Then the published site could follow a more consistent structure.

Alternative is that we decouple/split doc publishing so it's always aren't tied at the hip by Quarkus core releases.

[geoand]

In most cases, it was not done to separate public from private build items but to allow the extension to contribute to another extension without forcing a dependency on that extension.

I would say in all cases currently :)

[geoand]

For the issue of private vs public build items, one way to start would be to introduce some annotations on the build items that our annotation would handle.

Obviously it would be interim solution that would nonetheless take a non trivial effort to introduce as we would need to go through each and every one

[emmanuelbernard]

One big drawback of the Quarkiverse in the context of documentation is that they don't show up in quarkus.io making for a very disconnected experience when trying to find docs to start something or figure some problem out. If some move happen, I would like for us to find a way to make things easier for Quarkus users to find information on what's in the platform.

[gsmet]

One big drawback of the Quarkiverse in the context of documentation is

that they don't show up in quarkus.io making for a very disconnected experience when trying to find docs to start something or figure some problem out. If some move happen, I would like for us to find a way to make things easier for Quarkus users to find information on what's in the platform. Yeah that's part of the work on the search engine: quarkusio/search.quarkus.io#143 . That being said, I really think we should take a step back and reorganize the doc index page. It's not really browsable atm and I have been pushing the idea of using faceting for a while to better show the depth of what we have. If we use faceting, we could expose everything that is in the search engine. I would say this discussion is a bit orthogonal to the goals of this WG.

…

On Tue, Sep 10, 2024 at 1:46 PM Emmanuel Bernard ***@***.***> wrote: One big drawback of the Quarkiverse in the context of documentation is that they don't show up in quarkus.io making for a very disconnected experience when trying to find docs to start something or figure some problem out. If some move happen, I would like for us to find a way to make things easier for Quarkus users to find information on what's in the platform. — Reply to this email directly, view it on GitHub <#43130 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYOBNZ7NB2JKWHYNRHWG3ZV3LXZAVCNFSM6AAAAABN4DXII6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRQGE3DANQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cescoffier]

+1 to what @gsmet said.
With the current layout, it's hard to include quarkiverse extensions.
It would be a big win if we could include the latest compatible version of an extension's documentation on the quarkus.io page. Even if it's just an iframe, it would provide a better experience than we have today.

However, I'm not sure we need to wait for this before reducing the number of core extensions. It feels like a way to procrastinate again and do nothing while the current situation is close to being out of control.

[michalvavrik]

However, I'm not sure we need to wait for this before reducing the number of core extensions. It feels like a way to procrastinate again and do nothing while the current situation is close to being out of control.

Fact that core project has so many extensions is making my work easier. Often my changes have impact on multiple extensions. I can see CI passing for them without doing extra work in separate PRs. But main reason why I like it that it makes development so easy, you can check other invocations or usage of methods, datatypes. You will never stick to whatever worked for you before because you can constantly see others doing stuff better. If I had to check to tens of projects before I make some change how they do things, I think I'd just do things the old way.

+1 for separation of concerns and clear, stable, well defined API / SPI
-1 for moving extensions out of https://github.com/quarkusio/quarkus