Metadata binding

[sebersole]

https://hibernate.atlassian.net/browse/HHH-16114
https://github.com/sebersole/hibernate-annotation-source

NOTE: Much of this is based on the older https://github.com/sebersole/hibernate-orm/tree/jandex-binding branch

This is discussion about binding metadata sources (annotated classes and xml) into the org.hibernate.mapping "boot model". More specifically, this is about what we currently call "annotation binding" to differentiate it from hbm.xml binding even though "annotation binding" also includes handling of orm.xml files.

There are a few moving and inter-related parts to this proposal.

As a background, annotation binding currently uses HCANN to process the annotations along with its MetadataProvider hook to apply orm.xml metadata. To be clear, HCANN itself does not do anything related to the xml - ORM itself does all of that via the hook. It also eagerly uses reflection.

So a few (inter-related) goals -

1. Support for dynamic models.
2. Remove direct reliance on reflection.
3. Remove direct reliance on Annotation.
4. Remove direct exposure of external libraries (HCANN) in the binding code.
5. Better organize the binding code for logic and maintainability.
6. Prepare for eventual move from HCANN to Jandex

Although not a direct goal, all this work also presents the possibility for a potential simplified model for "additional mapping" info as used by Envers, Hibernate Search, etc. At the moment, these providers are forced to use JAXB model based on hbm.xml

At a high-level, the proposal is for a multi-step process for binding - from source -> intermediate -> boot. I have a PoC that

1. defines the source model, as well as building one from both HCANN and reflection
2. defines intermediate, as well as transformation from source

The PoC does not yet build the boot model, though a previous Jandex PoC defines much of that using very similar model and approach.

Metadata Source

At the "lowest level" in this processing we have abstractions for handling "reflection" information as well as "annotations".

Abstraction for Annotation

https://hibernate.atlassian.net/browse/HHH-15996

This is proofed already. It is largely based on 2 contracts -

1. AnnotationDescriptor which describes aspects of the annotation class (is it repeatable, etc)
2. AnnotationUsage which is a particular usage of a given AnnotationDescriptor. This correlates to Annotation

An ancillary contract is AnnotationTarget which exposes access to AnnotationUsage references from a "target" (package, class, field, ...)

Abstraction for Class

https://hibernate.atlassian.net/browse/HHH-16012

This is also proofed already and is defined by -

1. PackageDetails
2. ClassDetails
3. FieldDetails
4. MethodDetails

which all act as AnnotationTarget.

This step alleviates the reliance on reflection (it is targeted reflection abstraction) and supports dynamic models. It also acts as an abstraction for the sources of these details (HCANN, Jandex, etc) and therefore supports migration between these sources.

This model is designed as an abstraction over reflection to support dynamic models, as well as Hibernate's generalized entity-name feature which might map a single class with multiple names.

This model is one potential solution for "additional mapping" info.

Binding

Binding here is defined as the process of creating the org.hibernate.mapping references.

Currently this happens via complicated static calls that stash details in various Maps as intermediate steps. The proposal is to create an actual intermediate model that is a generalized categorization of the model.

Intermediate Model

When I say "generalized categorization" this means that we understand that one ClassDetails represents an entity while another represents a converter e.g.

This is where my PoC currently stands so this is very much in flux. But at the moment, we have:

• EntityHierarchy
• EntityTypeMetadata
• EmbeddableTypeMetadata
• AttributeMetadata
• ConversionMetadata
• AttributeOverrideMetadata
• AssociationOverrideMetadata

These contracts also expose the underlying "source" model parts for access to AnnotationUsage, etc. where appropriate

Currently, the understanding of the categorization of an attribute (basic, embedded, etc) is simply an enum. Largely an implementation detail, but important to note.

This model also performs some basic interpretation such as resolving mapping inheritance type, hierarchies, caching, etc

This model is another potential solution for "additional mapping" info.

Boot Model

Here, boot model is the org.hibernate.mapping model which is ultimately the target for processing mapping sources.

We'd build this boot model using the intermediate model along with its underlying source model.

This part is still largely undefined in my PoC. I have most of RootClass binding done aside from attributes.

[yrodiere]

Hey,

I think this goes in the direction we discussed a few days ago, so I don't have much to add except minor details.

Abstraction for Class

I know this is in flux and there's enough work already, so I'm not asking for the plan right now but more potentially, in the medium term.

Is this intended to be generic and potentially used by other projects (like HCANN was), or to stay in ORM forever and potentially have ORM-specific bits? I'm not counting the HCANN/Jandex implementations as ORM-specific, as those could work for other projects as well.

I'm asking because this seem similar to work we did in HSearch (though there's no concept of package in HSearch, and we don't differentiate fields/methods, there's just properties). I feel we often have similar needs and might benefit from merging our code at some point in the future, be it only to ensure consistent behavior when interpreting annotations in both projects.

Another reason for merging would be complexity: believe it or not, some of the type resolution stuff can be relatively complex and while HCANN is able to do it to some extent (not sure how well), Jandex is not, to the best of my knowledge (nor should it, IMO).

That being said, having a separate implementation in each project does have its merits, too. I remember HCANN causing release headaches and getting in the way of swift changes when you had to handle... records, I believe? A problem you wouldn't have if your abstraction was inside ORM. And, of course, there's the topic of backwards compatibility that you wouldn't need to worry about with internal code.

These contracts also expose the underlying "source" model parts for access to AnnotationUsage, etc. where appropriate

I'm a bit surprised by that, shouldn't this model be independent of annotation-based mapping?

Or maybe I misunderstood; do you intend to use AnnotationUsage even in the case of hbm.xml, where there actually isn't any annotation? Essentially everything would be modeled as annotations, just in some cases the annotations would be "virtual"?

[yrodiere]

I guess the pertinent question then is whether you'd even want to change this in Search to use this new model? If so, let's discuss this some more.

I suspect I may have to change that part of Search anyway once these changes hit the main branch of ORM... But if I don't have to, I'd probably prefer to do it later, indeed. If you manage to keep all this completely internal, we might be able to defer this.

At the moment the main incompatibility I can see is the fact this model exposes fields and methods (which I guess includes getters/setters) instead of exposing properties (fields or getters with optional setters) and methods. But then, to avoid duplicate properties, you would probably need to incorporate the concept of access type, which is JPA-specific...

EntityTypeMetadata wraps and exposes the ClassDetails

I see. So you wouldn't use annotationUsages anymore at that point, except for legacy code, right? Because annotations have been processed prior to that, to create EntityTypeMetadata et. al. So EntityTypeMetadata exposes ClassDetails exclusively for legacy code and... "reflection", I suppose? Like the type identifier, or the type of fields/methods?

[sebersole]

If you manage to keep all this completely internal, we might be able to defer this.

It would be an SPI, especially if we allow its use with AdditionalMappingContributor e.g.

At the moment the main incompatibility I can see is the fact this model exposes fields and methods (which I guess includes getters/setters) instead of exposing properties (fields or getters with optional setters) and methods. But then, to avoid duplicate properties, you would probably need to incorporate the concept of access type, which is JPA-specific...

As I tried to explain when we talked about it earlier, ORM needs more than just "attribute" - we need methods.

Also, keep in mind that HCANN does not categorize anything. You simply ask it for fields that might be an attribute (it fits a pattern) and you ask it for methods which might be an attribute (it fits a pattern). The same "attribute" will be in both. It is still up to the consumer (ORM, Search) to properly interpret all this. This proposal is not different in functionality; it just makes that functionality more logical (imo).

I think this is perfectly logical - a class has fields and methods. Keep in mind that the "source" model is intended to just be an abstraction for Class, Method, etc. Any categorization of a field or method as "special" is instead directly related to the categorization of the "class". E.g., we'd have ClassDetails for converter and listener classes as well; "attributes" have no meaning there.

Not sure if it helps or not, but that is the point of another contract -

public interface PersistentAttributeMemberResolver {
	List<MemberDetails> resolveAttributesMembers(
			ClassDetails classDetails,
			AccessType classLevelAccessType,
			LocalAnnotationProcessingContext processingContext);
}

That comes in to play as we move from "source" to "intermediate". Though, as you can see the contract is solely based on "source" contracts - it could in theory live with the "source" code if we decide to make it separate library. The outcome of this call is singular references to the backing members of the attributes for a managed-type. Meaning it has already accounted for AccessType. So here it is truly the Set of attribute members rather than a potentially "duplicated" collection of members which might be attributes.

I see. So you wouldn't use annotationUsages anymore at that point, except for legacy code, right? Because annotations have been processed prior to that, to create EntityTypeMetadata et. al. So EntityTypeMetadata exposes ClassDetails exclusively for legacy code and... "reflection", I suppose? Like the type identifier, or the type of fields/methods?

First, I'm not sure what you mean by "legacy code". But I'll try to explain how I envision all of this.

I had considered having the intermediate model represent the entirety of consuming annotations; but I do not believe that is the best approach. I expect we will still handle AnnotationUsage references after we have the intermediate model as we perform the boot model binding. If the intermediate model fully consumes annotations then it is really not any different from the boot model.

Now layering orm.xml processing on top of all this adds a wrinkle. Initially I intend to have orm.xml feed into AnnotationUsage instances, but I'll have to see how that "feels". This was the point we were working on in that older PoC when it got set aside. The main concern there was the idea of building an "augmented" Jandex index which was not really possible and is still not really. The introduction of this source model helps get around that.

I look at this entire process as akin to "interpretation" (ala HQL interpretation e.g.) in that it is best modeled using phases. The source model is similar, if you will, to the lexer. We walk that model (the lexical parse tree in the analogy) and perform some interpretation into the intermediate model. This interpretation is a macro-level interpretation though. What I mean by that is that we perform "just enough" interpretation to efficiently drive the next phase - binding to the boot model.

Roughly, this intermediate model would process ClassDetails as a number of special categorizations: entity, embeddable, converter, listener, ... From there, we know the types of details we need to capture for each -

1. managed-types expose attributes (using PersistentAttributeMemberResolver from above). Additionally, entity and mapped-superclass references are arranged into hierarchy
2. converter details expose methods - specifically the 2 methods that define the conversions
3. listener details expose method - specifically the callback methods

But the attributes here, e.g., are not sub-typed as BasicAttributeMetadata e.g.. This level of categorization is more efficiently handled as a subsequent phase (intermediate to boot). Instead, they "understand" the type of attribute:

public interface AttributeMetadata {
	String getName();
	AttributeNature getNature();
	MemberDetails getMember();

	enum AttributeNature {
		BASIC,
		EMBEDDED,
		ANY,
		TO_ONE,
		PLURAL
	}
}

Hope this helps clarify

[yrodiere]

It would be an SPI, especially if we allow its use with AdditionalMappingContributor e.g.

That settles it then. We probably don't want to leak an external library into an SPI (again).

As I tried to explain when we talked about it earlier, ORM needs more than just "attribute" - we need methods.

You could have attributes and methods, instead of fields and methods.

But ok, I see what you mean.

Search will probably need some way of inferring the ORM access type, but I think the current solution will hold even after your changes: we simply test the type of the Member exposed by org.hibernate.property.access.spi.Getter#getMember.

I expect we will still handle AnnotationUsage references after we have the intermediate model as we perform the boot model binding

Ok. So for those few references, you will effectively have an untyped model that users (e.g. AdditionalMappingContributor) need to guess the (implicit) schema of. I suppose if there's a 1-1 mapping with JPA/Hibernate annotations, that's not too bad. Especially considering that's an advanced feature.

Best not keep ClassDetails/AnnotationUsage around forever though; beyond some phase, it will become a problem for Quarkus, especially if those classes are backed by reflection (discouraged in GraalVM) or Jandex (not present on the classpath at runtime in Quarkus). But you're probably already discussing that in the other thread with Sanne.

[sebersole]

These are, by definition, only around at boot time. Not sure where you even got the alternative to be honest ;)

[sebersole]

You could have attributes and methods, instead of fields and methods.

This is what we have today with HCANN. It is kludgy at best.

[Sanne]

Not meaningt to make this even more intertwined, but there might be an opportunity for an additional "nice to have" regarding:

2. Remove direct reliance on reflection.
3. Remove direct reliance on Annotation.
4. Remove direct exposure of external libraries (HCANN) in the binding code.

It would be great if the internal contracts were able to ensure that reliance on reflection and annotation reading, and indeed using external libraries for such purposes, was forcibly restricted to happen only during the bootstrap phases during which it's necessary.

As an example, I know we had a couple of corner cases in which reflection/annotations/HCANN were being relied upon at runtime (after bootstrap has completed). I'm not sure if we still have such corner cases (it's possible we do, but I don't recall of concrete examples so I'm not certain); what I'm suggesting is that it would be beneficial if the internal Metadata binding contract could somehow help to prevent such uses at runtime.

In practice this would imply that in frameworks like Quarkus (in which the metadata building aspect is potentially run on a different JVM than the one using the generated metadata at runtime) that dependencies such as HCANN could be omitted from the runtime dependencies. Also, this could lead to reducing the amount of reflection metadata necessary for GraalVM.

N.B. this would be a nice to have, but less so important than the other goals that @sebersole already listed.

[sebersole]

Oh, I had actually forgotten another goal... The ability to offload some of this work to build time. I was really thinking about it at the Jandex level - Jandex has some level of support for serializing and later deserializing an Index. If we did this offloading at the Jandex level, it would still need Jandex at runtime.

We could also consider the ability to serialize this "source" model. Or maybe even the "intermediate" model.

[Sanne]

That's certainly interesting - and as I mentioned above, the ability of ORM to consider all the code which leads to building such model as optional (to not ship it with the application if the model was already produced).

[sebersole]

Like I said, I had not considered that specific part. Not needing dependencies at runtime was never a specific goal when I did any of this older PoC. That's strictly a Quarkus thing.

Also, there is a practical aspect of how you accomplish that in the real world. Are those now "optional" dependencies? "Optional" dependencies are widely considered the devil's spawn. You could make them "proivided" dependencies, but now the user would be responsible for adding a dependency on libraries on which Hibernate has an internal dependency; also not great.