Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deferencing $ref in json schema is not working after upgrade to v3 [part 2] #5362

Closed
mmmmillar opened this issue Oct 17, 2024 · 9 comments
Closed

Comments

@mmmmillar
Copy link

Deferencing $ref in json schema is not working after upgrade to v3. The error I'm getting is "Unknown/unsupported data model type or version"

This is related to #5296 - a PR was merged for it but it's still not working. As you can see I'm running 3.0.1

INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/apicurio-registry-app-3.0.1-SNAPSHOT-runner.jar

To reproduce

/groups/GROUP/artifacts/ARTIFACT/versions/VERSION/content?references=DEREFERENCE

Parent schema:

{
  "$id": "https://x.com/night_time/front_door_open_detected.json",
  "$schema": "http://json-schema.org/draft-07/schema",
  "description": "xxx.",
  "type": "object",
  "properties": {
    "hub_id": {
      "type": "string"
    },
    "hub_timezone": {
      "$ref": "https://x.com/_common/timezone.json"
    },
    "detected_at_utc": {
      "type": "string",
      "format": "date-time"
    }
  },
  "required": [
    "hub_id",
    "hub_timezone",
    "detected_at_utc"
  ]
}

Referenced schema:

{
  "$id": "https://x.com/_common/timezone.json",
  "$schema": "https://json-schema.org/draft-07/schema",
  "name": "Timezone",
  "description": "An enum field that must be a valid timezone identifier string.",
  "enum": [
    "Africa/Abidjan",
    "Africa/Accra"
  ]
}
2024-10+00-17 14:01:00 INFO [io.quarkus.vertx.http.runtime.filters.accesslog.JBossLoggingAccessLogReceiver] (executor-thread-2428) "apicurio-registry.access method="GET" path="/apis/registry/v3/groups/notifications_courier/artifacts/a1c5c953-f2bd-002c-fc0a-d360c7852121/versions/1/content?references=DEREFERENCE" response_code="500" response_time="-" remote_ip="172.31.42.252" remote_user="-" user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36""
2024-10+00-17 14:01:00 ERROR [io.apicurio.registry.services.http.CoreRegistryExceptionMapperService] (executor-thread-2428) [500 ERROR DETECTED] : Unknown/unsupported data model type or version.: java.lang.RuntimeException: Unknown/unsupported data model type or version.
at io.apicurio.datamodels.ModelTypeDetector.discoverModelType(ModelTypeDetector.java:67)
at io.apicurio.datamodels.Library.readDocument(Library.java:89)
at io.apicurio.registry.content.dereference.ApicurioDataModelsContentDereferencer.dereference(ApicurioDataModelsContentDereferencer.java:23)
at io.apicurio.registry.rest.v3.AbstractResourceImpl.handleContentReferences(AbstractResourceImpl.java:59)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass.handleContentReferences$$superforward(Unknown Source)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass$$function$$1.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.apicurio.common.apps.logging.LoggingInterceptor.logMethodEntry(LoggingInterceptor.java:53)
at io.apicurio.common.apps.logging.LoggingInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass.handleContentReferences(Unknown Source)
at io.apicurio.registry.rest.v3.GroupsResourceImpl.getArtifactVersionContent(GroupsResourceImpl.java:521)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass.getArtifactVersionContent$$superforward(Unknown Source)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass$$function$$20.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
at io.apicurio.common.apps.logging.LoggingInterceptor.logMethodEntry(LoggingInterceptor.java:53)
at io.apicurio.common.apps.logging.LoggingInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
at io.apicurio.registry.auth.AuthorizedInterceptor.authorizeMethod(AuthorizedInterceptor.java:64)
at io.apicurio.registry.auth.AuthorizedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor.restMethodInvoke(QuarkusRestPathTemplateInterceptor.java:39)
at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_Subclass.getArtifactVersionContent(Unknown Source)
at io.apicurio.registry.rest.v3.GroupsResourceImpl_ClientProxy.getArtifactVersionContent(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:229)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:222)
at io.quarkus.resteasy.runtime.ResteasyFilter$ResteasyResponseWrapper.service(ResteasyFilter.java:70)
at io.quarkus.resteasy.runtime.ResteasyFilter$ResteasyResponseWrapper.sendError(ResteasyFilter.java:76)
at io.undertow.servlet.handlers.DefaultServlet.doGet(DefaultServlet.java:172)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at io.quarkus.resteasy.runtime.ResteasyFilter.doFilter(ResteasyFilter.java:31)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.HSTSFilter.doFilter(HSTSFilter.java:47)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.ResourceCacheControlFilter.doFilter(ResourceCacheControlFilter.java:76)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.rest.RegistryApplicationServletFilter.doFilter(RegistryApplicationServletFilter.java:51)
at io.apicurio.registry.rest.RegistryApplicationServletFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.RedirectFilter.doFilter(RedirectFilter.java:76)
at io.apicurio.registry.ui.servlets.RedirectFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:63)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:67)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:133)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:65)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:247)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$9$1.call(UndertowDeploymentRecorder.java:630)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:126)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:284)
at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$5$2.run(UndertowDeploymentRecorder.java:445)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:599)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
@apicurio-bot
Copy link

apicurio-bot bot commented Oct 17, 2024

Thank you for reporting an issue!

Pinging @andreaTP to respond or triage.

@carlesarnal
Copy link
Member

To be able to help I would need to know how you registered the schema. For some reason it's detecting the wrong dereferencer, so I need to know that the internal state of your data (more specifically the artifactType) to be able to reproduce and eventually fix this.

@mmmmillar
Copy link
Author

mmmmillar commented Oct 17, 2024

Create the child

POST: http://apicurio/apis/registry/v3/groups/x/artifacts

Body: {
    "artifactId": "100",
    "artifactType": "JSON",
    "name": "child",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/language_code/json\",\"$schema\":\"https://json-schema.org/draft-07/schema\",\"enum\":[\"en-GB\",\"fr-FR\"]}",
            "contentType": "application/json"
        }
    }
}

Create the parent which references the child

POST: http://apicurio/apis/registry/v3/groups/x/artifacts

Body: {
    "artifactId": "200",
    "artifactType": "JSON",
    "name": "parent",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/test.json\",\"$schema\":\"https://json-schema.org/draft-07/schema\",\"type\":\"object\",\"properties\":{\"language_code\":{\"$ref\":\"https://x.com/language_code.json\"}}}",
            "contentType": "application/json",
            "references": [
                {
                    "groupId": "x",
                    "artifactId": "100",
                    "name": "child",
                    "version": "1"
                }
            ]
        }
    }
}
GET http://apicurio.dev.intelligentlilli.com/apis/registry/v3/groups/x/artifacts/200/versions/1/content

returns the parent schema without any dereferencing as expected

GET http://apicurio/apis/registry/v3/groups/x/artifacts/200/versions/1/content?references=DEREFERENCE

returns

{
    "detail": "RuntimeException: Unknown/unsupported data model type or version.",
    "title": "Unknown/unsupported data model type or version.",
    "status": 500,
    "name": "RuntimeException"
}

I wasn't sure whether to use the name or the $id as the $ref value but tried both name and $id with the same result. Have checked artifactType for each schema by getting the version - can confirm they are both JSON.

@carlesarnal
Copy link
Member

I just tested your example and it works fine except for the fact that I get the following error, due to the draft being incorrectly set:


{
    "detail": "IllegalArgumentException: Unsupported draft identifier: https://json-schema.org/draft-07/schema",
    "title": "Unsupported draft identifier: https://json-schema.org/draft-07/schema",
    "status": 500,
    "name": "IllegalArgumentException"
}

The draft identifier is missing a # at the end.

This is what I've done:

docker run -it -p 8080:8080 quay.io/apicurio/apicurio-registry:3.0.1

POST: http://apicurio/apis/registry/v3/groups/x/artifacts

Body: {
    "artifactId": "100",
    "artifactType": "JSON",
    "name": "child",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/language_code/json\",\"$schema\":\"https://json-schema.org/draft-07/schema\",\"enum\":[\"en-GB\",\"fr-FR\"]}",
            "contentType": "application/json"
        }
    }
}
POST: http://apicurio/apis/registry/v3/groups/x/artifacts

Body: {
    "artifactId": "200",
    "artifactType": "JSON",
    "name": "parent",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/test.json\",\"$schema\":\"https://json-schema.org/draft-07/schema\",\"type\":\"object\",\"properties\":{\"language_code\":{\"$ref\":\"https://x.com/language_code.json\"}}}",
            "contentType": "application/json",
            "references": [
                {
                    "groupId": "x",
                    "artifactId": "100",
                    "name": "child",
                    "version": "1"
                }
            ]
        }
    }
}

And then get the content:

GET http://apicurio/apis/registry/v3/groups/x/artifacts/200/versions/1/content?references=DEREFERENCE

There is one thing I noticed, and is the fact that, in the $ref in the main content, you're using a full URL. That won't work. That value must be the same as the name of the reference.

@mmmmillar
Copy link
Author

Yeah I mentioned above that I tried both the name and the $id (full url) as the $ref but neither worked.

It's strange that you're getting a unsupported draft identifier exception and I am not. Is there a setting that could cause that to happen?

@carlesarnal
Copy link
Member

Yeah I mentioned above that I tried both the name and the $id (full url) as the $ref but neither worked.

It's strange that you're getting a unsupported draft identifier exception and I am not. Is there a setting that could cause that to happen?

No, no setting. One thing I notice is that you're using what seems to be a local build of the application, since the version is 3.0.1-SNAPSHOT. Can you try it using the docker command I shared that used the 3.0.1 version released?

@mmmmillar
Copy link
Author

Thanks for that, I somehow had a SNAPSHOT version. The unknown/unsupported data model error is no more.

However, the setup we had for refs is no longer working - we'd previously named the artifacts using full urls so that any reference matched the id of the reference, eg "$ref": "https://x.com/_common/timezone.json" would resolve to the artifact named "https://x.com/_common/timezone.json"

This worked in v2, but works no more in v3. Now I get

{
    "detail": "UnsupportedOperationException: Can't resolve '_base_message.json', only internal refs are supported.",
    "title": "Can't resolve 'https://x.com/_common/timezone.json', only internal refs are supported.",
    "status": 500,
    "name": "UnsupportedOperationException"
}

@carlesarnal
Copy link
Member

I don't know how you created the references, but doing this worked just fine:

POST: http://apicurio/apis/registry/v3/groups/x/artifacts

    "artifactId": "100",
    "artifactType": "JSON",
    "name": "child",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/language_code/json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"enum\":[\"en-GB\",\"fr-FR\"]}",
            "contentType": "application/json"
        }
    }
}

POST: http://apicurio/apis/registry/v3/groups/x/artifacts

    "artifactId": "200",
    "artifactType": "JSON",
    "name": "parent",
    "firstVersion": {
        "content": {
            "content": "{\"$id\":\"https://x.com/test.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"language_code\":{\"$ref\":\"https://x.com/language_code/json\"}}}",
            "contentType": "application/json",
            "references": [
                {
                    "groupId": "x",
                    "artifactId": "100",
                    "name": "https://x.com/language_code/json",
                    "version": "1"
                }
            ]
        }
    }
}```


In those two requests I'm using the id of the nested schema as the name of the reference, and that worked just fine.

@mmmmillar
Copy link
Author

Thanks for your patience! I removed all my schemas and re-added and it is now working fine

@github-project-automation github-project-automation bot moved this from Backlog to Done in Registry 3.0 Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

2 participants