Can't load JSON schemas with URN value in id field (draft V4)

[Non0mnisMoriar]

We are designing our own JSON schemas using URNs as unique id values (such as "id": "urn:foo:bar:object"), as is supposed to be compliant with the JSON schema definition. We're using (and stuck to) the Draft V4 version for compatibility with other components reasons.
I am trying to load the JSON schema file into a JsonSchema object using this simple function:

public static JsonSchema getJsonSchemaFromClasspath(String name) {
    JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
    InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream(name);
    return factory.getSchema(is);
}

(name argument value being here "json_schema/my_foobar_schema.json" for instance)

It does not work though, the error thrown being:

com.networknt.schema.JsonSchemaException: urn:foo:bar:object: null is an invalid segment for URI {2}
	at com.networknt.schema.JsonSchema.combineCurrentUriWithIds(JsonSchema.java:112) ~[json-schema-validator-1.0.77.jar:?]

When trying to debug what was happening, I figured out the above exception message above is thrown after catching another exception happening in the code, which is the actual cause of the error:

java.lang.IllegalArgumentException: Unsupported URI scheme encountered: urn
	at com.networknt.schema.uri.URISchemeFactory.getFactory(URISchemeFactory.java:55)

Meaning that URNs might not be supported at all?

I found a similar comment by @Arno538 on issue 100: #100 (comment), and therefore tried to revert back to version 1.0.29 since it was apparently working for them up to that version. It still wouldn't work though, the same "Unsupported URI scheme encountered: urn" exception appearing during debugging, with the only difference being the actual exception thrown being an NPE:

java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because "currentUri" is null
	at com.networknt.schema.JsonSchema.combineCurrentUriWithIds(JsonSchema.java:87) ~[json-schema-validator-1.0.29.jar:?]

Note that none of my JSON are referencing one another, nor referencing any external schema through an URN (or any URI for that matter) either. So it is an issue only on the "id" level of JSON schemas being loaded.

I couldn't find out how to make it work with a specific JsonSchemaFactory configuration either?

I feel that must be pretty standardly supported for a while now though? Also, I believe it is the same cause for issue #517 (which is using the draft V4 as well - interestingly). If I just replace the URN with a (dummy) URL, everything works smoothly, but for other reasons I can't use that on my project...

Version used: v1.0.77

[stevehu]

There is a urnFactory, and it is the one you should use instead of uriFactory. Take a look at the ValidationContext, and there are uri and urn folders in the src along with the test cases. Once you figure it out, could you please write a document on how to use it? Thanks.

[Non0mnisMoriar]

Sorry but I still can't figure it out. I checked the URNFactory and the associated test sample here: https://github.com/networknt/json-schema-validator/blob/master/src/test/java/com/networknt/schema/UrnTest.java

...But I still don't understand how that would apply to my case. It seems the custom URNFactory is used to transform/format the string value into a "valid" URI (or URL even, in that test case), whereas I don't want to transform any of my id field values, they are already correct URNs (and therfore valid URIs) and should be directly recognized as such when loading a JSON schema?

Plus it seems the URNFactory is only used in the code during ref value management (a JSON schema referencing another through a URN value), whereas I want to use URNs in id values directly (as should be compliant with URI definition), such as:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "urn:foo:bar:object",
  "title": "MyFooBarObjectSchema",
  "description": "Version: 1.0\r\n",
  "definitions": {
     (...)
  }
}

[javadan]

I think I was encountering this today.

I generated a json schema with Jackson, and now I'm trying to validate against it. It didn't like id, ($id), then it didn't like " null is an invalid segment for URI {2}"
and I thought it might be the URNs.

Did you resolve this?

[fdutton]

@javadan Can you provide an example? Draft 4 uses id instead of $id and and the second segment of a URN must be at least 3 characters.

[javadan]

Code example:

		    final File jsonSchemaFile3 = new File("src/main/resources/schema3.json");
		    final URI uri = jsonSchemaFile3.toURI();
	            JsonNode schema3Node = objectMapper.readTree(jsonSchemaFile3);

	            String schema3JsonString = mapper.writeValueAsString(schema3Node);
		    System.out.println(schema3JsonString);

		    JsonSchemaFactory schemaFactory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
            
	            com.networknt.schema.JsonSchema schema3 = schemaFactory.getSchema(schema3Node);
	            String schemaJsonFileString3 = mapper.writeValueAsString(schema3);
		    System.out.println(schemaJsonFileString3);
	        

schema3.json contents:
{ "type": "object", "id": "urn:jsonschema:za:co:example:SubmitAuth", "properties": { "value": { "value": { "type": "object", "id": "urn:jsonschema:za:co:example:Auth", "properties": { "authNumberField": { "type": "object", "id": "urn:jsonschema:za:co:example:AuthNumberType", "properties": { "dateIssuedField": { "type": "integer", "format": "utc-millisec" } } } } } } } }

Error:

Exception in thread "main" com.networknt.schema.JsonSchemaException: urn:jsonschema:za:co:example:SubmitAuth: null is an invalid segment for URI {2}
	at com.networknt.schema.JsonSchema.combineCurrentUriWithIds(JsonSchema.java:111)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:82)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:67)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:62)
	at com.networknt.schema.JsonSchemaFactory.newJsonSchema(JsonSchemaFactory.java:340)
	at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:479)

Thrown at this line: com.networknt.schema.JsonSchema schema3 = schemaFactory.getSchema(schema3Node);

[fdutton]

@javadan The actual exception is being swallowed. The real error is Unsupported URI scheme encountered: urn.

[javadan]

Thanks, I presume so. What would you recommend?

I've got this list of validators in java I could try:
https://json-schema.org/implementations.html#validator-java

So far I tried erosb/json-sKema - they also don't handle URNs "unknown protocol: urn".

Do you think there's another library that does handle it?
Or is it something fairly straightforward that could be fixed in networknt?

It does seem like URNs should work as IDs, no?

Otherwise, why is Jackson outputting URNs by default, when it generates a schema from an Object.class?
Seems pretty useless, if it's not possible to do anything with the schema.
(Maybe there's a way to change Jackson's object mappings to use some scheme other than URNs?)

[Non0mnisMoriar]

@javadan This is exactly the behavior I was getting, and no, still not resolved...

It is coded like that in networknt, it goes down to the supported schemes when reading the JSON schema, in class URISchemeFactory (especially line 53) :

        final URIFactory uriFactory = this.uriFactories.get(scheme);

And the only supported schemes that have a URIFactory associated are:

ftp=com.networknt.schema.uri.URLFactory
file=com.networknt.schema.uri.URLFactory
classpath=com.networknt.schema.uri.ClasspathURLFactory
resource=com.networknt.schema.uri.ClasspathURLFactory
http=com.networknt.schema.uri.URLFactory
jar=com.networknt.schema.uri.URLFactory
https=com.networknt.schema.uri.URLFactory

• Class URLFactory:

    public static final Set<String> SUPPORTED_SCHEMES = Collections.unmodifiableSet(new HashSet<String>(
            Arrays.asList("http", "https", "ftp", "file", "jar")));

• Class ClasspathURLFactory (referencing class ClasspathURLStreamHandler):

    public static final Set<String> SUPPORTED_SCHEMES = Collections.unmodifiableSet(new HashSet<String>(
            Arrays.asList("classpath", "resource")));

Hence why the exception is thrown, "urn" not being part of the accepted schemes of any URIFactory.
If you replace "urn:" with "resource:" in your generated JSON schemas and try, you will see that it should work.

And yes indeed, as per the JSON Schema, referencing the RFC 3986 defining URIs, since URNs are valid URIs, they should work as IDs.

[javadan]

Hey thanks for the reply.

I can't believe I'm the only person on the planet to try validate Jackson-generated JSON against a Jackson-generated JSON schema.

But it sure seems like it.

Anyway, issue seems to be that 'currentUri' is null.

	at com.networknt.schema.JsonSchema.combineCurrentUriWithIds(JsonSchema.java:111)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:82)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:67)
	at com.networknt.schema.JsonSchema.<init>(JsonSchema.java:62)
	at com.networknt.schema.JsonSchemaFactory.newJsonSchema(JsonSchemaFactory.java:340)
	at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:375)
	at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:383)

But that seems unavoidable, given JsonSchemaFactory.java:375 passing a null to the schemaUri parameter.

Any input from the authors?

[javadan]

(Oh right, I did try resource: as suggested. Didn't work unfortunately)

[javadan]

(Stackoverflow error if I use resource:)

	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
Caused by: java.lang.StackOverflowError
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at com.fasterxml.jackson.databind.JsonMappingException.prependPath(JsonMappingException.java:455)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:792)

[javadan]

Also please note that my minimal example schema is rubbish. I'm back to testing with the real schema and json.

[Arno538]

I encountered this issue as well a few years ago, as mentioned by @Non0mnisMoriar. I worked around it:

private JsonSchema getMainJsonSchemaFromClasspath(
     String mainPath, String... referencedSchemasRelativePaths) throws IOException {
   JsonSchemaFactory factory = JsonSchemaFactory.getInstance(JSON_SCHEMA_VERSION);
   InputStream mainInputStream =
       Thread.currentThread().getContextClassLoader().getResourceAsStream(mainPath);
   List<String> lines = IOUtils.readLines(mainInputStream, StandardCharsets.UTF_8);
   if (lines.size() > 0) {
     lines.set(1, lines.get(1).replaceAll(".*\"\\$id\":.*", ""));
   }
   StringWriter stringWriter = new StringWriter();
   IOUtils.writeLines(lines, System.lineSeparator(), stringWriter);
   // The validator has problems with an $id starting with 'urn:'. It will also not work for
   // starting with 'http://',
   // because then it treats it as a URL, but this schema is not locatable.

   String main = stringWriter.toString();
   // The validator requires the referenced subschema to be 'resource:{the-path-to-the-subschema}'
   // for referenced resources in /src/main/resources
   if (referencedSchemasRelativePaths != null) {
     for (String subSchemaPath : referencedSchemasRelativePaths) {
       main = main.replace(subSchemaPath, "resource:/schema/" + subSchemaPath);
     }
   }
   return factory.getSchema(new ByteArrayInputStream(main.getBytes()));
 }

[javadan]

Hi @Arno538

Thanks I tried this, unfortunately didn't work.

Here's a minimal test:

{
	"$id": "resource:jsonschema:za:co:example:SubmitAuth",
	"type": "object",
	"properties": {
		"value": {
			"type": "object",
			"id": "resource:jsonschema:za:co:example:Auth",
			"properties": {
				"authNumberField": {
					"type": "object",
					"id": "resource:jsonschema:za:co:example:AuthNumberType",
					"properties": {
						"dateIssuedField": {
							"type": "integer",
							"format": "utc-millisec"
						}
					}
				}
			}
		}
	}
}

The code just removes the line with "$id": "resource:jsonschema:za:co:example:SubmitAuth", and then dies in StackOverflow as above.

Maybe I'm missing the idea though, of the resource: scheme.

Calling your code like so, using json-schema-validator v 1.0.80:

final String mainPath = "schema.json";
final String[] referencedSchemasRelativePaths = null;
com.networknt.schema.JsonSchema mainSchema = getMainJsonSchemaFromClasspath(mainPath, referencedSchemasRelativePaths);

[Arno538]

I am using v1.0.57, I don't know if it works for v1.0.80. I am using draft-07 of the json-schema.org specification. My schemas are in src/main/resources/schema. For my case, the sub-schemas don't have an $id at the top of the file, only the main schema.
I guess after "resource:" you should put the path to the file, using "/" and not ":". So if the schemas are in src/main/resources/schema, the path must probably be: resource:/schema/your-main-schema.json
Here is a simplified version of my main schema:

{
  "$id": "urn:the-organisation:jsonschema:bla:v1.1",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "A representation of bla",
  "type": "object",
  "required": [
    "metadata",
    "bla"
  ],
  "properties": {
    "bla2" : { "$ref" : "sub-schema4.schema.json"},
    "metadata": {
      "type": "object",
      "required": [
        "modelVersion",
        "reportingDateTime"
      ],
      "properties": {
        "dataModelVersion": {
          "description": "The version of the data model",
          "type": "string",
          "title": "Version"
        },
        "reportingDateTime": {
          "description": "The time this report has been created (as yyyy-MM-dd'T'HH:mm:ss.SSS'Z', see standard ISO 8601).",
          "type": "string",
          "format": "date-time",
          "title": "bla reporting time",
          "examples": "2020-04-16T16:56:23.942Z"
        }
      }
    },
    "bla": {
      "type": "array",
      "items": {"$ref": "#/definitions/blaType"}
    }
  },
  "definitions": {
    "blaType": {
      "properties": {
        "bla4": {
          "description": "The bla4 type",
          "type": "number",
          "title": "Bla4",
        },
        "bla5": {
          "description": "Bla5",
          "title": "Bla5",
          "type": "string"
        },
        "blaID": {
          "description": "The bla identifier",
          "type": "string"
        },

    }
  }
}