$ref that passes lint but fails in validate as error: Could not resolve schema reference

[robbat2]

I'm trying to look at alternate options for validation against JSON Schema for https://github.com/backstage/backstage - and I think I found a case where the schema is valid, but jsonschema fails to handle the $ref behavior correctly.

AJV does handle this case correctly. P.S. Globbing for the -r argument or accepting a list of files as another file would help in making the commandline much shorter.

Shorter part of the error:

error: Could not resolve schema reference
  #reference
    at schema location "/definitions/relation/properties/target/$ref"

This is here in the schema

If I remove target from the schema, as it is deprecated, I get a similar error for #statusItem:

error: Could not resolve schema reference
  #statusItem
    at schema location "/definitions/status/properties/items/items/$ref"

Against a checkout of Backstage:

$ export BIN=/tmp/jsonschema.bug.lVrZDy/jsonschema-4.1.5-linux-x86_64/bin/jsonschema
$ export MAINSCHEMA=/code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json
rjohnson@rjohnson-1:/code/gh/backstage/backstage$ ${BIN} lint \
$(find packages -name '*.schema.json'  -printf " -r %h/%f\n" |sort | uniq ) \
${MAINSCHEMA}   -v
Linting: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/Entity.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/EntityEnvelope.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/EntityMeta.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Domain.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Group.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Location.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Resource.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/System.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/User.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/shared/common.schema.json
PASS: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json


$ ${BIN} validate $(find packages -name '*.schema.json'  -printf " -r %h/%f\n" |sort | uniq ) ${MAINSCHEMA} /code/gh/coreweave/backstage/data/e
xample_catalogs/observability/mimir/catalog.yaml    -v
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/Entity.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/EntityEnvelope.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/EntityMeta.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Domain.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Group.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Location.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/Resource.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/System.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/kinds/User.v1alpha1.schema.json
Importing schema into the resolution context: /code/gh/backstage/backstage/packages/catalog-model/src/schema/shared/common.schema.json
error: Could not resolve schema reference
  #reference
    at schema location "/definitions/relation/properties/target/$ref"

[jviotti]

Hey @robbat2 , thanks for reporting. Let me take a proper look

[jviotti]

@robbat2 OK, I think you are hitting this: https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#section-8.2.1-3

If present, the value for this keyword MUST be a string, and MUST represent a valid URI-reference [RFC3986]. This URI-reference SHOULD be normalized, and MUST resolve to an absolute-URI [RFC3986] (without a fragment), or to a URI with an empty fragment.

So in JSON Schema, having a top-level $id that is not an absolute URI (like $id: common in your case) is considered undefined behaviour. Some implementations like AJV might handle it correctly, but many may not! I actually have an existing issue for this (sourcemeta/jsontoolkit#960) and plan to support it at some point.

In the mean-time though, I still suggest, as a best practice, setting the top-level URI to an absolute one. It can be a https:// URL, a URN, or even a tag URI (https://en.wikipedia.org/wiki/Tag_URI_scheme). Your schema works fine for me once I do that!

That said, let's keep this issue open, and I'll close it once we fully support relative base URIs :)

P.S. Globbing for the -r argument or accepting a list of files as another file would help in making the commandline much shorter.

That is great feedback! Though keep in mind -r also takes a directory, and you can filter down files using --extension. Maybe this works much more nicely already?

jsonschema validate schema.json instance.json --resolve catalog-model/src/schema --extension .schema.json

?

[jviotti]

I'm also just realising we don't document --extension very well for the validate command in particular. I'll send a PR

[jviotti]

OK, I have a PR that fixes the relative base URI undefined behaviour issue and works fine with your schema! #188. I'll release a new version of the JSON Schema CLI with it in a bit.

[jviotti]

Done! Take a look at v4.1.6. It should work fine now: https://github.com/sourcemeta/jsonschema/releases/tag/v4.1.6. Thanks a lot for reporting this! 🙏🏻

[robbat2]

@jviotti thank you - I confirm it works.

Also, as a long-time open source author & maintainer, this was also one of the best interactions I've seen in my life ❤️

[jviotti]

Always happy to help and thank a lot for the kind words. Truly made my day! 😍

If you feel inclined to rephrase this message as a testimonial in the README (https://github.com/sourcemeta/jsonschema?tab=readme-ov-file#what-our-users-are-saying) I would be forever grateful!

And of course, happy to help on anything JSON Schema related any time!