Ability to dynamically fetch API spec definition from a URL or a built file

[taras]

Our organization uses GraphQL federation. We have a bunch of services that are federated into a single gateway.
We would like to show schema from these services in the catalog. We use Nexus to express schema as TypeScript and compile the TypeScript to GraphQL SDL.

Feature Suggestion

We need a way to get the schema from the result of a build. In our case, the schema for each service is available on the service running in QA, inside of the GitHub Action while preparing tests and in Jenkins when building docker images. We need a way to use schema from one of these methods.

Possible Implementation

1. Allow specifying a URL where the schema can be fetched from

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: my-federated-service
spec:
  type: graphql-federated-service
  definition:
    url: https://my-federated-service.my-cloud.com/graphql

One challenge that I see with this approach is that we're not going to have a clear relationship between schema in QA and Git commit that schema was built from.

2. We could not change anything and update the catalog whenever we build the service in GitHub Actions or Jenkins. This seems less preferable because it'll require configuring each service per repository. This mechanism could become fragile because it might stop updating and no one will know.

3. Have some way to build the code while fetching schema updates. I'm not sure how this would work.

[Fox32]

Yeah allowing to load the definition from a url is something we would like to implement too. But I would prefer to reuse the location processors for this: It would be useful to load the definition from locations like GitHub repositories, files, or other locations too.

I hope this is something we can completely implement in the processors, so that we load the definition into the entity.

We had some discussion here already: #2001 (comment)

[freben]

Yeah my initial rough idea was that you could define that API as something along the lines of

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: my-federated-service
  annotations:
    backstage.io/graphql-schema-source: https://my-federated-service.my-cloud.com/graphql
spec:
  type: graphql-federated-service

Note the annotation. And then a processor would inject the actual definition based on what was introspected from that endpoint.

For schemas that are next to the catalog-info.yaml file, one could have an annotation backstage.io/file-source: ./openapi.json or something like that. We could make it know how to resolve relative paths to the current location, and then leverage the already existing processors to read the data out of the resulting absolute URL.

[Fox32]

Edit: Nevermind, I think I read that point the wrong way

[taras]

@freben your suggestion above sounds like it could work well.

Is this something that we could implement now or are there blockers?

[freben]

That can be implemented today. What isn't there is

• A more helpful code structure for injecting custom processors (you can, just a couple unwieldy lines of code)
• Standard annotation names that are well known and useable for multiple purposes (you'll have to invent some annotation, possibly the one i suggest above is decent)
• A connection from a single processor out to its "context" - as in, a processor can't easily ask the "cloud of processors" to just read out a URL for it, because it doesn't know about the other ones (either implement the reading in some more hard coded way, or contribute like a new context parameter to the processor method or something)

[Fox32]

A connection from a single processor out to its "context" - as in, a processor can't easily ask the "cloud of processors" to just read out a URL for it, because it doesn't know about the other ones (either implement the reading in some more hard coded way, or contribute like a new context parameter to the processor method or something)

I think that's the biggest issue right now.

[Fox32]

That was not the "cancel" button 😆

[freben]

I think that's the biggest issue right now.

Does it require auth or so? I was hoping that it was directly reachable (from the deployed catalog backend's point of view) so you could just call it directly from the processor :)

[taras]

In our particular situation, introspection doesn't require authentication.

[freben]

Cool. Otherwise, as I mentioned I wouldn't be averse to adding a new context parameter that essentially has a read method in here down the line so the readers get that new ability.

[freben]

(but maybe that doesn't help specifically the graphql introspection thing, that's a more complex request chain isn't it)

[taras]

A more helpful code structure for injecting custom processors (you can, just a couple unwieldy lines of code)

Where is this?

Standard annotation names that are well known and useable for multiple purposes (you'll have to invent some annotation, possibly the one i suggest above is decent)

backstage.io/graphql-schema-source sounds good

[Fox32]

Does it require auth or so?

I guess in this graphql case it's planned to load the spec directly from the running service. But in our case we would like to load them from GitHub, so having the same flexibility as for normal entity loading (I mean locations) would be nice here.

But if one don't need it, a custom processor should be possible already and easy to implement.

Where is this?

You can pass them here:

https://github.com/spotify/backstage/blob/master/plugins/catalog-backend/src/ingestion/LocationReaders.ts#L97

Which would be here in your backend: https://github.com/spotify/backstage/blob/master/packages/backend/src/plugins/catalog.ts#L34

[freben]

A more helpful code structure for injecting custom processors (you can, just a couple unwieldy lines of code)

Where is this?

they are given to LocationProcessors https://github.com/spotify/backstage/blob/master/plugins/catalog-backend/src/ingestion/LocationReaders.ts#L97 (order is important, you want yours to be after reading but before validation)

[freben]

too slow!

[Fox32]

I drafted a PR for this here #2597

[taras]

👋 you guys are amazing

[freben]

@taras however, see #2628 too which came after :)