[RFC] From Identity to Ownership, v2

[freben]

Status: Open for comments

Need

This proposal is a replacement for #3870, which will be closed. There is a lot of context and discussion in that issue, which serves as background to this. Rather than polluting and extending all the discussion in there, let's start with a clean slate - but we suggest reading it.

The short summary is:

• There is an authentication system that is responsible for establishing the identity of the current user in Backstage.
• There is a catalog that may or may not contain organizational data: users and groups, and their relationships.
• There are entities in the catalog, that have an owner field that is filed in either by users or by auxiliary systems such as codeowners.

Given the above, devise a solution that

• In a sensible way lets the logged in user see the entities that they own, for a definition of the word "own" that makes sense to the user
• Supports hand written spec.owner entity fields primarily
• Does not require that organizational data exists or is complete in the catalog

As a lower priority goal, the solution might

• Support ownership of multiple distinct "domains" of entities, for example if there are entities that are sourced from multiple different companies (perhaps into separate namespaces), or from open-source VCS as well as internal VCS, where the user has different identities in each.
• Play well with codeowners

Current State

The current identity API presents the following pieces of information, that the auth backend supplies after signing in:

• A user ID, which is an opaque string that somehow represents the user's identity
• An ID token, issued by Backstage, that can be used to verify the user
• Some profile information such as the full name, email, and picture.

Different auth providers in the backend may provide this info in different ways. They are all hard coded and not pluggable - except if you replace or add an entire provider yourself.

When the catalog frontend plugin wants to get the current user entity, it does so by looking for a User kind entity in the default namespace with a metadata.name that matches what came out of the identity API.

When the catalog wants to deduce whether you are an owner of an entity, it does so by requiring the above User entity and comparing it with the target entity via relations. If you do not have a User entity that matches, you are out of luck.

Proposal

Let the identity of the current user, as returned from the auth backend plugin and exposed to the frontend via the identityApiRef, be extended as follows:

• It contains a new field claims, which is an array of entity references. Each is a claim about an identity or a membership that is relevant to the user, for example User:default/freben and Group:default/my-team-name. There is no requirement that these correspond to actual existing catalog entities.
• The existing ID token is extended to contain these claims as well.

Let the definition of the ownership of an entity E, for a user U, be as follows:

• Get all the ownedBy relations of E, and call them O
• Get all the claims of the user U and call them C
• If any C matches any O, return true
• Get all Group entities that U is a member of, using the regular memberOf/hasMember relation mechanism, and call them G
• If any G matches any O, return true
• Otherwise, return false

Of course, this is a formalization and the actual process can be heavily cached with a single lookup at login of all your claims and extending them with the corresponding memberships etc.

The auth providers of the auth backend will be extended with a new plugin interface, effectively

// Takes basically the passport response, and somehow produces claims
type AuthClaimBuilder = (authResponse: any) => Promise<List<EntityName>>;

The default implementation of these, when not overridden by the Backstage integrator, will be similar to the user ID logic of today, but on a User entity form. So if today's identityApiRef would have a getUserId() returning 'freben', then there would be a single corresponding claim User:default/freben.

Users may replace this logic in any way that they see fit. An example flow could be,

• Authentication happens with a Google sign in, resulting in a john.knowles@example.com email as the established identity in the passport response.
• The custom AuthClaimBuilder makes a company specific lookup in the local LDAP system by email, to find that the LDAP identity of the user is john, member of the group infra-ninjas. Via custom attributes it is found that his public GitHub user is johnHaxxesBezt, and in GHE he is known to be john because the company by convention has GHE usernames equal to LDAP usernames.
• The same function makes a request to GitHub's and GHE's APIs to look up the domain specific group memberships of the above users.
• The final list of claims is user:default/john, group:default/infra-ninjas, user:github/johnHaxxesBezt, and a a number of groups that were returned by the GitHub/GHE APIs.

An important point here is that the auth backend does not need to have any connection to the catalog. The only way that this is linked to the catalog, is that the claims are on the form of complete entity references instead of just standalone IDs. Whether you import your org data or not, the Backstage integrator can choose the usage of ID spaces and namespaces freely here, and make those correctly aligned with other plugins' configuration.

Further Work (Optional)

There is one missing piece to sew this up properly with e.g. codeowners in multiple version control systems. Note how this is not explicitly in scope for this RFC, as mentioned at the top.

Notice how the johnHaxxesBezt user was placed in a namespace they chose to call github. Entity definition files placed on public GitHub can't reasonably be expected to have their spec.owner set to internal, secret LDAP group names; you'd put the GitHub team names in there. Same thing goes if you are using codeowners; those would use the public GitHub user or team identities as well.

This means that as we ingest these yaml files out of GitHub into the catalog, the end result entity as stored in the catalog should have ownedBy relations using the namespace github - because that's what the Backstage integrator chose to use here. For example, User:github/johnHaxxesBezt or Group:github/snoo-maintainers. Remember that this github string is not set in stone nor is it a convention; it could be anything that the integrator chose, on a per-version-control-system basis.

There are a few ways of achieving this.

One is to tell users to write their yaml files on public GitHub with an explicit namespace in the owner field. They will then have to remember to do so, because if they don't, the generated relation will instead point to the default namespace where it likely will not match an actual group - or, perhaps worse, match the wrong group that happens to be similarly named. It also doesn't make a lot of sense as seen from "the outside"; the string github was chosen entirely by the internal integrator and therefore this is a very leaky abstraction. Nonetheless, it is a solution that will work on a technical level, without any additional code changes. If it is a rare occurrence, it can be the initial approach to pick.

spec:
  owner: github/snoo-maintainers

The alternative approach is to make processors namespace aware, on a per-integration basis. For example, we could conceivably add a defaultNamespace parameter to the integrations config. The codeowners processor could take this into account, and fill in the owner field with that namespace. We could even make a general pre processor step that injects these namespaces, only when missing, in all such fields for all entities, based on their origin location.

integrations:
  github:
    - host: github.com
      defaultNamespace: github
      token:
        $env: GITHUB_TOKEN

Alternatives

See #3870 for discussions.

Specifically, this RFC does NOT address the purely token-based ideas mentioned therein. We welcome discussions under this new RFC, to see how it may align with a possible token-based approach.

Risks

See #3870.

[tragiclifestories]

If any G appears directly in any C, return true

Should that be "appears directly in O"? Otherwise it seems tautological ...

Maybe I've not understood how claims relate to groups in this context.

[freben]

Indeed, my typo, thanks for being thorough! Corrected.

[erikxiv]

Impressive work, reconciling a lot of different comments and options, kudos!

Minor thought - would it be valuable to augment the AuthClaimBuilder to include claim type to be able to support other types of claims than user identity and group membership, for example authorization claims or similar? No need from my side, just a thought.

Have you considered the use-case when a user is looking at other users, groups, components to deduce ownership? In theory, I like the idea of being able to see that the github-namespace-team is the same team as the default-namespace-team and similar when it comes to users and ownership. I think the model above supports that by more fully entering that data into the catalog but still allowing other adopters to choose another strategy, but it might be worth considering.

[Rugvip]

Awesome! Think we're getting ready to 😁

@freben regarding namespace as org layers, we arrived at wanting to use namespaces as the main mechanism for access control. The namespace that a user is a member of would then have an impact of what parts of the catalog that user has access to and we wouldn't want to also use that for providing support for multiple org layers.

Wanna be a bit cautious about using "clams" as the term too, it's very generic and conflicts pretty heavily with JWTs, ending up with token.claims.claims. Wonder if we can think about it as "aliases" or "alternate names", bit like SANs in x.509.

I find it the most useful to think about this in terms of what goes in the JWT, as that's where it ends up being most important that we can have a very solid and precise definition of what these additional claims mean.

One possibility could be something like this:

{
  "sub": "user:default/john",
  // Entities that the token bearer claims ownership of, and via
  "ent": ["user:default/john", "group:deafult/infra-ninjas"],
  // External identities of the user, although perhaps not "ext" as that's usually used for extensions
  "ext": {
    "github": "johnHaxxesBezt"
  }
}

Still not quite sure how the external identities would be used in practice though, and whether we should have them at all. It's kinda fine to use to assign ownership of a single component, but when we zoom out and look at groups and tooling e.g. like tech- and cost-insights it gets trickier. Tbh there's nothing preventing someone from just synching in GitHub teams as groups mixing in with any other org data though right? Perhaps prefixing groups form GitHub and allowing squads to add their team as a member of their squad group. Although tbh I'm simply leaning towards codeowners processing not being that useful for orgs that don't use GitHub as their main source of org data.

[tragiclifestories]

@freben regarding namespace as org layers, we arrived at wanting to use namespaces as the main mechanism for access control.

I didn't spot this when reading through but I think it's a very good point - especially as this is an important factor of how namespaces are used in Kubernetes land, and the deliberate similarity of the two manifest schemas will inevitably create expectations around what a namespace means in Backstage based on K8s.

[adamdmharvey]

This RFC is great; really gets to the heart of accountability and driving alignment. Excited to see this progress!

Given the recent changes to the system model with system and domain kinds, I would almost think (or assume?) some sort of hierarchy built into this:

Get all the ownedBy relations of E, and call them O

In a sense, it might be:

Get all of the ownedBy relations of E, and call them O1
Get all partOf entities of E , and call them P
Get all of the ownedBy relations of P, and call them O2
Concatenate O1 and O2 into O

What I'm not sure is if this was the intention of the hierarchy, but if my team (myDepartment) owns a system, I would thus infer I own all components in it, even if a certain component might have more specific owners (like a smaller subteam) or an individual user. If the model allows teams to potentially dilute accountability, that's a nuance for sure but something we may not be able to prevent?

This could be held off for a second iteration of the RFC for future updates, or just flat out parked so as not to make it too confusing. 😊

[freben]

Minor thought - would it be valuable to augment the AuthClaimBuilder to include claim type to be able to support other types of claims than user identity and group membership, for example authorization claims or similar? No need from my side, just a thought.

Makes sense! I originally experimented a bit with a more precise field name to at least scope it better, and with other more structured shapes, but had trouble coming up with something that felt right so I kept it lean. Good alternative suggestions for the output of this function are already cropping up, which is great.

Have you considered the use-case when a user is looking at other users, groups, components to deduce ownership?

Let's see.

If there IS org (user/group entity) data in the catalog, then you can visit their corresponding entity pages. They would have ownerOf relations pointing at other entities, and those other entities would have ownedBy relations in the other direction. So both the users, groups, and components should show proper info. However you might not see directly what john owns unless we make a specific card for that which does the extra query through his memberships to groups. Without that you'd just see what groups he's a member of, and clicking through to those you'll see what they own.

If we do not have org data, things are a bit different. Based on the spec.owner field, a pair of ownedBy/ownerOf relations would be generated by the owned entity. But with the way that the code currently works, we couldn't really click on the link to john, we'd get an Entity not found error. We could conceivably change the entity page a bit such that it works better for that case. It could inform you that the entity was not found, BUT here are all of the "dangling" relations associated with it, so you could still navigate the relation-mesh of things. You could also never be at the john skeleton-view and see what things his groups own - because the needed group membership relation data just wouldn't exist in the catalog.

In theory, I like the idea of being able to see that the github-namespace-team is the same team as the default-namespace-team and similar when it comes to users and ownership. I think the model above supports that by more fully entering that data into the catalog but still allowing other adopters to choose another strategy, but it might be worth considering.

Hm, I haven't considered a mechanism to put equality between two groups. That could be a relation too, I suppose, perhaps in a loose sense as relatedTo. If the data for generating that exists.

[freben]

@freben regarding namespace as org layers, we arrived at wanting to use namespaces as the main mechanism for access control. The namespace that a user is a member of would then have an impact of what parts of the catalog that user has access to and we wouldn't want to also use that for providing support for multiple org layers.

I see what you mean, but consider that the actual logged-in identity is still meant to be in default, and the normal situation is that there is one central VCS that also ingests into default, so in that regard nothing changes. Any non-default data would fall under the "Future Work" section.

My initial (not very qualified - asking for feedback) hunch is that:

• The number of such extra namespaces is very small, equal to or fewer than the number of extra VCS integrations
• They would often be exactly one, which would be the open source hosted VCS of choice. Then, the access management "cost" would essentially be to add a rule that "All users have access to X". If we scope and limit it like that, maybe the value gained is still significant.

Wanna be a bit cautious about using "clams" as the term too, it's very generic and conflicts pretty heavily with JWTs, ending up with token.claims.claims. Wonder if we can think about it as "aliases" or "alternate names", bit like SANs in x.509.

Yeah I wasn't really happy about the choice of naming either. Some great suggestions below.

I find it the most useful to think about this in terms of what goes in the JWT, as that's where it ends up being most important that we can have a very solid and precise definition of what these additional claims mean.

One possibility could be something like this:

Having a distinct primary sub apart from the ent is definitely useful. But for these external identities, I feel we could just keep them as entity name triplets anyway, and throw groups in there as well. It would let us treat external ownership uniformly with internal ones. Plus, without external groups to go with the external users, we won't be able to deduce ownership anyway (hopefully most ownership is via groups rather than users there, too).

Still not quite sure how the external identities would be used in practice though, and whether we should have them at all. It's kinda fine to use to assign ownership of a single component, but when we zoom out and look at groups and tooling e.g. like tech- and cost-insights it gets trickier.

I would expect those to actually be scoping themselves intentionally to a given namespace. It might even be a parameter in their config - what namespaces do you want to compute cost for? With the default of ['default'] of course. I think that namespaces present no actual issue there; their greater problem is rather for example, how do they know what actual runtime systems accrue cost for this particular entity X? And that's a problem they have no matter what.

Tbh there's nothing preventing someone from just synching in GitHub teams as groups mixing in with any other org data though right? Perhaps prefixing groups form GitHub and allowing squads to add their team as a member of their squad group. Although tbh I'm simply leaning towards codeowners processing not being that useful for orgs that don't use GitHub as their main source of org data.

Mm I though of prefixes. But those really mess with alignment with the actual keyspaces coming out of those third party systems. Can get ugly I think, but should be considered for sure.

Won't we ourselves really want to track our open source things in internal Backstage? Won't we want to see component:default/backstage and component:github/backstage, and their completely different cost breakdowns, and ongoing builds, and monorepo structures, etc etc? :) To me, that sounds like a pretty valuable prospect!

[freben]

Given the recent changes to the system model with system and domain kinds, I would almost think (or assume?) some sort of hierarchy built into this:

While that does sound logical, does it usually hold? I'm not sure yet if owning a system is really connected to owning the pieces. I concede that it most certainly will be very common! But maybe some will like to put an overarching org entity such as a product area as owners of a system, while subteams own the individual pieces?

But besides that, in fact every entity kind in question here already has a mandatory spec.owner field at least in the current specification. So there's no transitivity to account for - you'd have to explicitly set the owner to a certain team anyway. So they would be picked up by the mechanisms proposed herein.

On the topic of monorepos, I think it's pretty common to have the monorepo itself be owned by some kind of infra, or delivery focused, team, while subfeatures are owned by specialised smaller feature teams. But this may be a place where transitivity is sometimes really convenient: you would probably want the root-owners be the default owners of everything within but then "slice out" chosen folders as owned by feature teams where relevant. And that, I think, is a dimension that codeowners is perfectly suited for, which leaves the problem space out of the catalog so to speak. Well, apart from using the codeowners proessor of course :)

Does that make sense?

[adamdmharvey]

@freben Makes a lot of sense. I think the key point you have there is the mandatory spec.owner, which thus infers you don't need to inherit. And it could be the ingestion which if CODEOWNER wasn't defined say for an entity like an API could be assumed to be owned by the same team who owns the system the API is attached to, but it would thus COPY the value into the spec.owner of the resulting new entity, which solves the problem. (and could always be reassigned in future)

So would agree, parking hierarchical topic seems to make sense. Thanks for noodling it out!