InnerSource @ DAZN | "Building A DX Team: Lessons Learned"

[spier]

This post from DAZN Engineering contains various stories that could be cross-referenced with existing patterns:
https://medium.com/dazn-tech/building-a-dx-team-lessons-learned-4a66446088bc

There are other posts from DAZN as well that may be related e.g.
https://medium.com/dazn-tech/developer-experience-dx-at-dazn-e6de9a0208d2
https://medium.com/@nodefortytwo/what-is-platform-engineering-a6e8bff1d9c6
https://medium.com/dazn-tech/integrating-backstage-at-dazn-b8ef5268b347

Who knows, maybe even some new patterns could be extracted?

What I spotted in the article:

• DAZN InnerSource Marketplace => InnerSource Portal, maybe also Gig Marketplace (also an example of using Backstage for these)
• RFC process => Transparent Cross-Team Decision Making using RFCs
• likely more ...

[loujaybee]

CC: @cirpo / @nodefortytwo

[hunter]

The manifest idea from the first post might be an other interesting pattern. My team has talked about a similar idea for a machine readable document to describe a project (including Innersource related data/metadata)

[spier]

Nice one @hunter!

What metadata would you keep in such a metadata file?
Would be great to hear more about your use case.

From the post:

.dazn-manifest.yml file, which is a metadata file that describes the contents of a repo. By having a common metadata file, we’ve been able to enable things like search and discovery, through our Backstage instance. Through this file we can configure tooling, like our service level objects (SLO), so that they “just work” with only a few lines of configuration. These features wouldn’t be possible if we had many different approaches or opinions.

From a patterns perspective, we have mentioned this idea as part of the InnerSource Portal pattern. However it might make sense to expand on this.

There is an open discussion about such a metadata format in the patterns working group as well.

An interesting reference for that topic is the syntax definition of innersource.yaml that SAP is using for their InnerSource portal.

[nodefortytwo]

Hey, nice to see some interest in our manifest file.

I have created a public gist of the current version of the RFC that defines the dazn-manifest.yml. obviously, there is plenty of stuff in here that is specific to DAZN and our internal tooling, and you can see by the SemVer we have been through quite a number of iterations.

RFC
JSON Schema

Happy to answer questions on how we use it and @petetanton, who currently looks after this internally, would also be happy to chat.

I also mentioned this point in my blog post about platform engineering:

“Metadata — The single biggest win platform engineering has delivered is something called ‘dazn manifest’ which is a json file in the root of every repo that describes the services built by the repo and the team that owns it along with…” — @nodefortytwo

https://link.medium.com/GHls1erESnb

[spier]

Very cool @nodefortytwo. Thanks for sharing this!

I need to look through the InnerSource-specific fields of your manifest more thoroughly.

In the meantime, can you share more about what the main purpose of that metadata for your org is?
e.g. is it mainly discoverability of projects, or do you have other secondary goals in mind as well?

Also I am curious what the stats are on InnerSource projects by distribution-method?
e.g. is there a distribution method that lends itself more for InnerSource projects than others?

[petetanton]

Hi @spier - at DAZN, we use the Manifest for several things (here is a quick summary):

• Identifying which team owns the repo (and thus any thing else defined in the Manifest)
• Registering services and systems (where a service is a "single deployable thing" and a system can contain one or more services)
• Registering dependencies between services (sync, async, retries etc)
• Defining support configuration that we use to automatically configure PagerDuty (where we also push relationship mappings)
• Defining any libraries or projects defined in the repo that are inner source with maintainers etc
• SLO definitions for services - we use this to automatically configure our tooling to track SLOs and alert on error budgets
• Annotations and tagging - we use this to attribute costs in our cost tooling and configure Backstage
• Security - here we define any security risks and risk owners

Essentially, we use this metadata to drive a lot of our platform tooling including service dictionaries, observability platforms and paging tools. The other benefit is we can abstract any integrations with third parties reducing the vendor lock in (as migrating requires one code change in whatever process converts the data in the Manifest to configuration in a 3rd party)

With the distribution-method specifically, this is an enum with the following options:

• Drone Plugin
• NPM
• Go Module
• Browser Extension
• Document
• Cloud Infrastructure
• CLI
• GitHub
• Web App

This is more about how another developer might come along and consume the inner source project and we provide search tooling

[spier]

That is great extra info @petetanton.

Do you have stats on how many InnerSource "things" you have per distribution-method? I am curious if there is a story in there about what types are more likely or less likely to be shared as InnerSource?

For the point about "Identifying which team owns the repo":

This is a problem that we facing as well in my org, and from conversations I believe that many other orgs do too.

I our case it is almost always a team owning a thing (very rarely an individual however that is typically an indication of a potential future maintenance/ownership issue).

The team names we use are the same as what we use to specify teams in GitHub.

However as GitHub doesn't show on the repo-view which team has access to the repo, we also end up specifying the owning team in the README. So that's some redundancy there.

In your approach, does the manifest become the only place that specifies ownership or do you still have to manually maintain it in other places too?

Then there is also the issue of choosing team names in a way that are stable long enough over time. But that topic might be Pandora's box and maybe not strictly related to the InnerSource topic here, so I am leaving the lid closed ;)

[petetanton]

Hi @spier - sorry for my slow response!
For stats on inner source projects, they are as follows:
1 Browser Extension
3 CLI
45 Cloud Infrastructure
7 Document
10 Drone Plugin (we are moving away from Drone so I suspect this will be replaced with GHA soon)
19 GitHub
7 Go Module
108 NPM
13 Web App

With team ownership, we consider the Manifest to be the source of truth. It then drives other services like Backstage (so you can filter repos and services in Backstage by owning team)
Github Teams for us are a bit of a mess but we only really care about the "maintainer" in Github as we work on the principle that any employee should be able to open a PR against any repo.
To sort of answer your question, we currently manually manage Github teams but will probably look at driving Github ownership from the Manifest in combination with Active Directory at some point in the future.

Don't get me started on team names...

[spier]

@petetanton thanks for the stats above.
Now I noticed that some of the stats also show up in the screenshots in DAZNs blog posts :)

One other question:

I guess the reason for defining your own dazn-manifest.yml was to be independent of any specific tool and vendor?
Or maybe you already had that in place before you started using Backstage?

In our case we are just getting started with all of this, so rather than building something from scratch, I am wondering if would be a good approach to start the service definition with the Backstage catalog-info.yaml file instead. We could then extend that to implement further DX tooling based on that. Note: I haven't checked yet if Backstage would actually be happy with that i.e. if it would silently ignore additional custom properties in the YAML that are not part of their Descriptor Format.

[spier]

Another reference to something like a manifest file is in here. They call it the Codebase Governor (or CBG) for short:
https://innersource.flutter.com/docs/cbg/

@robtuley was the CBG something you built from scratch, or is it the extension of something that already existed?

[robtuley]

It was actually originally introduced for us by GitHub professional services to help us do inner source and then we've evolved it forward from there. GitHub actually released their internal tool open source recently to that is similar(ish) so maybe it was partially copied from that -- Entitlements.

I think it's worth differentiating between a reporting informational manifest (like the backstage catalogue file) ... vs settings configuration thing that allows users to apply actions to their stuff. Our CBG tool is the latter as it our way of simplifying the setup of a GitHub repo into a state that is safe for contribution, but we also use it for reporting. Change the "manifest" and some settings change in your repo, it's a GitOps type flow for settings with the nice side effect you can use it for reporting too...

[spier]

The whole manifest solution sounds more and more like a pattern in its own right to me.

While it might be related to other patterns like the InnerSource Portal, the solution has various properties that go way beyond this, like the "default repo settings for safe InnerSource contributions" and "reporting" use cases that you are describing.

A challenge will be to slice this up properly, to determine whether it is one or multiple patterns.
We could start writing a single pattern, and if it grows too big we break it up further.