Install a process for managing team changes

[chadwhitacre]

We are at a size where we need more process discipline around EPD team changes. There are a number of stakeholders for this, including EMs/PMs, IT/Security, Ops, DevInfra, OSPO, and GTM teams. Product is the main lens to use here, and the fundamental question we need to be able to answer is, "Who owns this?" Given a part of our product that I have a question about, who do I talk to? Where does the buck stop? This question is not easy to answer today.

---

---

Today, we have a number of systems with a concept of teams—Google Directory, Okta, Namely, GitHub, GCP, 1password—with unclear data flows between them. We also have a number of documentation pages, none of which is systematically kept up to date.

Brainstorm

The sketch of a solution is:

1. Google Directory is our source of truth.
2. Managers are responsible for maintaining their immediate teams.
3. We propagate to other systems and documentation resources automatically where possible.
4. We document and monitor our manual processes for propagating updates where we don't have automations.

The outputs of this epic are:

• Alignment with EPD and GTM leadership teams on a process for maintaining EPD teams and product mappings, documented in Notion.
• A big one-time cleanup.
• Probably some automations.

To Do

• Write a rough draft #94
• sort out relation to RBAC initiative
• Cleanup teams and labels and docs #60
• figure out which teams we want
• Put GitHub teams into code #127
• Switch routing from teams to product areas #126
• Update docs: Engaging Customers🔒, Structure
  • https://github.com/getsentry/static-sites/pull/1671
  • https://github.com/getsentry/static-sites/pull/1675
• Ship it! – https://vanguard.getsentry.net/p/clh8f9vk70000s60l5dcrlv8y

┆Issue is synchronized with this Jira Epic by Unito

[chadwhitacre]

Ingest has a bunch of items they want to hand off, but there's no-one clear to hand off to. Do we need a Team: Undefined for this case? Who would review it when?

[chadwhitacre]

I think the way to do this is to try to draft a Notion doc capturing all of the existing systems and processes, use that to drive clarification where missing and then prioritize cleanups and automations.

[chadwhitacre]

I've asked🔒 for elevated Google perms to support my work on this project.

[chadwhitacre]

After starting to noodle on this under #90 I think I'm going to have to limit focus to product teams (vs. all teams across Sentry), and also maybe break this up based on what questions we're trying to be able to answer. The main one for me personally is for routing "Given feedback from a user, what team should it be routed to?"

[chadwhitacre]

We've got a parallel initiative to clean up RBAC for GCP🔒. Architecture sketch is: security-as-code repo ⇒ Google Directory ⇒ GCP. Question is where "⇒ GitHub" could fit into that, from repo or from Google Directory. Either way, consider Terraform? 🤔

[chadwhitacre]

Met w/ crew ...

1. In scope for me is managing propagation from GG to GitHub teams.
2. Aiming to use the eng-pipes codebase but run a separate instance for security reasons (high privilege to do user/group changes).
3. Expectation is that we can listen for GG events and take action in GitHub.

[chadwhitacre]

EPD announced a re-org yesterday. Trying to wrap my head around:

1. Changes in EPD for FY24🔒 (email to epd-leadership)
2. EPD FY'24 H1🔒 (lucidchart org chart)
3. How this relates to Namely🔒.

[chadwhitacre]

I talked to Vlad about the recent reorg, his process is to design in Lucidchart and then delegate to his reports to make updates in Namely and wherever else.

[chadwhitacre]

I also sent a big email about product vision🔒, I've been getting caught up thinking about that because team structure is related to product structure. Mostly just needed to get that off my chest, for this project I need to focus on implementing systems to codify the teams defined by others, vs. caring about what the team structure actually is.

[chadwhitacre]

I've published the Notion page🔒 (had to shuffle some things around to make a good place for it).

I've started reaching out to Vlad's reports to understand the next level of detail in how information flows.

[chadwhitacre]

Bottom line seems to be:

• EMs/DEs make decisions and drive changes.
• POPS and IT implement changes.

[chadwhitacre]

Next step: meet with POps! This Jira intake🔒 is the contact point.

[chadwhitacre]

Big deal. Need to organize comms around this, people care about it.

[chadwhitacre]

Look through audit log to understand GH team usage.

[chadwhitacre]

Put product vision rant in a doc🔒 as well.

[chadwhitacre]

From an internal Slack thread:

We need a clear definition of the structure of our product ~= our code. Then we need a mapping of teams to the parts of our product/code. It should be an intentional decision for there to be gaps in coverage, with some backstop process for occasionally reviewing otherwise-unowned backlog.

[chadwhitacre]

No company is ever resourced to where there is an owner for every line of code or feature. We gotta decide what’s most important all the time. If a feature ever comes up that someone wants to fight for sentry is an open door

[BYK]

No company is ever resourced to where there is an owner for every line of code or feature.

I think this is a bit of a hyperbole but prioritization is very important, that's for sure. You'd still need a contact team/person for potentially every single line. Think about when your product has a security vulnerability.

[chadwhitacre]

Internal debate raging. 🐭

[BYK]

🍿

[chadwhitacre]

I had a meeting with HR and IT.

IT states that "100% should be the HR system that is the system of record / source of truth." Our HR system is Namely. Apparently it's not clean enough to fulfill it's destiny, however. 🤔

[chadwhitacre]

Alright, what's going on here?

There's two outcomes I need:

• solve routing
• solve GH team creation

They can be linked but don't necessarily have to be.

Link dump of tabs I've got open on this, historical + current attempts:

• https://lucid.app/lucidchart/c980b81b-618f-4e1c-bbae-d816c0332883/edit?invitationId=inv_0fda72ba-b90a-423f-a2c1-0326a1fa5f83&page=0_0#
• https://www.notion.so/sentry/Agenda-for-3-25-2020-36d95b0ce9384f36ab7213c7b8b83f9e#fa502343dd4843299042e0fdb67ed0f4
• https://www.notion.so/sentry/Agenda-for-02-10-21-888146c6272049588ba4845d9d2b4a58#22ca4ea29b984dca908398058a8544de
• https://docs.google.com/document/d/167gAKYgTV7dtDnIWKgNKyUVOwASfGLxzuhMUwGpBsbE/edit
• https://www.notion.so/sentry/2022-03-22-78773a76650f4292807794b3b06709fe#fcaefda0d8114b908ced7b69f5034032
• https://www.notion.so/sentry/Service-Feature-Directory-Proposal-55a7f746d6eb4dc6bf9e9eec9e587ebe#ca32e0040a26463c96a19df19f62174f
• https://www.notion.so/sentry/Ownership-Boundaries-at-Sentry-5719ca75d876456981021b7dcf2f5f21#566651e9483846b98a686b5d2d247d85
• https://docs.google.com/spreadsheets/d/11afZ8Ve7isA53GxN__hKWBILCWa_LvzVbd9RoDlDKko/edit#gid=0
• https://www.notion.so/sentry/Product-and-Service-Registry-f90c702816794dcaa7f81ef8e79e40ea
• https://github.com/getsentry/sentry/blob/master/.github/CODEOWNERS
• https://www.ycombinator.com/blog/how-to-maintain-engineering-velocity-as-you-scale
• https://www.notion.so/sentry/Triage-Process-4e2127cbd9304d6b922dc71ccb85cebc
• https://docs.google.com/spreadsheets/d/1bYlLGRgIMlqGGWWBQbyAHild3_l_3wyrMTU3VwBUpgg/edit#gid=0

[chadwhitacre]

Current thinking is to have a structure of product owners that covers the whole product surface from an outside-in perspective, slightly separate from the org chart for EPD teams working on strategic product investments (inside-out).

[BYK]

Why can this not be in a repo like the publish repo, where each team change is a PR updating the configuration.

Once approved and merged, a GitHub action syncs it to other services.

[chadwhitacre]

I'd love to integrate it somehow with the code for the left nav itself. 😁

[BYK]

If it's a static yaml file, you can do a build-time include? I think this is already done for Swagger API definitions or something?

[BYK]

Yeah it is being done for OpenAPI -> Docs sync: https://github.com/getsentry/sentry-api-schema/blob/main/.github/workflows/cascade-to-sentry-docs.yml

[chadwhitacre]

Hey who wrote that? 😏

[chadwhitacre]

Here's my current thinking:

1. Generate a comprehensive set of product areas, as seen from the user's point of view. Start with the left nav, but also review our open tickets to pick up any additional areas. The goal is to be able to associate every ticket with exactly one product area.
2. Add a required dropdown field to the issue template with all of the product areas. This eliminates routing entirely from our process (even better than Route with AI #123), possibly frees up Support to do L1 triage.
3. Create Product Area: Foo labels for each product area. Eventually I'd like to explore using Projects for this instead of or in addition to labels, but one step at a time.
4. Update eng-pipes to work with Product Area: * instead of Team: *. Map labels directly to Slack channels so we don't have to deal with GH teams. Possibly start a UI for eng-pipes to easily see what areas are mapped to which Slack channels. If any areas have no slack channel associated, consider hard-closing them as wont-fix.
5. Add an OSPO chore to review and update product areas monthly.

[chadwhitacre]

Okay, I've got buy-in on switching from Team labels to Product Area labels. 👍

I'm tempted to use Projects instead of Labels for this, but a) I'm not sure that's actually the best solution (labels are more visible on the issue detail), and b) that's probably a lot more work in eng-pipes.

[chadwhitacre]

How should I think about GitHub Teams here? I'm pretty sure I want to use teams to encode the owners for each product area. How about this teams structure:

• Product Owners > individual product areas
• Code Owners > only subteams of this in CODEOWNERS
• Eng - review perms, why do we need other teams in GitHub? Can we get away with a single Eng team that has access to everything?

[chadwhitacre]

I made a proof of concept for manipulating a Project V2 programmatically from eng-pipes. This is a standalone script that updates metadata fields in a project based on properties of an issue, number of comments in this case.

demo.mov

Next step is to wire this into eng-pipes to fire on issue changes. (GitHub doesn't provide an event when reactions change, but we can approximate for now by updating on adjacent events such as commenting, and maybe someday we can poll.) That will get the ball rolling, and even then there will be value in adding issues to 76 (this board number is going to be A Thing™️, I can already feel it).

From there we can build out automations to use the status field on that board to drive our responsiveness goals. I've reached out to the Crons crew to be our guinea pigs. Once we build it out it will be simple to retire the label-based system, since the two will operate essentially independently.

The workflow will be approximately:

1. Add all new issues from non-engineers to this board with no product area, assigned to Support.
2. Add all issues where a non-engineer comments to this board, probably also assigned to Support for routing.
3. Routing means assigning a product area on the board.
4. Set due dates automatically on the board.
5. Probably also still comment on the issue as well.

So basically port over the existing to here. Advantages of a project vs. issues/labels are:

1. Projects are decoupled from repos. We can have a single process that spans all repos we care about.
2. Projects are decoupled from each other. Product teams can have their own boards with their own metadata and workflows, without any of that showing up here, or vice versa.

I am also leaning towards trying to get away without using GitHub Teams here. Let's just use the Slack notifications instead of Team mentions.

[chadwhitacre]

https://vanguard.getsentry.net/p/clh8f9vk70000s60l5dcrlv8y