🔌 Plugin: Backstage metrics

[OrkoHunter]

Summary

A plugin which shows metrics and insights about a Backstage instance. This kind of information is of great value for the Platform teams/Product managers to figure out what is working well and what can be improved.

Some metrics that can be shown by this plugin are

Org and trends

• Number of plugins installed
• Plugins usage (Active/Inactive/Popular/etc.)
• Number of users (Active/Inactive/All)
• Catalog entities and trends
• ...and more

Technical

• Time to Create (new components)
• Time to load Backstage
• ...and more

Some of these metrics require analytics to be setup already so that plugin usage data exists somewhere (Google Analytics, etc.). The UI of the plugin can be pretty flexible and show any numbers its database can store.

One other feature of this plugin could be to send weekly reports to key stakeholders of Backstage in an organization.

Question to backstage adopters

• What other metrics do you use or you think are useful in your decision making?
• How are you tracking Backstage usage metrics? Will this plugin be of any help?

[mufaddal7]

Interesting, would love to work on this. This plugin would be very helpful for many organizations. A good way to measure adoption. We did something similar to gather metrics on our backstage portal using New Relic. We were focussing on gathering user information (events, views, etc)

• Number of plugins installed
• Plugins usage (Active/Inactive/Popular/etc.)
• Number of users (Active/Inactive/All)
• Catalog entities and trends
• ...and more

Technical

• Time to Create (new components)
• Time to load Backstage
• ...and more

These are some good helper points to start off with.

[benjdlambert]

@mufaddal7 assigned! Maybe it would be a great start for plugin-debug or something where it could also log some great things for creating of bug tickets or something.

[Expalex]

These are great metrics!
Our team is working to add some of the following to ours at the moment

• Health status (G, Y, R) and trends
• Availability
• Latency rate
• Failure rate (Frequency of deployment failures)
• Frequency (Average deployments released)
• Lead time (Time between commit and deployment)
• Average down time
• Mean time to detect, know, fix, resolve
• Security risk plugins
• Operational risk plugins
• Summaries, view by dates, sort, and drill down capabilities
• On/off switch
• Number of plugins built
• Subscribed (Added to favorites)

[satrox28]

Any update on this issue @mufaddal7. @Expalex will you be sharing the code?.

[Expalex]

We have some created and are creating that'd be useful. Our team need to go through the list to see which ones would be best to contribute first and in order. I'll update here once I find out. I do not have an ETA at this time.

[regicsolutions]

@Expalex interested to see what your team has built, any plans on open-sourcing?

[gman0922]

Is there a current update on this issue @mufaddal7? :)

[mufaddal7]

@guillermomanzo on a tight schedule , didn't get the time to compose a plugin.

If anyone opens up a PR , will be happy to contribute the bits and pieces, I developed .

[Rymania]

Looking at outcomes we'd be really interested in looking at "reuse" and "experience" (as well as the standard user stats - let's put them in "reach").

Reuse

• Catalogue Additions
• Catalogue item usage (overall and for each item --> 'most popular' at item level)
• GitHub Enterprise click-through (this is where a component is not a template, but an Internal Action or similar)

These all seem pretty similar to your thinking.

We're just starting off on Backstage, so these stats will be really useful to demonstrate adoption and value. Right now the engineers will be busy on the core implementation but I'd love to see this.
Developer Experience
A trickier one to put in but could consider the below. I think that there's a case for an NPS plug-in separate to this.

• Repeat users (how many users are active once a day/once a week/once a month etc)
• Popular templates (see above) and pages

Reach

• Unique users
• Total users
• Active users

[benjdlambert]

@fcorti what do you think about the above metrics?

[fcorti]

This is a relevant topic for most adopters and I frequently receive questions about it.
The topic of "telemetry" is important and it is definitely on our radar.

I can see two challenges at this stage:

1. Having clarity on the "right" metrics to be developed (and this thread is an example where we talk about "metrics for operators" and "metrics for product owners").
2. Ensure the bandwidth to start the development (this is always a challenge).

I personally think that it is good to have a place where to discuss and coordinate the effort, but not sure if we should start developing an MVP of the plugin (with a few basic metrics) and iterate through contributions or discuss the proper metrics first.

On Spotify's side, we are discussing item 1 (with @michael-bellato and @iamEAP) but it would be great to coordinate the effort.

If someone will be happy to invest time and effort in the development of an MVP, let me know.

We'll keep the community updated on the evolution on our side.

[Rymania]

Thanks @fcorti - I personally am interested in the Product Owner type metrics, because they will show the value proposition for scaling and also help us understand what our users want.

In terms of contributing - we are still working on the core installation right now, but I will keep this in mind once we've done that because I see being able to measure outcomes as as part of our MVP - and we are looking at other options given that the Google Analytics Plug-in cannot be used now due to some of the GDPR rulings of the GA product in the EU.

[binitan]

Some initial thoughts - I am looking for trends on contribution by developers community. Based on this we can design rewards & recognition for the top contributors in 3 different categories - catalog, template, docs and encourage innersourcing within our organization. Glad to discuss other ideas that anyone has regarding the rewards system as an incentive for contribution.

All the above mentioned metrics are also relevant for us.

[marcelovcpereira]

This is something we also would like to have and will soon start considering the development of our own plugin.
Do we have any news on such a metric plugin?

[kurtaking]

We want a way to view all repositories created from a specific Backstage template within the Backstage UI. We have considered building an internal plugin to accomplish this.

[awanlin]

@kurtaking there's some Prometheus metrics for the scaffolder that might have that, I just not sure what labels ended up being used: https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/prometheus-metrics.md

[awanlin]

Yup, they include what template and what user: https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/CHANGELOG.md#minor-changes-7

[kurtaking]

@awanlin ah, thank you. I came across this issue before the PR that introduced these metrics. I'll take a look.

[kurtaking]

@awanlin do you know if anyone has considered building a UI to surface these metrics?

[awanlin]

@kurtaking, well generally speaking Prometheus metrics would get graphed or visualized in a tool like Grafana so I think that's what most people are doing.

[iamEAP]

Glad to see some activity in this thread! Wanted to throw down a few thoughts that've been swirling around in my head, and hopefully try and turn this issue into something a little more actionable.

Synthesizing some of the ideas here, I want to propose that there are up to three (perhaps more) distinct categories of utility that people are looking to unlock with Backstage in this thread. In my mind, those are:

• Usage Metrics: This would be metrics related to how an R&D community is interacting with an instance of Backstage, which could inform champions/stakeholders about the efficacy of Backstage as a solution to organizational challenges. Here, I would include many of the metrics proposed in @OrkoHunter's first category (plugins installed, usage, number of users, etc.) as well as other ideas from the thread (catalog usage, popular templates, etc).
• Application Performance Metrics: This would be operational metrics related to the health of a Backstage instance that a group of engineers/ops folks would use to troubleshoot and tune the application itself. Here, I would include items from @OrkoHunter's second category (time to load backstage, etc) as well as some of @Expalex's ideas (availability, latency, etc), plus @awanlin's pointers toward using prometheus.
• Productivity Metrics: This would be high-level metrics that may or may not be related to a Backstage instance at all that engineering leadership could use to inform cultural and organizational efforts related to developer experience more broadly. Here, I would include many of @Expalex's remaining suggestions (frequency of deploys, mean time to resolution, etc).

I'm inclined to say that the utility of showing any of this data within Backstage is quite low. The audience for this data is pretty limited to the people operating and iterating on Backstage at an organization, as well as their leadership, and not necessarily users of Backstage. With that in mind, I'm inclined to recommend that any kind of UI for these should live elsewhere (be it Google Analytics, Grafana, New Relic, etc).

What's left to do

On: Usage Metrics: The Plugin Analytics API was specifically designed to unlock this use-case. Some of the Catalog and Template-related suggestions in this thread are now (only as of relatively recently) possible to derive from Analytics API data (much of from backstage/backstage#14118). What's missing seems like...

• Alternative implementations (instead of just Google Analytics) of the Analytics API
• Guides on how to put together reports on these kinds of metrics using data in those tools

On: Application Performance Metrics: I'm honestly out of the loop on where things are in Backstage with respect to APM. The guide @awanlin points to seems like a great start for those using prometheus. For orgs who use other technologies, I'm not sure what's possible. Would someone be willing to summarize the state of Backstage APM and recommend possible next steps like above?

On: Productivity Metrics: IMO, this is a totally unexplored area. Are there SaaS or OSS products out there that offer this? Could they be integrated with Backstage?

[julioz]

@iamEAP on productivity metrics, there are many tools out there specialized in measuring DORA/SPACE/etc, a famous one being dx. For example, in our usage at SoundCloud, we've built some integration within our GoCD CI Server and also help build a CodeScene plugin. These are "within Backstage" up to the point where a user wants to drill down more specifically, and our plugins then lead the users to the external tool UI (in other words, we didn't reinvent the wheel).

I do agree with you something like the "Analytics Plugin" would be very useful for our customers here too. In terms of Prometheus metrics, though - we have used the "temporary" instrumentation (that if I remember correctly is being replaced with OTEL), and added a custom flavor of observability through log ingestion (via mtail), to generate alerts to the operational team. I agree these don't fit in the Backstage UI, and are more a "behind the scenes" thing for maintainers.

[bforbis]

I'm inclined to say that the utility of showing any of this data within Backstage is quite low. The audience for this data is pretty limited to the people operating and iterating on Backstage at an organization, as well as their leadership, and not necessarily users of Backstage.

A use case for surfacing this to users directly in backstage is to reach some sort of feature parity with the engagement statistics functionality that is built into Confluence (using this addon). This allows content authors to understand if the documentation they have written is useful to their stakeholders

[binitan]

@bforbis Could you point me to the PR for this feature or is it proprietary?

[bforbis]

@binitan that's just a screenshot of confluence. I'm suggesting it as a possible use case for why users may be interested in page analytics. Backstage could be updated to do something similar.

[canda]

I'm inclined to say that the utility of showing any of this data within Backstage is quite low. The audience for this data is pretty limited to the people operating and iterating on Backstage at an organization, as well as their leadership, and not necessarily users of Backstage. With that in mind, I'm inclined to recommend that any kind of UI for these should live elsewhere (be it Google Analytics, Grafana, New Relic, etc).

@iamEAP I agree that showing this data on Backstage itself doesn't add much value for regular backstage users
But I think there is value in implementing the UI within Backstage because of the simplicity of implementation once such a plugin is developed.
Otherwise, you end up with suggestions like the one @awanlin mentioned.
https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/prometheus-metrics.md
Which is not good or bad in itself but does not apply to everyone's tech stack.
To take advantage of the open source traction, we should pay the engineering cost of implementing something that works quickly and cheaply for everyone up front. Then everyone will quickly join to help iterate. And there is no more ubiquitous technology within Backstage users than... well, Backstage 😂 .

[iamEAP]

Yeah, the in-context engagement data is an interesting use-case. Some other use-cases that aren't specifically front-end could include decorating search index data with engagement numbers and using that as a basis for biasing results.

we should pay the engineering cost of implementing something that works quickly and cheaply for everyone up front.

Maybe! I'm not sure it's going to end up being all that cheap in the end, though.

On the data collection side, it was much more straightforward to put together an API interface that could quickly/easily be implemented across analytics providers because the business logic for each provider is basically just "accept JSON in one form, mutate it to match the platform's expected format."

...I can imagine it being a lot less straightforward to provide a general purpose interface for querying usage data across analytics platforms.

[tduniec]

Hey,

In terms of backstage adoption and time savings generated by using templates. This may be useful in templates context

please find the plugin:
https://github.com/tduniec/backstage-timesaver-plugin

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.