🚀GitHub Copilot Metrics API GA release now available

[Akash1134]

🚀 Now Available: GitHub Copilot Metrics API in General Availability! 🚀

We’re thrilled to introduce the GA release of the GitHub Copilot Metrics API, now open to all GitHub Copilot Business and GitHub Copilot Enterprise users. This release is packed with features to give you greater visibility into Copilot's impact across your development workflow. Ready to dive into the data?

---

📊 What’s the GitHub Copilot Metrics API?

Imagine having detailed insights into how Copilot is being used across your enterprise. The Metrics API brings this to life by providing you with data on Copilot's usage—across teams, organizations, and individual features. Perfect for tracking trends, understanding usage patterns, and integrating with your own dashboards, the Metrics API helps you see where Copilot is adding value and how you can optimize its use.

🎉 What's New in the GA Release?

Here’s a peek at what’s included:

• New metrics for Pull Request summaries and Copilot Chat on GitHub.com.
• Clarity on code completions and Copilot Chat usage in IDEs.
• User engagement summaries updated daily.
• Custom model slicing support, coming soon!
• Data is aggregated by enterprise, organization, and team.
• Up to 28 days of historical data available for trend analysis.

With data loaded daily (UTC) and summarized by day, you can view patterns in real-time and make more data-informed decisions.

🤔 Will My Current Reporting Be Affected?

To make the transition smooth, we’ve kept the Beta route online through the end of the year. This means any current reporting tied to the Beta API will continue without interruption, while you explore the newly revised schema and expanded metrics in the GA version.

📚 Resources to Get Started

• Docs: Ready to get started? Explore the API documentation for the full breakdown of metrics and schema.

---

💬 Your Turn!

How are you planning to use the Copilot Metrics API? What specific metrics would give you the best insight into Copilot's impact on your projects? Whether it’s tracking usage, surfacing adoption trends, or something else entirely, we’d love to hear your ideas. Drop your thoughts below and let’s shape the future of Copilot analytics together! 👇

[charliew33]

@Akash1134 This looks great, would love to see more detailed metrics about Copilot usage and engagement.

Question: it looks like this is disabled for my org, do I need an enterprise owner to enable? The docs are a little light but it sounds like if I get an enterprise owner to enable the Copilot Metrics API for the enterprise I can then get them for my org?

To access this endpoint, the Copilot Metrics API access policy must be enabled or set to "no policy" for the enterprise within GitHub settings. Only owners and billing managers for the enterprise that contains the enterprise team can view Copilot metrics for the enterprise team.

[samus-aran]

Hi @charliew33,

Yes, you would need to enable at the Enterprise level. 👍

[DevOps-zhuang]

Thanks for the GA, it looks more detailed, and indeeed more complex :-)
like for the previous usage route, it displays the aggregate code summary data for each day, then break down it by editor/language.

day": "2023-10-15",
    **"total_suggestions_count": 5000,
    "total_acceptances_count": 3000,
    "total_lines_suggested": 7000,
    "total_lines_accepted": 3500,**
    "total_active_users": 15,
    "total_chat_acceptances": 45,
    "total_chat_turns": 350,
    "total_active_chat_users": 8,

for the GA metric route, I was doubting whether we still need an aggregate high-leve total_code summary for each day, maybe per day per language? currently, for the copilot_ide_code_completions part, it only has one direct KV which is "total_engaged_users": 20. in order to get the total_code_* metircs as before, we have to check the breakdown data, which is by editor/models/language, the merge them togeterh. based on my observation, the total_code_* data per day per language is a strong requirement. so it is better to add it. thanks for your consideration.

like currently, for the lanuages part under "copilot_ide_code_completions", it only display the total_engaged_users part, is it possible to add total_code_* part for each language? thanks.

"copilot_ide_code_completions": {
"total_engaged_users": 20,
"languages": [
{
"name": "python",
"total_engaged_users": 10
},
{
"name": "ruby",
"total_engaged_users": 10
}
],

[DevOps-zhuang]

and the other consideration is about the 'engaged user' and 'active users', to be honest, I am not quite understand the difference, and I am still oberving and check the difference.

[rhlbhatnagar]

Hey,
While using this API I get:

"Your organization has disabled Copilot Metrics API access. Enable it in GitHub settings to access this endpoint."

In github settings I'm unable to enable access to the metrics API as admin. I see a link to a doc on how for github pre release license terms
https://docs.github.com/en/site-policy/github-terms/github-pre-release-license-terms

But don't actually see how to get my enterprise access to these pre release features.

[veverkap]

You need to go into your enterprise settings first (https://github.com/enterprises/{YOUR_ENTERPRISE_SLUG}/settings/copilot?tab=policies) and chose either "No Policy" or "Enabled"

If you chose "Enabled", then the policy is enabled for all organizations and they cannot change the setting at the organizaiton level.

If you chose "No Policy", then the policy is configurable at the organization level (https://github.com/organizations/{YOUR_ORGANIZATION}/settings/copilot/policies

[rajeshharne]

Accessing GitHub Copilot Usage API with a Business Account – Is it supported?

Question: @Akash1134 / Team

Hello everyone,

I’m looking to track GitHub Copilot usage metrics through the API for our organization, which currently has a GitHub Business license. Specifically, I’d like to use the following endpoint:

/enterprises/{enterprise}/copilot/usage

Before diving in, I wanted to confirm: does the GitHub Business license provide access to the Copilot usage API, or is this feature exclusively available for GitHub Enterprise accounts? If it’s enterprise-only, are there any alternative methods for tracking Copilot usage in a Business account?

Thanks for any insights or guidance!

[veverkap]

Yes, a Copilot Business subscription with users is all that is needed.

[karpikpl]

how to access new metrics?
the docs say

To make the transition smooth, we’ve kept the Beta route online through the end of the year.
what's the difference between beta route and new route?

I get old data for
x-github-api-version-selected: 2022-11-28
GET https://api.github.com/orgs//copilot/usage

also the link to the documentation doesn't work

[DevOps-zhuang]

copilot/usage is the old usage API. and copilot/metrics is the new one

[karpikpl]

lol, thanks. I don't know how I missed that :)

[DevOps-zhuang]

we found some data issues, the total_suggested is also 0. though there are accepated, can you double check the data quality?
the data is as below:
"copilot_ide_code_completions": {
"editors": [
{
"name": "vscode",
"models": [
{
"name": "default",
"languages": [
{
"name": "typescript",
"total_engaged_users": 1,
"total_code_acceptances": 39,
"total_code_suggestions": 0,
"total_code_lines_accepted": 22,
"total_code_lines_suggested": 0
}
],
"is_custom_model": false,
"total_engaged_users": 1
}
],
"total_engaged_users": 1
}
],
"languages": [
{
"name": "typescript",
"total_engaged_users": 1
}
],
"total_engaged_users": 1
}
},

[DevOps-zhuang]

and "copilot_ide_code_completions": {
"editors": [
{
"name": "JetBrains",
"models": [
{
"name": "default",
"languages": [
{
"name": "java",
"total_engaged_users": 1,
"total_code_acceptances": 8,
"total_code_suggestions": 0,
"total_code_lines_accepted": 19,
"total_code_lines_suggested": 0
}
],
"is_custom_model": false,
"total_engaged_users": 1
}
],
"total_engaged_users": 1
}
],

[karpikpl]

I think we've seen similar issues in sample data and API data before this was GA :(

[DevOps-zhuang]

I think we've seen similar issues in sample data and API data before this was GA :(

Yes, we see different issues. hope it can be solved ASAP. since it is GAed, good quality is hoped.

[nazifrp]

I think we've seen similar issues in sample data and API data before this was GA :(

[nazifrp]

---

[juusto1981]

to 21.11.2024 klo 10.25 nazifrp ***@***.***) kirjoitti:

…

I think we've seen similar issues in sample data and API data before this was GA :( — Reply to this email directly, view it on GitHub <#141071 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEMZ6UD34DSVQNJEQYPJPZ32BWKGBAVCNFSM6AAAAABPWY2JYGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZTGM2TMMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>