How much CO2/GHG does Copilot emit?

[merlindru]

I'm wondering how much energy the use of GitHub Copilot costs and how much CO2/GHG it emits.

I realize GitHub is carbon neutral, but what about Copilot? If GitHub wasn't carbon neutral, how much would we be looking at here?

Getting some stats about the training would be great, too, as I've read articles about GPT-3 training emitting 3 times the "emissions of a round trip flight of a passenger jet between San Francisco and New York"

Are there projections for how this might change in the coming years?

Much appreciated!

[david-engelmann]

How much energy is used compiling broken code?

[JonasDoe]

Alright, a couple of things:

• Regarding the training, we were talking about different things. Both apply, ofc.: the pretraining of the model, as well as the "personalization" training done by your LLM provider
• The power consumption on use differs probably quite a lot, depending which LLM is used. What I've heart of GPT (and I figure Microsoft is utilizing that by now), it's rather intense on calculations b/c it needs to run over and over again for each generated word/token. If you have more knowledge here than I have, I'ld happy to hear about it, though (no sarcasm intended)!
• You have a good point here that keystrokes in IDEs are probably enforcing rather much calculations anyway. I'm not sure agree that the use of Copilot is a balanced trade-off here. I fully agree that if you start with googling, reading, writing for new, non-straight forward functions, generating it with Copilot might be advantagous. But I'm having my doubts about a simple autocompletion feature for words, lines or simple clauses. These won't take 10 minutes to write and neither require much googling, but a couple of seconds, and your IDE just have to lookup for keywords and (often) just the autocompletion derived from static types. Our job profiles might differ here, but speaking for me, most of the daily work consists of the latter - I rarely have to come up with a new algorithm worth the name.

Lots of speculation here, therefore hearing some numbers from Microsoft/Github would be nice. I've read that Microsoft is heading for a solution base on nuclear energy to satisfeed the ever-increasing need stemming from AI (source), but ofc. you can't account all of that to the use of Copilot, and as you pointed out: we can't just ignore that power is spent at other places if we dropped Copilot, so it's not really insightful on its own.

[david-engelmann]

Training wise you are hundred percent correct, pretraining a model is energy expended once for all the downstream users of the model, the energy spent to create chatgpt-3.5 is spread out by every user of that model for each call (per request a very small number). The "personalization" is based on the LLM provider and your contract with them. If you've paid for a premium service such as Sourcegraph they will create a graph representation of your code (think tree-sitter). When its time for the LLM to generate an answer, they take the question asked and do a RAG search on the graph representation of the code and they prompt the generally trained model with the relevant code its retrieved from the RAG search and your question (or in the case of autocomplete, what's been typed already). While RAG search is vector math and not without a resource cost, modern machine are really good at these calculations. The frequency that call is made is going to be based on the tool you use, whether that's per key stroke, per word or whatnot (Sourcegraph for example can be a chat triggered calls from within the IDE, which only occur when you interact with the chat). There may be tools that are creating custom trained LLM models based on your code base but they aren't the typical solution, as that process is expensive and the improvement to the results would be minimal and become dated very quickly. Comparing a developers resource usage with and without LLM-based tools are difficult and is widely influenced by the type of programmer. Someone using lots of third-party tools with lots of functions they've not wrote could see huge benefits in a tool that immediately tells you what function to call and what arguments to pass and what those arguments correspond to in the current scope. If you already wrote the code many times. then yes copy and pasting it will be more efficient. That's not where my time is spent. All this being said it makes the assumption that the LLM generated code is valid and efficient or the programmer using the tools has the ability to notice and correct it's deficiency easily. For experienced users, its probably more efficient, for novice its probably a huge time and resource suck

[JonasDoe]

Oh, so understand correctly that Copilot's proposals don't require a full-fledged GPT request under all circumstances, but often just rely on a Graph-based search powered by LLM provider? I tried to get an idea what RAG search is doing: looks like it uses GPT to transform its result into an adequate, request-related response. But doesn't this last step mean the same power consumption as if we queried GPT directly, plus the - probably neglectable - power required for the Graph search on top?

[david-engelmann]

The power consumption is the graph search to get the relevant code for the query and the LLM query itself. It depends on how the plugin is setup. my assistant makes a query when i stop typing for a quarter of a second, while typing the previous queries results are present in the drop down until I pause typing and it refreshes the results.

[JonasDoe]

I see, but this is what I'm expecting to be rather heavy on power consumption - lots of queries against the LLM.

[cmp-nct]

Given that CO2 is the second most important food of plants, right after sunlight I do hope my use of Copilot does emit a lot of extra CO2.
I like plants, I also like food. It's win-win.

Though I fear the impact is neglectable :( After all training is only done rarely and then millions of users use the finished models.
Also a single jet round trip sounds really like not enough :( After all aviation produces just 2% of the annual human CO2 (all trips together).

We will need to look at another solution for generating plant food! (Do you run a greenhouse by chance ?)

[zuckerruebe]

Your plants don't like it so much when the temperature changes too fast for them to evolve with it. Neither do I.

[gokhan-or]

I'm wondering how much energy the use of GitHub Copilot costs and how much CO2/GHG it emits.

I realize GitHub is carbon neutral, but what about Copilot? If GitHub wasn't carbon neutral, how much would we be looking at here?

Getting some stats about the training would be great, too, as I've read articles about GPT-3 training emitting 3 times the "emissions of a round trip flight of a passenger jet between San Francisco and New York"

Are there projections for how this might change in the coming years?

Much appreciated!

[ahengst]

Food for thought: (not an answer) Your Q feeds into a bigger question: could IP or HTTP be modified to reflect the energy used* to produce a response? I've thought/discussed this enough to know it's probably impossible to pull off. But at least one RFC was proposed - it failed (I think it was flawed).
*code and CPUs are able to estimate energy much more readily than CO2 byproducts, which vary depending on grid. KISS principle.
*reducing energy (even green energy) consumption will indirectly reduce fossil energy production.

We live in a world of energy blindness and I've been putting off using ANY kind of AI because it just sounds terrible from an energy perspective. It's OK to ask about working "harder not smarter", but we can't do that without having any solid numbers either way.

[ahengst]

-- I'm writing this just as I'm (finally) signing up for copilot. Useful comments above, thx.
-- Whether github is "CO2 neutral" does not change the fact that emissions are occurring and could be reduced.

[ycouble]

I do not have the complete answer, but here are some sourced elements that can help thinking about this problem :

• Yes training an LLM is energetically intensive and produces more or less carbon depending on the grid CO2 intensity of the location where is is run. GPT-3 and BLOOM, two LLMs that are probably similar in size to GH Copilot (?), required respectively 1287 MWh and 433 MWh, generating respectively 522 tCO2eq (US) and 30 tCO2eq (France) (source and gpt-3 source)
• Inference is not negligible and neither equivalent to a simple API request. LLM inference requires to run full forward passes through the neural network which represent a huge amount of computation, and running an LLM for inferences is still highly hardware intensive (BLOOM required running on 16 GPU (A100 80GB) for inference). (same source)
• more recent and larger LLMs are much more hardware and energy intensive (GPT-4 expected to be running on 128GPUs clusters for inference)
• Estimates about the energy consumption of a LLM query are still quite rare, but i calculated from the bloom experiment paper data an energy cost of 3.96 Wh / request and corresponding carbon emissions 1,56 gCO2eq / request (GCP US-central), to compare to the 0.3Wh average energy used for an average google request reported by google here in an NYT article which is far from a simple API request.

[JonasDoe]

Thanks for these thoughts! Sounds rather discouraging. 🙈

[ycouble]

There’s a community driven estimator for environnemental impact being built right now: ecologits.ai, they try to estimate co2eq and kwh in production scenarios, on a per-token basis. They are also starting to dig into code completion tools, although i’m not sure how far they went on the topic.