Grafana Pyroscope is an open source software project for aggregating continuous profiling data. Continuous profiling is an -observability signal that allows you to understand your workload's resources (CPU, memory, etc...) usage down to the line number.
+Grafana Pyroscope is an open source software project for aggregating continuous profiling data. +Continuous profiling is an observability signal that allows you to understand your workload's resources (CPU, memory, etc.) usage down to the line number. Grafana Pyroscope is fully integrated with Grafana allowing you to **correlate** with other observability signals, like metrics, logs, and traces. +## Continuous profiling + +Continuous profiling is more than just a performance analysis tool. +It's a crucial component in modern software development and operations. +It goes past traditional profiling techniques by providing ongoing, in-depth insights into application performance. + +Continuous profiling goes past the ephemeral, localized nature of traditional profiling (which historically has been more similar to `console.log` or `print statement` debugging) to a structured, centralized approach allows for effective use in production environments. +Put more simply, Pyroscope takes you from a bunch of flamegraph files on your desktop to a database where you can query and analyze production data in a structured way. + +{{% admonition type="tip" %}} +To learn more about continuous profiling, read [the Introduction section]({{< relref "./introduction" >}}). +{{% /admonition %}} + +Pyroscope, in particular, offers the flexibility to either visualize more traditional adhoc data or evolve your applications observability tooling to include more modern continuous profiling capabilities. + + +## Features + Core features of Grafana Pyroscope include: - **Easy to install:** Using its monolithic mode, you can get Grafana Pyroscope up and @@ -43,3 +60,7 @@ Core features of Grafana Pyroscope include: - **Natively multi-tenant:** Grafana Pyroscope's multi-tenant architecture enables you to isolate data and queries from independent teams or business units, making it possible for these groups to share the same database. + +## Get started with Pyroscope + +Begin your journey with Pyroscope. Visit our [Getting Started Guide](/docs/pyroscope/latest/get-started/) to learn about the different ways to instrument your application with Pyroscope. Join our [Slack community](https://grafana.slack.com/archives/C049PLMV8TB) and contribute to the evolving world of continuous profiling. diff --git a/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md b/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md index 3f4ac4c868..7b59c98951 100644 --- a/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md +++ b/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md @@ -5,7 +5,6 @@ description: How to use the Pyroscope UI to analyze performance of your applicat weight: 40 keywords: - pyroscope - - UI - performance analysis - flamegraphs --- @@ -96,16 +95,4 @@ Using the same examples from above here is a diff between two label sets: -### Using Pyroscope within Grafana -One of the major benefits of Pyroscope is that it can be used alongside the other Grafana tools such as Loki, Tempo, Mimir, and k6. This allows you to use Pyroscope to get the most granular insight into your application and how you can use it to fix issues that you may have identified via metrics, logs, traces, or anything else. - -You can use Pyroscope within Grafana by using the Pyroscope datasource plugin. This plugin allows you to query Pyroscope data from within Grafana and visualize it alongside your other Grafana data. - -For example here is a screenshot of the explore page where we've combined traces and profiles to be able to see granular line-level detail when available for a trace span. This allows you to see the exact function that is causing a bottleneck in your application as well as a specific request. - - - -And here is an example of how you can integrate profiles into your dashboards. In this case we showing memory profiles alongside panels for logs and metrics to be able to debug OOM errors alongside the associated logs and metrics. - - diff --git a/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md b/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md new file mode 100644 index 0000000000..002f9240b4 --- /dev/null +++ b/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md @@ -0,0 +1,35 @@ +--- +title: Use Pyroscope UI +menuTitle: Use the Pyroscope UI +description: How to use the Pyroscope UI to analyze performance of your applications. +weight: 40 +keywords: + - pyroscope + - performance analysis + - flamegraphs +--- + +## Pyroscope: Continuous Profiling in Action + + + +Pyroscope's UI is designed to make it easy to visualize and analyze profiling data. +There are several different modes for viewing, analyzing, uploading, and comparing profiling data. +We will go into more detail about these modes in the [Pyroscope UI documentation]. +For now, it's important to note that one of the major benefits of continuous profiling is the ability to compare and diff profiling data from two different queries: + +- Comparing two different git commits before and after a code change +- Comparing Staging vs production environments to identify differences in performance +- Comparing performance between two different a/b tests or feature flag experiments +- Comparing memory allocations between two different time periods before and after a memory leak +- etc + +With traditional profiling getting any of this information is much more difficult to organize, properly label, share, or store for later analysis. With Pyroscope, all of this is just a matter of writing the two queries you'd like to compare and clicking a button. + +This UI will also expand over time to better help dig deeper into the data and provide more insights into your application. + +## Seamless integration with observability tools + + + +Pyroscope enhances its value through seamless integration with leading observability tools like Grafana, Prometheus, and Loki. This integration facilitates deeper insights into application performance and aids in addressing issues identified through other monitoring methods. \ No newline at end of file diff --git a/docs/sources/introduction/_index.md b/docs/sources/introduction/_index.md index e3d7910527..0f0c0762fc 100644 --- a/docs/sources/introduction/_index.md +++ b/docs/sources/introduction/_index.md @@ -1,9 +1,8 @@ --- -title: What is Pyroscope? -menuTitle: What is Pyroscope? +title: Introduction +menuTitle: Introduction description: Learn about Pyrsocope and profiling. weight: 10 -draft: true keywords: - Pyroscope - Profiling @@ -11,23 +10,27 @@ keywords: -# What is Pyroscope? +# Introduction - +## Why Pyroscope -Introduce the concept. +Continuous profiling helps teams to quickly identify performance bottlenecks and optimize their applications. +With Grafana Pyroscope, teams can easily profile their applications in production with minimal overhead. +Starting with system-wide observability and drilling down to actionable code-level insights allows teams to identify performance issues in context no matter where they occur, so they can optimize their applications with precision. - +## Core functionality -## Sections +With Pyroscope, you get access to the core profiling functionality, which you can use to find performance bottlenecks and optimize applications. The core functionality includes: - +- Minimal CPU overhead and efficient compression ensure optimal performance. +- Architecture consistent with Loki, Mimir, and Tempo, promoting a smoother user experience. + - Horizontally scalable + - Reliable: Highly available setup ensures consistent uptime, even amidst upgrades or system failures. + - Multi-tenancy Support: makes it possible to run one database for multiple independent teams or business units. + - Cost Effective at Scale: Utilizes object storage, which allows extensive historical data storage without significant costs. +- Advanced Analysis UI: Provides an advanced UI, high-cardinality tag/label handling, and the ability to differentiate performance between tags/labels and time intervals. -## Useful links - - diff --git a/docs/sources/introduction/continuous-profiling/_index.md b/docs/sources/introduction/continuous-profiling/_index.md new file mode 100644 index 0000000000..1fa0da4a3a --- /dev/null +++ b/docs/sources/introduction/continuous-profiling/_index.md @@ -0,0 +1,60 @@ +--- +title: When to use continuous profiling +menuTitle: When to use continuous profiling +description: Discover the benefits of continuous profiling and its role in modern application performance analysis. +weight: 20 +keywords: + - pyroscope + - phlare + - continuous profiling + - flamegraphs +--- + +## When to use continuous profiling + +**Continuous profiling** is a systematic method of collecting and analyzing performance data from production systems. + +Traditionally, profiling has been used more as an ad-hoc debugging tool in languages like Go and Java. +You are probably used to running a benchmark tool locally and getting a pprof file in Go or maybe connecting into a misbehaving prod instance and pulling a flamegraph from a JFR file in Java. +This is great for debugging but not so great for production. + + + +Continuous profiling is a modern approach which is safer and more scalable for production environments. +It makes use of low overhead sampling to collect profiles from production systems and stores them in a database for later analysis. +You can get a more holistic view of your application and how it behaves in production. + +## Benefits + + + +Why prioritize continuous profiling? + +1. **In-Depth Code Insights:** It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance. +2. **Complements Other Observability Tools:** Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy. +3. **Proactive Performance Optimization:** Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications. + +## Use cases + + + +Adopting continuous profiling with tools like Pyroscope can lead to significant business advantages: + +1. **Reduced operational costs:** Optimization of resource usage can significantly cut down cloud and infrastructure expenses +2. **Reduced latency:** Identifying and addressing performance bottlenecks leads to faster and more efficient applications +3. **Enhanced incident management:** Faster problem identification and resolution, reducing Mean Time to Resolution (MTTR) and improving end-user experience + +### Reduce operational costs + +Pyroscope's low-overhead profiling enables precise optimization of resource usage, directly impacting various cost centers in technology infrastructure. +By providing in-depth insights into application performance, Pyroscope allows teams to identify and eliminate inefficiencies, leading to significant savings in areas like observability, incident management, messaging/queuing, deployment tools, and infrastructure. + +### Reduced latency + +Pyroscope plays a pivotal role in reducing application latency by identifying performance bottlenecks at the code level. +This granular insight allows for targeted optimization, leading to faster application response times, improved user experience, and consequently, better business outcomes like increased customer satisfaction and revenue. + +### Enhanced incident management + +Pyroscope streamlines incident management by offering immediate, actionable insights into application performance issues. +With continuous profiling, teams can quickly pinpoint the root cause of an incident, reducing the mean time to resolution (MTTR) and enhancing overall system reliability and user satisfaction. \ No newline at end of file diff --git a/docs/sources/introduction/flamegraphs.md b/docs/sources/introduction/flamegraphs.md new file mode 100644 index 0000000000..43a7160924 --- /dev/null +++ b/docs/sources/introduction/flamegraphs.md @@ -0,0 +1,37 @@ +--- +title: "Flamegraphs: Visualizing performance data" +menuTitle: Flamegraphs +description: Learn about flamegraphs to help visualize performance data +weight: 100 +keywords: + - Pyroscope + - Profiling +--- + +# Flamegraphs: Visualizing performance data + +A fundamental aspect of continuous profiling is the flamegraph, a convenient way to visualize performance data. +These graphs provide a clear, intuitive understanding of resource allocation and bottlenecks within the application. Pyroscope extends this functionality with additional visualization formats like tree graphs and top lists. + +## How is a flamegraph created? + + + +This diagram shows how code is turned into a flamegraph. In this case Pyroscope would sample the stacktrace of your application to understand how many CPU cycles are being spent in each function. It would then aggregate this data and turn it into a flamegraph. This is a very simplified example but it gives you an idea of how Pyroscope works. + +## What does a flamegraph represent? + + + +Horizontally, the flamegraph represents 100% of the time that this application was running. +The width of each node represents the amount of time spent in that function. +The wider the node, the more time spent in that function. The narrower the node, the less time spent in that function. + +Vertically, the nodes in the flamegraph represent the heirarchy of which functions were called and how much time was spent in each function. +The top node is the root node and represents the total amount of time spent in the application. +The nodes below it represent the functions that were called and how much time was spent in each function. +The nodes below those represent the functions that were called from those functions and how much time was spent in each function. +This continues until you reach the bottom of the flamegraph. + +This is a CPU profile, but profiles can represent many other types of resource such as memory, network, disk, etc. +To understand more about how to read a flamegraph, what the different colors mean, and what other types of profiles exist and when to use them see our flamegraph documentation or [profile types documentation](/docs/pyroscope/next/analyzing-profiles-in-pyroscope-ui/) \ No newline at end of file diff --git a/docs/sources/introduction/profiling.md b/docs/sources/introduction/profiling.md new file mode 100644 index 0000000000..16f332f952 --- /dev/null +++ b/docs/sources/introduction/profiling.md @@ -0,0 +1,69 @@ +--- +title: Profiling fundamentals +menuTitle: Profiling fundamentals +description: Discover the benefits of continuous profiling and its role in modern application performance analysis. +weight: 20 +keywords: + - pyroscope + - continuous profiling + - flamegraphs +--- + +# Profiling fundamentals + +**Profiling** is a technique used in software development to measure and analyze the runtime behavior of a program. By profiling a program, developers can identify which parts of the program consume the most resources, such as CPU time, memory, or I/O operations. This information can then be used to optimize the program, making it run faster or use fewer resources. + +## Traditional profiling (non-continuous) + +Traditional profiling, often referred to as "sample-based" or "instrumentation-based" profiling, has its roots in the early days of computing. Back then, the primary challenge was understanding how a program utilized the limited computational resources available. + +- **Sample-based profiling**: In this method, the profiler interrupts the program at regular intervals, capturing the program's state each time. By analyzing these snapshots, developers can deduce the frequency at which parts of the code execute. + +- **Instrumentation-based profiling**: Here, developers insert additional code into the program that records information about its execution. This approach provides detailed insights but can alter the program's behavior due to the added code overhead. + +### Benefits + +Traditional profiling provides: + +- **Precision**: Offers a deep dive into specific sections of the code. +- **Control**: Developers can initiate profiling sessions at their discretion, allowing for targeted optimization efforts. +- **Detailed Reports**: Provides granular data about program execution, making it easier to pinpoint bottlenecks. + +## Continuous profiling + +As software systems grew in complexity and scale, the limitations of traditional profiling became evident. Issues could arise in production that were not apparent during limited profiling sessions in the development or staging environments. + +This led to the development of **continuous profiling**, a method where the profiling data is continuously collected in the background with minimal overhead. By doing so, developers gain a more comprehensive view of a program's behavior over time, helping to identify sporadic or long-term performance issues. + +### Benefits + +Continuous profiling provides: + +- **Consistent monitoring**: Unlike traditional methods that offer snapshots, continuous profiling maintains an uninterrupted view, exposing both immediate and long-term performance issues. +- **Proactive bottleneck detection**: By consistently capturing data, performance bottlenecks are identified and addressed before they escalate, reducing system downtime and ensuring smoother operations. +- **Broad performance landscape**: Provides insights across various platforms, from varied technology stacks to different operating systems, ensuring comprehensive coverage. +- **Bridging the Dev-Prod gap**: Continuous profiling excels in highlighting differences between development and production: + - **Hardware discrepancies**: Unearths issues stemming from differences in machine specifications. + - **Software inconsistencies**: Sheds light on variations in software components that might affect performance. + - **Real-world workload challenges**: Highlights potential pitfalls when real user interactions and loads don't align with development simulations. +- **Economical advantages**: + - **Resource optimization**: Continual monitoring ensures resources aren't wasted, leading to cost savings. + - **Rapid problem resolution**: Faster troubleshooting means reduced time and monetary investment in issue rectification, letting developers channel their efforts into productive endeavors. +- **Unintrusive operation**: Specifically designed to work quietly in the background, continuous profiling doesn't compromise the performance of live environments. +7. **Real-time response**: It equips teams with the ability to act instantly, addressing issues as they arise rather than post-occurrence, which is crucial for maintaining high system availability. + +## How to choose between traditional and continuous profiling + +In many modern development workflows, both methods are useful. + +### Traditional profiling + + - **When**: During development or testing phases. + - **Advantages**: Offers detailed insights that can target specific parts of code. + - **Disadvantages**: Higher overhead provides only a snapshot in time. + +### Continuous profiling + + - **When**: In production environments or during extended performance tests. + - **Advantages**: Provides a continuous view of system performance, often with minimal overhead, making it suitable for live environments. + - **Disadvantages**: It might be less detailed than traditional profiling due to the need to minimize impact on the running system. \ No newline at end of file diff --git a/docs/sources/introduction/pyroscope-in-grafana.md b/docs/sources/introduction/pyroscope-in-grafana.md index fb5e69fb65..92edb20388 100644 --- a/docs/sources/introduction/pyroscope-in-grafana.md +++ b/docs/sources/introduction/pyroscope-in-grafana.md @@ -3,7 +3,6 @@ title: Pyroscope and profiling in Grafana menuTitle: Pyroscope in Grafana description: Learn about how you can use profile data in Grafana. weight: 200 -draft: true keywords: - Pyroscope - Profiling @@ -14,21 +13,14 @@ keywords: # Pyroscope and profiling in Grafana - +For example here is a screenshot of the explore page where we've combined traces and profiles to be able to see granular line-level detail when available for a trace span. This allows you to see the exact function that is causing a bottleneck in your application as well as a specific request. -Introduce the concept. + - +And here is an example of how you can integrate profiles into your dashboards. In this case we showing memory profiles alongside panels for logs and metrics to be able to debug OOM errors alongside the associated logs and metrics. -## Sections - - - -## Useful links - - + diff --git a/docs/sources/upgrade-guide/_index.md b/docs/sources/upgrade-guide/_index.md index e59ac0fcab..88bab4c0f3 100644 --- a/docs/sources/upgrade-guide/_index.md +++ b/docs/sources/upgrade-guide/_index.md @@ -1,7 +1,7 @@ --- -title: "Upgrade to Grafana Pyroscope v1.0" -menuTitle: "Upgrade to v1.0" -description: "Upgrading to Grafana Pyroscope v1.0" +title: Upgrade your Grafana Pyroscope installation +menuTitle: Upgrade +description: Upgrade your Pyroscope installation to the latest version. weight: 10 keywords: - pyroscope @@ -10,17 +10,29 @@ keywords: - upgrading --- -# Upgrade to Grafana Pyroscope v1.0 +# Upgrade your Grafana Pyroscope installation -Version 1.0 of Pyroscope is a major release that includes breaking changes. This guide explains how to upgrade to v1.0 from previous versions. +You can upgrade an existing Pyroscope installation to the next version. +However, any new release has the potential to have breaking changes that should be tested in a non-production environment prior to rolling these changes to production. + +The upgrade process changes for each version, depending upon the changes made for the subsequent release. + +This upgrade guide applies to on-premise installations and not for Grafana Cloud. + +## Upgrade to Pyroscope 1.0 + +Version 1.0 of Pyroscope is a major release that includes breaking changes. +This guide explains how to upgrade to v1.0 from previous versions. This document describes in detail the changes that we've made to Pyroscope and how they affect you. For convinience, at the end of this guide we provide short checklists for you to follow. -## New Architecture +### New architecture -We're excited to announce the main change to Pyroscope since its acquisition by Grafana Labs: a new horizontally scalable architecture. Our team took unique learnings that we have gained over the years about profiling data and combined it with a battle-tested Cortex architecture that powers other Grafana Labs databases such as Loki, Mimir and Tempo. This means you can now provision Pyroscope as a highly available service backed by cheap object storage, with the ability to scale up and down as needed. +We're excited to announce the main change to Pyroscope since its acquisition by Grafana Labs: a new horizontally scalable architecture. +Our team took unique learnings that we have gained over the years about profiling data and combined it with a battle-tested Cortex architecture that powers other Grafana Labs databases such as Loki, Mimir, and Tempo. +This means you can now provision Pyroscope as a highly available service backed by cheap object storage, with the ability to scale up and down as needed. -## License Change +### License change Pyroscope server is now licensed under the [AGPLv3](https://opensource.org/license/agpl-v3/). All of our client integrations are still licensed under the [Apache 2.0 license](https://opensource.org/license/apache-2-0/). @@ -28,23 +40,23 @@ Pyroscope was founded in 2020 to build a sustainable business around the open so We believe that the AGPLv3 license is the best way forward for Pyroscope. It allows us to continue to build a sustainable business around Pyroscope, while also ensuring that the project remains open source and that the community can continue to use and contribute to it. -## New Docker Repository +### New Docker repository The new Pyroscope Docker repository is located at [grafana/pyroscope](https://hub.docker.com/r/grafana/pyroscope). The old repository at [pyroscope/pyroscope](https://hub.docker.com/r/pyroscope/pyroscope) will no longer be updated. -## Breaking Changes +### Breaking changes Making big leaps means that we have to break some things. We've tried to minimize the impact of these changes as much as possible, but some of them are unavoidable. We apologize for any inconvenience this may cause. We encourage you to contribute to the community by creating new issues or upvoting existing ones with an `og-feature` label in the [Pyroscope GitHub repository](https://github.com/grafana/pyroscope/labels/og-feature). -### Storage Format Changes +#### Storage format changes The new local storage format is entirely new, optimized for object storage. We do not support migrating from the old storage format to the new one. This means that you will lose data when upgrading to v1.0. -### Configuration File Changes +#### Configuration file changes The configuration file parameters as well as the default location for the configuration file have changed. The old config file is usually located at `/etc/pyroscope/server.yml` and the new config file is at `/etc/pyroscope/config.yaml`. You can find detailed descriptions of all configuration parameters [here]({{< relref "../configure-server/reference-configuration-parameters" >}}). -### Dropping Support for Certain Subcommands +#### Dropping support for certain subcommands We stripped the pyroscope CLI of all subcommands that were related to the client side of profiling and only kept the ones that are related to the server side. This means that the following subcommands are no longer supported: * `pyroscope exec` @@ -56,42 +68,42 @@ This strategic shift has been in the works for some time as we transition away f By adopting native integrations, we aim to provide users with a more streamlined and efficient profiling experience, leveraging language-specific tools and libraries to deliver better performance, ease of use, and seamless integration with their respective applications. -### Dropping OAuth Support +#### Dropping OAuth support OAuth was implemented with SQLite, which made sense when Pyroscope was a single binary, but is no longer feasible with the new distributed architecture. Transitioning to a new distributed architecture means that we had to drop support for OAuth. Our recommendation is to use Grafana for visualizing your profiling data. [Grafana 10](/docs/grafana/latest/whatsnew/whats-new-in-v10-0/) comes with native support for Pyroscope and very mature support for OAuth as well as many other authentication methods. -### API Stability +#### API stability Even though this is a major release, the HTTP API is not yet stable and is subject to changes. Going forward we will do our best to keep the APIs backwards compatible and minimize the impact of these changes and provide a migration path. -## Community Call To Action +### Community call to action Pyroscope v1.0 comes with a huge architectural change and we've tried to minimize the impact of these changes as much as possible, but some of them are unavoidable. Our goal is for our users to have as smooth of a transition as possible. Therefore, we encourage you to contribute to the community by creating new issues or upvoting existing ones with an `og-feature` label in the [Pyroscope GitHub repository](https://github.com/grafana/pyroscope/labels/og-feature). Thank you for your feedback and engagement, which play a crucial role in shaping the future of Pyroscope. -### Upgrade Checklists +### Upgrade Checklists for v1.0 -We provide the following checklists to help you upgrade to v1.0: +We provide the following checklists to help you upgrade to v1.0. -### Upgrade Checklist for Docker deployments +#### Upgrade Checklist for Docker deployments -When upgrading to v1.0 we suggest that you follow this checklist: +When upgrading to v1.0, we suggest that you follow this checklist: * Migrate your configuration from the old format to the new format (old config is usually located at `/etc/pyroscope/server.yml` and the new config is at `/etc/pyroscope/config.yaml`). There's a detailed description of all configuration parameters [here]({{< relref "../configure-server/reference-configuration-parameters" >}}). * Upgrade docker image image from `pyroscope/pyroscope` to `grafana/pyroscope`. Link to the new docker image is [here](https://hub.docker.com/r/grafana/pyroscope). * Delete old data (typically found at `/var/lib/pyroscope`). +#### Upgrade Checklist for Helm deployments -### Upgrade Checklist for Helm deployments +When upgrading to v1.0, we suggest that you follow this checklist: -When upgrading to v1.0 we suggest that you follow this checklist: * Migrate your configuration from the old format to the new format (old config is usually located at `/etc/pyroscope/server.yml` and the new config is at `/etc/pyroscope/config.yaml`). There's a detailed description of all configuration parameters [here]({{< relref "../configure-server/reference-configuration-parameters" >}}). -* Delete the old helm chart: +* Delete the old Helm chart: ```bash helm delete pyroscope # replace pyroscope with the name you used when installing the chart ``` -* Install the new helm chart: +* Install the new Helm chart: ```bash kubectl create namespace pyroscope helm repo add grafana https://grafana.github.io/helm-charts diff --git a/docs/sources/why-use-continuous-profiling/_index.md b/docs/sources/why-use-continuous-profiling/_index.md deleted file mode 100644 index fbb15dbcbb..0000000000 --- a/docs/sources/why-use-continuous-profiling/_index.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: "Why Use Continuous Profiling?" -menuTitle: "Why Use Continuous Profiling?" -description: "Discover the benefits of continuous profiling and its role in modern application performance analysis." -weight: 20 -keywords: - - pyroscope - - phlare - - continuous profiling - - flamegraphs ---- - -# Why Use Continuous Profiling? - - - -Continuous profiling is more than just a performance analysis tool; it's a crucial component in modern software development and operations. It goes past traditional profiling techniques by providing ongoing, in-depth insights into application performance. - -Continuous profiling goes past the ephemeral, localized nature of traditional profiling (which historically has been more similar to "console.log" or "print statement" debugging) to a structured, centralized approach allows for effective use in production environments. Put more simply, Pyroscope takes you from a bunch of flamegraph files on your desktop to a database where you can query and analyze production data in a structured way. - -Pyroscope in particular, offers you the flexibility to either visualize more "traditional" adhoc data or evolve your applications observability tooling to include more "modern" continuous profiling capabilities. - -## What is Continuous Profiling? - -Continuous profiling is a systematic method of collecting and analyzing performance data from production systems. - -Traditionally, profiling has been used more as an ad-hoc debugging tool. While used in many languages, particularly in Go and Java many are used to running a benchmark tool locally and getting a pprof file in go or maybe ssh'ing into a misbehaving prod instance and pulling a flamegraph from a JFR file in Java. This is great for debugging but not so great for production. - - - -Continuous profiling is a much more modern approach which is safer and more scalable for production environments. It makes use of low overhead sampling to collect profiles from production systems and stores them in a database for later analysis. This allows you to get a much more holistic view of your application and how it behaves in production. - -## The Core Benefits of Continuous Profiling - - - -Why prioritize continuous profiling? Here are the key reasons: -1. **In-Depth Code Insights:** It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance. -2. **Complements Other Observability Tools:** Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy. -3. **Proactive Performance Optimization:** Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications. - -## Business Impact of Continuous Profiling - - - -Adopting continuous profiling with tools like Pyroscope can lead to significant business advantages: -1. **Reduced Operational Costs:** Optimization of resource usage can significantly cut down cloud and infrastructure expenses -2. **Latency reduction:** Identifying and addressing performance bottlenecks leads to faster and more efficient applications -3. **Enhanced Incident Management:** Faster problem identification and resolution, reducing Mean Time to Resolution (MTTR) and improving end-user experience - -### How Pyroscope helps cut costs -Pyroscope's low-overhead profiling enables precise optimization of resource usage, directly impacting various cost centers in technology infrastructure. By providing in-depth insights into application performance, Pyroscope allows teams to identify and eliminate inefficiencies, leading to significant savings in areas like observability, incident management, messaging/queuing, deployment tools, and infrastructure. - -### How Pyroscope helps reduce latency -Pyroscope plays a pivotal role in reducing application latency by identifying performance bottlenecks at the code level. This granular insight allows for targeted optimization, leading to faster application response times, improved user experience, and consequently, better business outcomes like increased customer satisfaction and revenue. - -### How Pyroscope helps with incident management -Pyroscope streamlines incident management by offering immediate, actionable insights into application performance issues. With continuous profiling, teams can quickly pinpoint the root cause of an incident, reducing the mean time to resolution (MTTR) and enhancing overall system reliability and user satisfaction. - -## Flamegraphs: Visualizing Performance Data - -A fundamental aspect of continuous profiling is the flamegraph, a convenient way to visualize performance data. These graphs provide a clear, intuitive understanding of resource allocation and bottlenecks within the application. Pyroscope extends this functionality with additional visualization formats like tree graphs and top lists. - - - -This diagram shows how code is turned into a flamegraph. In this case Pyroscope would sample the stacktrace of your application to understand how many CPU cycles are being spent in each function. It would then aggregate this data and turn it into a flamegraph. This is a very simplified example but it gives you an idea of how Pyroscope works. - - - -Horizontally, the flamegraph represents 100% of the time that this application was running. The width of each node represents the amount of time spent in that function. The wider the node, the more time spent in that function. The narrower the node, the less time spent in that function. - -Vertically, the nodes in the flamegraph represent the heirarchy of which functions were called and how much time was spent in each function. The top node is the root node and represents the total amount of time spent in the application. The nodes below it represent the functions that were called and how much time was spent in each function. The nodes below those represent the functions that were called from those functions and how much time was spent in each function. This continues until you reach the bottom of the flamegraph. - -This is a cpu profile, but profiles can represent many other types of resource such as memory, network, disk, etc. To understand more about how to read a flamegraph, what the different colors mean, and what other types of profiles exist and when to use them see our flamegraph documentation or . - -## Pyroscope: Continuous Profiling in Action - - - -Pyroscope's UI is designed to make it easy to visualize and analyze profiling data. There are several different modes for viewing, analyzing, uploading, and comparing profiling data. We will go into more detail about these modes in the [Pyroscope UI documentation]. For now, it is important to note that one of the major benefits of continuous profiling is the ability to compare and diff profiling data from two different queries: -- Comparing two different git commits before and after a code change -- Comparing Staging vs production environments to identify differences in performance -- Comparing performance between two different a/b tests or feature flag experiments -- Comparing memory allocations between two different time periods before and after a memory leak -- etc - -With traditional profiling getting any of this information is much more difficult to organize, properly label, share, or store for later analysis. With Pyroscope, all of this is just a matter of writing the two queries you'd like to compare and clicking a button. - -This UI will also expand over time to better help dig deeper into the data and provide more insights into your application. - -## Seamless Integration with Observability Tools - - - -Pyroscope enhances its value through seamless integration with leading observability tools like Grafana, Prometheus, and Loki. This integration facilitates deeper insights into application performance and aids in addressing issues identified through other monitoring methods. - -## Getting Started with Pyroscope - -Begin your journey with Pyroscope. Visit our [Getting Started Guide](/docs/pyroscope/latest/get-started/) to learn about the different ways to instrument your application with Pyroscope. Join our [slack community](https://grafana.slack.com/archives/C049PLMV8TB) and contribute to the evolving world of continuous profiling.