Release Standard Operating Procedures (SOP) (#3100)

* commit to view in github

* more release docs

* take 2

* Update releasing.md

* Update prerequisites.md

* rename releasing.md to README.md

* Update create-release-branch.md

* more docs

* Update create-release-branch.md

* Update create-release-branch.md

* Update create-release-branch.md

* Update README.md

* more release pnp

* tidy up a bit

* more docs

* couple doc tweaks

* Update publish-release-candidate.md

* add new actions plus more tuning to docs

* word

* words

* words

* add testing steps

* words

* Update test-release-candidate.md

* doc improvements as I do the stable release

* words

* bunch more docs after publishing a stable release for the first time

* Update README.md

* words

* fix broken link

* some more words

* work in progress

* more docs after doing patch release

* remove old release guide

Signed-off-by: erikbaranowski <39704712+erikbaranowski@users.noreply.github.com>

* remove irrelevant internal process from public release docs

* Apply suggestions from code review

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* merge, probably messed up

Signed-off-by: erikbaranowski <39704712+erikbaranowski@users.noreply.github.com>

* Refactor into task based release docs

Signed-off-by: erikbaranowski <39704712+erikbaranowski@users.noreply.github.com>

* tune ups

Signed-off-by: erikbaranowski <39704712+erikbaranowski@users.noreply.github.com>

---------

Signed-off-by: erikbaranowski <39704712+erikbaranowski@users.noreply.github.com>
Co-authored-by: Robert Fratto <robertfratto@gmail.com>

docs/developer/release/1-create-release-branch.md

@@ -0,0 +1,26 @@
+# Create Release Branch
+
+A single release branch is created for every major or minor release. That release
+branch is then used for all Release Candidates, the Stable Release, and all
+Patch Releases for that major pr minor version of the agent.
+
+## Before you begin
+
+1. Determine the [VERSION_PREFIX](concepts/version.md).
+
+## Steps
+
+1. Determine which commit should be used as a base for the release branch.
+
+2. Create and push the release branch from the selected base commit:
+
+    The name of the release branch should be `release-VERSION_PREFIX`
+    defined above, such as `release-v0.31`.
+
+        > **NOTE**: Branches are only made for VERSION_PREFIX; do not create branches for the full VERSION such as `release-v0.31-rc.0` or `release-v0.31.0`.
+
+    - If the consensus commit is the latest commit from main you can branch from main.
+    - If the consensus commit is not the latest commit from main, branch from that instead.
+
+    > **NOTE**: Don't create any other branches that are prefixed with `release` when creating PRs or
+    those branches will collide with our automated release build publish rules.

docs/developer/release/2-cherry-pick-commits.md

@@ -0,0 +1,17 @@
+# Cherry Pick Commits
+
+Any commits not on the release branch need to be cherry-picked over to it.
+
+## Before you begin
+
+1. If the release branch already has all code changes on it, skip this step.
+
+## Steps
+
+1. Create PR(s) to cherry-pick additional commits into the release branch as needed from main:
+
+    ```
+    git cherry-pick -x COMMIT_SHA
+    ```
+
+    - For example, refer to PR [#3188](https://github.com/grafana/agent/pull/3188) and [#3185](https://github.com/grafana/agent/pull/3185). 

docs/developer/release/3-update-version-in-code.md

@@ -0,0 +1,50 @@
+# Update Version in Code
+
+The project must be updated to reference the upcoming release tag whenever a new release is being made.
+
+## Before you begin
+
+1. Determine the [VERSION](concepts/version.md).
+
+2. Determine the [VERSION_PREFIX](concepts/version.md)
+
+## Steps
+
+1. Create a branch from `main` for [grafana/agent](https://github.com/grafana/agent).
+
+2. Update the `CHANGELOG.md`:
+
+    1. `CHANGELOG.md` Header
+        - First Release Candidate or a Patch Release
+            - Add a new header under `Main (unreleased)` for `VERSION (YYYY-MM-DD)`.
+        - Additional RCV or SRV
+            - Update the header `PREVIOUS_RELEASE_CANDIDATE_VERSION (YYYY-MM-DD)` to `VERSION (YYYY-MM-DD)`. The date may need updating.
+
+    2. Move the unreleased changes we want to add to the release branch from `Main (unreleased)` to `VERSION (YYYY-MM-DD)`.
+
+    3. Update appropriate places in the codebase that have the previous version with the new version determined above.
+
+        * Do **not** update the `operations/helm` directory. It is updated independently from Agent releases.
+
+3. Create a PR to merge to main (must be merged before continuing).
+
+    - Release Candidate example PR [here](https://github.com/grafana/agent/pull/3065)
+    - Stable Release example PR [here](https://github.com/grafana/agent/pull/3119)
+    - Patch Release example PR [here](https://github.com/grafana/agent/pull/3191)
+
+4. Create a branch from `release-VERSION_PREFIX` for [grafana/agent](https://github.com/grafana/agent).
+
+5. Cherry pick the commit on main from the merged PR in Step 3 from main into the new branch from Step 4:
+
+    ```
+    git cherry-pick -x COMMIT_SHA
+    ```
+
+    For a Release Candidate, delete the `Main (unreleased)` header and anything underneath it as part of the cherry-pick. Alternatively, do it after the cherry-pick is completed.
+
+6. Create a PR to merge to `release-VERSION_PREFIX` (must be merged before continuing).
+
+    - Release Candidate example PR [here](https://github.com/grafana/agent/pull/3066)
+    - Stable Release example PR [here](https://github.com/grafana/agent/pull/3123)
+    - Patch Release example PR [here](https://github.com/grafana/agent/pull/3193)
+        - The `CHANGELOG.md` was updated in cherry-pick commits prior for this example. Make sure it is all set on this PR.

docs/developer/release/4-tag-release.md

@@ -0,0 +1,53 @@
+# Tag Release
+
+A tag is required to create GitHub artifacts and as a prerequisite for publishing.
+
+## Before you begin
+
+1. All required commits for the release should exist on the release branch. This includes functionality and documentation such as the `CHANGELOG.md`. All versions in code should have already been updated.
+
+2. Make sure you are up to date on the release branch:
+
+   ``` 
+   git checkout release-VERSION_PREFIX
+   git fetch origin 
+   git pull origin 
+   ```
+
+3. Determine the [VERSION](concepts/version.md).
+
+4. Follow the GitHub [instructions](https://docs.github.com/en/authentication/managing-commit-signature-verification) to set up GPG for signature verification.
+
+5. Optional: Configure git to always sign on commit or tag.
+
+```bash
+git config --global commit.gpgSign true
+git config --global tag.gpgSign true
+```
+
+If you are on macOS or linux and using an encrypted GPG key, `gpg-agent` or `gpg` may be unable
+to prompt you for your private key passphrase. This will be denoted by an error
+when creating a commit or tag. To circumvent the error, add the following into
+your `~/.bash_profile`, `~/.bashrc` or `~/.zshrc`, depending on which shell you are using.
+
+```
+export GPG_TTY=$(tty)
+```
+
+## Steps
+
+1. Tag the release:
+
+    Example commands:
+
+    ```
+    git tag -s VERSION
+    git push origin VERSION
+    ```
+
+2. After a tag has been pushed, GitHub Tasks will create release assets and open a release draft for every pushed tag.
+
+    - This will take ~20-40 minutes.
+    - You can monitor this by viewing the drone build on the commit for the release tag.
+
+    If the Homebrew Formula fails to update, close the existing open PR and re-run the failed CI.

docs/developer/release/5-update-release-branch.md

@@ -0,0 +1,20 @@
+# Update Release Branch
+
+The `release` branch is a special branch that is used for grafana cloud to point at our install scripts and example kubernetes manifests. This is not to be confused with `release-VERSION_PREFIX` created in [Create Release Branch](./1-create-release-branch.md)
+
+## Before you begin
+
+1. The release tag should exist from completing [Tag Release](./4-tag-release.md)
+
+## Steps
+
+1. Force push the release tag to the `release` branch
+
+    ```
+    git fetch
+    git checkout main
+    git branch -f release VERSION
+    git push -f origin refs/heads/release
+    ```
+
+    > **NOTE**: This requires force push permissions on this branch. If this fails, reach out to one of the project maintainers for help. 

docs/developer/release/6-publish-release.md

@@ -0,0 +1,29 @@
+# Publish Release
+
+This is how to publish the release in GitHub.
+
+## Before you begin
+
+1. You should see a new draft release created [here](https://github.com/grafana/agent/releases). If not go back to [Tag Release](./4-tag-release.md).
+
+## Steps
+
+1. Edit the release draft by filling in the `Notable Changes` section with all `Breaking Changes` and `Features` from the CHANGELOG.md.
+
+2. Add any additional changes that you think are notable to the list.
+
+3. Add a footer to the `Notable Changes` section:
+
+    `For a full list of changes, please refer to the [CHANGELOG](https://github.com/grafana/agent/blob/RELEASE_VERSION/CHANGELOG.md)!`
+
+    Do not substitute the value for `CHANGELOG`.  
+
+4. At the bottom of the release page, perform the following:
+    - Tick the check box to "add a discussion" under the category for "announcements".
+    - For a Release Candidate, tick the checkbox to "pre-release".
+    - For a Stable Release or Patch Release, tick the checkbox to "set as the latest release".
+
+5. Optionally, have other team members review the release draft if you wish
+   to feel more comfortable with it.
+
+6. Publish the release!

docs/developer/release/7-test-release.md

@@ -0,0 +1,9 @@
+# Test Release
+
+Validate the new version is working by running it.
+
+## Steps
+
+1. Validate performance metrics are consistent with the prior version.
+
+2. Validate Flow components are healthy.

docs/developer/release/8-update-helm-charts.md

@@ -0,0 +1,37 @@
+# Update Helm Charts
+
+Our Helm charts require some version updates as well.
+
+## Before you begin
+
+1. Install [helm-docs](https://github.com/norwoodj/helm-docs) on macOS/Linux.
+
+## Steps
+
+1. Create a branch from `main` for [grafana/helm-charts](https://github.com/grafana/helm-charts).
+
+2. Update the code:
+
+    1. Copy the content of the last CRDs into helm-charts.
+
+        Copy the contents from agent repo `production/operator/crds/` to replace the contents of helm-charts repo `charts/agent-operator/crds`
+
+    2. Update references of agent-operator app version in helm-charts pointing to release version.
+
+    3. Bump up the helm chart version.
+
+    > **NOTE**: Do not update the README.md manually. Running the 
+    > [helm-docs](https://github.com/norwoodj/helm-docs) utility in the `charts/agent-operator` 
+    > directory will update it automatically.
+
+3. Open a PR, following the pattern in PR [#2233](https://github.com/grafana/helm-charts/pull/2233).
+
+4. Create a branch from `main` for [grafana/agent](https://github.com/grafana/agent).
+
+5. Update the code:
+
+    1. Update `Chart.yaml` with the new helm version and app version.
+    2. Update `CHANGELOG.md` with a new section for the helm version.
+    3. Run `make generate-helm-docs generate-helm-tests` from the repo root.
+
+6. Open a PR, following the pattern in PR [#3126](https://github.com/grafana/agent/3126).

docs/developer/release/9-announce-release.md

@@ -0,0 +1,24 @@
+# Announce Release
+
+You made it! This is the last step for any release.
+
+## Steps
+
+1. Announce the release in the Grafana Labs Community #agent channel.
+
+    - Example RCV message:
+
+    ```
+    :grafana-agent: Grafana Agent RELEASE_VERSION is now available! :grafana-agent:
+    Release: https://github.com/grafana/agent/releases/tag/RELEASE_VERSION
+    Full changelog: https://github.com/grafana/agent/blob/RELEASE_VERSION/CHANGELOG.md
+    We'll be publishing STABLE_RELEASE_VERSION on STABLE_RELEASE_DATE if we haven't heard about any major issues.
+    ```
+
+    - Example Stable Release or Patch Release message:
+
+    ```
+    :grafana-agent: Grafana Agent RELEASE_VERSION is now available! :grafana-agent:
+    Release: https://github.com/grafana/agent/releases/tag/RELEASE_VERSION
+    Full changelog: https://github.com/grafana/agent/blob/RELEASE_VERSION/CHANGELOG.md
+    ```

docs/developer/release/README.md

@@ -0,0 +1,65 @@
+# Releasing
+
+This document describes the process of creating a release for the
+`grafana/agent` repo. A release includes release assets for everything inside
+the repository, including Grafana Agent and Grafana Agent Operator.
+
+The processes described here are for v0.24.0 and above.
+
+# Release Cycle
+
+A typical release cycle is to have a Release Candidate published for at least 48 
+hours followed by a Stable Release. 0 or more Patch Releases may occur between the Stable Release
+and the creation of the next Release Candidate.
+
+# Workflows
+
+Once a release is scheduled, a release shepherd is determined. This person will be 
+responsible for ownership of the following workflows:
+
+## Release Candidate Publish
+1. [Create Release Branch](./1-create-release-branch.md)
+2. [Cherry Pick Commits](./2-cherry-pick-commits.md)
+3. [Update Version in Code](./3-update-version-in-code.md)
+4. [Tag Release](./4-tag-release.md)
+5. [Publish Release](./6-publish-release.md)
+6. [Test Release](./7-test-release.md)
+7. [Announce Release](./9-announce-release.md)
+
+## Additional Release Candidate[s] Publish
+1. [Cherry Pick Commits](./2-cherry-pick-commits.md)
+2. [Update Version in Code](./3-update-version-in-code.md)
+3. [Tag Release](./4-tag-release.md)
+4. [Publish Release](./6-publish-release.md)
+5. [Test Release](./7-test-release.md)
+6. [Announce Release](./9-announce-release.md)
+
+## Stable Release Publish
+1. [Cherry Pick Commits](./2-cherry-pick-commits.md)
+2. [Update Version in Code](./3-update-version-in-code.md)
+3. [Tag Release](./4-tag-release.md)
+4. [Update Release Branch](./5-update-release-branch.md)
+5. [Publish Release](./6-publish-release.md)
+6. [Test Release](./7-test-release.md)
+7. [Update Helm Charts](./8-update-helm-charts.md)
+8. [Announce Release](./9-announce-release.md)
+
+## Patch Release Publish (latest version)
+1. [Cherry Pick Commits](./2-cherry-pick-commits.md)
+2. [Update Version in Code](./3-update-version-in-code.md)
+3. [Tag Release](./4-tag-release.md)
+4. [Update Release Branch](./5-update-release-branch.md)
+5. [Publish Release](./6-publish-release.md)
+6. [Update Helm Charts](./8-update-helm-charts.md)
+7. [Announce Release](./9-announce-release.md)
+
+## Patch Release Publish (older version)
+- Not documented yet (but here are some hints)
+  - somewhat similar to Patch Release Publish (latest version)
+  - find the old release branch
+  - cherry-pick commit[s] into it
+  - don't update the version in the project on main
+  - changes go into the changelog under the patch release version plus stay in unreleased
+  - don't update the `release` branch
+  - don't publish in github as latest release
+  - don't update deployment tools or helm charts

docs/developer/release/concepts/version.md

@@ -0,0 +1,22 @@
+# Version
+
+Grafana Agent uses Semantic Versioning. The next version can be determined
+by looking at the current version and incrementing it.
+
+## Version
+
+To determine the `VERSION` for a Stable Release or Patch Release, use the Semantic Version `vX.Y.Z`.
+
+To determine the `VERSION` for a Release Candidate, append `-rc.#` to the Semantic Version.
+
+- Examples
+  - For example, `v0.31.0` is the Stable Release `VERSION` for the v0.31.0 release.
+  - For example, `v0.31.1` is the first Patch Release `VERSION` for the v0.31.0 release.  
+  - For example, `v0.31.0-rc.0` is the first Release Candidate `VERSION` for the v0.31.0 release.
+
+## Version Prefix
+
+To determine the `VERSION PREFIX`, use only the major and minor version `vX.Y`.
+
+- Examples
+  - `v0.31`