Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update network skeleton docs based on nv24 learnings #12524

Merged
merged 17 commits into from
Sep 30, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions LOTUS_RELEASE_FLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ This document aims to describe how the Lotus maintainers ship releases of Lotus.

- Lotus software use semantic versioning (`MAJOR`.`MINOR`.`PATCH`).
- **`MAJOR` releases** are reserved for significant architectural changes to Lotus.
- **`MINOR` releases** are shipped for network upgrades, API breaking changes, or non-backwards-compatible feature enhancements.
- **`MINOR` releases** are shipped for [network upgrades](./documentation/misc/Building_a_network_skeleton.md#context), API breaking changes, or non-backwards-compatible feature enhancements.
- **`PATCH` releases** contain backwards-compatible bug fixes or feature enhancements.
- Releases are almost always branched from the `master` branch, even if they include a network upgrade. The main exception is if there is a critical security patch we need to rush out. In that case, we would patch an existing release to increase release speed and reduce barrier to adoption.
- We aim to ship a new release of the Lotus Node software approximately every 4 weeks, except during network upgrade periods which may have longer release cycles.
Expand Down Expand Up @@ -71,7 +71,8 @@ Bumps to the Lotus software minor version number (e.g., 1.28.0, 1.29.0) are used
- API breaking changes
- Non-backwards-compatible feature enhancements

Users MUST upgrade to minor releases that include a network upgrade before a certain time to keep in sync with the Filecoin network. We recommend everyone to subscribe to status.filecoin.io for updates when these are happening, as well checking the release notes of a minor version.
Users MUST upgrade to minor releases that include a network upgrade before a certain time to keep in sync with the Filecoin network. We recommend everyone subscribe to status.filecoin.io for updates when these are happening, as well as checking the release notes of a minor version. ([Learn more about how network upgrades relate to Lotus and its key dependencies.](./documentation/misc/Building_a_network_skeleton.md#context))

Users can decide whether to upgrade to minor version releases that don't include a network upgrade. They are still encouraged to upgrade so they get the latest functionality and improvements and deploy a smaller delta of new code when there is a subsequent minor release they must adopt as part of a network upgrade later.

### Patch Releases
Expand Down
130 changes: 109 additions & 21 deletions documentation/misc/Building_a_network_skeleton.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,94 @@
# Network Upgrade Skeleton in Lotus
# Network Upgrade Skeleton in Lotus <!-- omit in toc -->

This guide will walk you through the process of creating a skeleton for a network upgrade in Lotus. The process involves making changes in multiple repositories in the following order:
This guide will walk you through the process of creating a skeleton for a network upgrade in Lotus.

- [Context](#context)
- [Network Upgrade Dependency Versions](#network-upgrade-dependency-versions)
- [Network Upgrade Dependency Relationships](#network-upgrade-dependency-relationships)
- [Setup](#setup)
- [Ref-FVM Checklist](#ref-fvm-checklist)
- [Filecoin-FFI Checklist](#filecoin-ffi-checklist)
- [Go-State-Types Checklist](#go-state-types-checklist)
- [Filecoin-FFI Checklist](#filecoin-ffi-checklist)
- [Lotus Checklist](#lotus-checklist)
- [Special Cases](#special-cases)
- [New types in go-state-types](#new-types-in-go-state-types)

Each repository has its own set of steps that need to be followed. This guide provides detailed instructions for each repository in the proper order.

## Context

### Network Upgrade Dependency Versions
There are these versions at play for a network upgrade:
- Network Version: an incrementing integer prefixed with `nv` that corresponds with the hard fork that Filecoin implementations coordinate around (e.g., nvX).
- FVM Version: The ref-fvm crate version (e.g., FVM_MAJOR_VERSION.y.x).
- Lotus Version: The Lotus go.mod version (e.g., 1.LOTUS_MINOR_VERSION.x).
- Actor Version: The incrementing integer that is associated with a builtin-actors bundle.

### Network Upgrade Dependency Relationships
```mermaid
graph TD
lotus[lotus]
ffi[filecoin-ffi]
gst[go-state-types]
ba[builtin-actors]
fvm[ref-fvm]
proofs[rust-filecoin-proofs-api]

lotus --> gst
lotus -->|via submodule| ffi
lotus -->|via pack script| ba

gst --> ba

ffi --> fvm
ffi --> proofs
ffi --> gst

ba --> fvm
```

Each repository has its own set of steps that need to be followed. This guide will provide detailed instructions for each repository.
The table below gives an overview of how Lotus and its critical dependencies relate to each other and are versioned relative to network versions.

| Repo | <div style="width:250px">For every network upgrade (increase in Network Version)...</div> | Versioning Scheme | Versioning Docs | go.mod direct dependencies | cargo.toml direct dependencies | Other direct dependencies |
| --- | --- | --- | --- | --- | --- | --- |
| `lotus` | There is at least one `lotus` minor version. [^0] | <small>1.LOTUS_MINOR_VERSION.x</small> | [link](https://github.com/filecoin-project/lotus/blob/master/LOTUS_RELEASE_FLOW.md#adopted-conventions) | `go-state-types` | n/a | * `filecoin-ffi` via git submodule<br/>* `builtin-actors` via pack script |
| `filecoin-ffi` | There is at least one `filecoin-ffi` minor version (since `filecoin-ffi` tracks `lotus`). | <small>1.LOTUS_MINOR_VERSION.x</small> | [link](https://github.com/filecoin-project/filecoin-ffi?tab=readme-ov-file#versioning) | `go-state-types` | * `ref-fvm`<br/>* `rust-filecoin-proofs-api` | None |
| `go-state-types` | There are zero or one `go-state-types` minor versions (since `go-state-types` minor versions track `builtin-actors` major versions) | <small>0.ACTORS_VERSION.x</small> | [link](https://github.com/filecoin-project/go-state-types?tab=readme-ov-file#versioning) | None | n/a | None |
| `builtin-actors` | There are zero or one actors major versions (i.e., we can have a new network upgrade without an actors bump) | <small>ACTORS_VERSION.0.x</small> | [link](https://github.com/filecoin-project/builtin-actors?tab=readme-ov-file#versioning) | n/a |`ref-fvm` | None |
| `ref-fvm` | There may be a major version bump. If there isn't, there is at least a minor version bump to enable support for the new network version. | <small>FVM_ MAJOR_VERSION.y.x</small> | [link](https://github.com/filecoin-project/ref-fvm?tab=readme-ov-file#versioning) | n/a | None | None |

[^0]: Exceptional case of no Lotus minor version for when we have two-stage upgrades where one network version enables some new feature and the next version disables the deprecated feature.

## Setup

1. Clone the [ref-fvm](https://github.com/filecoin-project/ref-fvm.git) repository.
1. Create a tracking issue for this effort:

rjan90 marked this conversation as resolved.
Show resolved Hide resolved
Title: Skeleton for nvXX to support development and testing
Body:<

BigLep marked this conversation as resolved.
Show resolved Hide resolved
````
## Done Criteria
There is a network skeleton in Lotus, which bubbles up all the other dependencies, and allows one to run a 2k-network and see that it switches network version from nv(XX-1) --> nvXX

## Notes
1. This is the overarching tracking issue for the network skeleton update, but thare are tasks that needed to be completed in other repos as well. All PRs for this effort can reference this issue.
2. How to create a skeleton in Lotus is documented here: https://github.com/filecoin-project/lotus/blob/master/documentation/misc/Building_a_network_skeleton.md

2. Clone the [filecoin-ffi](https://github.com/filecoin-project/filecoin-ffi.git) repository.
```[tasklist]
## Tasks
- [ ] Skeleton for nv24 to support development and testing ref-fvm
- [ ] Skeleton for nv24 to support development and testing go-state-types
- [ ] Skeleton for nv24 to support development and testing filecoin-ffi
- [ ] Skeleton for nv24 to support development and testing Lotus
```

3. Clone the [go-state-types](https://github.com/filecoin-project/go-state-types) repository.
````

4. Clone the [lotus](https://github.com/filecoin-project/lotus) repository.
2. Clone Repos
1. [ref-fvm](https://github.com/filecoin-project/ref-fvm.git)
2. [go-state-types](https://github.com/filecoin-project/go-state-types)
3. [filecoin-ffi](https://github.com/filecoin-project/filecoin-ffi.git)
4. [lotus](https://github.com/filecoin-project/lotus)

## Ref-FVM Checklist

Expand All @@ -45,17 +115,9 @@ You can take a look at [this Ref-FVM PR as a reference](https://github.com/filec
section. It may be appropriate to duplicate some entries across these crates if the changes are
relevant to multiple crates.

You can take a look at [this PR as a reference](https://github.com/filecoin-project/ref-fvm/pull/2002). Wait for the PR to be merged, then the reviewer will publish a new release.
You can take a look at [this PR as a reference](https://github.com/filecoin-project/ref-fvm/pull/2002).

## Filecoin-FFI Checklist

1. Update the `TryFrom<u32>` implementation for `EngineVersion` in `rust/src/fvm/engine.rs`
- Add the new network version number (XX+1) to the existing match arm for the network version.

2. Patch the FVM-dependency (fvm4 and fvm4_shared) in `rust/cargo.toml` to use the newly published Ref-FVM release.
- Add `features = ["nvXX+1-dev"]`.

You can take a look at this [Filecoin-FFI PR as a reference](https://github.com/filecoin-project/filecoin-ffi/pull/454), which added the skeleton for network version 23.
3. Wait for the PR to be merged and the reviewer to [publish a new release](https://github.com/filecoin-project/ref-fvm/blob/master/CONTRIBUTING.md#releasing).

## Go-State-Types Checklist

Expand Down Expand Up @@ -92,16 +154,36 @@ You can take a look at this [Filecoin-FFI PR as a reference](https://github.com/

👉 You can take a look at this [Go-State-Types PR as a reference](https://github.com/filecoin-project/go-state-types/pull/257), which added the skeleton for network version 23.

1. In a second PR based off your first PR, add a simple migration for the network upgrade:
2. In a second PR based off your first PR, add a simple migration for the network upgrade:

- Copy the system.go template [^1], and add it to your `/builtin/vXX+1/migration` folder.
- Copy the top.go template [^2], and add it to your `/builtin/vXX+1/migration` folder.

👉 You can take a look at this [Go-State-Types PR as a reference](https://github.com/filecoin-project/go-state-types/pull/258), which added added a simple migration for network version 23.
👉 You can take a look at this [Go-State-Types PR as a reference](https://github.com/filecoin-project/go-state-types/pull/258), which added a simple migration for network version 23.

3. [Follow the release process](https://github.com/filecoin-project/go-state-types#release-process) to publish `v0.NEW_VERSION.0-dev`

rjan90 marked this conversation as resolved.
Show resolved Hide resolved
👉 You can take a look at this [Go-State-Types PR as a reference](https://github.com/filecoin-project/go-state-types/pull/306), which was for network version 24.

## Filecoin-FFI Checklist

1. Update the `TryFrom<u32>` implementation for `EngineVersion` in `rust/src/fvm/engine.rs`
- Add the new network version number (XX+1) to the existing match arm for the network version.

2. Patch the FVM-dependency (fvm4 and fvm4_shared) in `rust/cargo.toml` to use the newly published Ref-FVM release.
- Add `features = ["nvXX+1-dev"]`.

👉 You can take a look at this [Filecoin-FFI PR as a reference](https://github.com/filecoin-project/filecoin-ffi/pull/454), which added the skeleton for network version 23.

3. [Follow the release process](https://github.com/filecoin-project/filecoin-ffi/blob/master/RELEASE.md) to publish `v1.NEW_LOTUS_MINOR_VERSION.0-dev`

rjan90 marked this conversation as resolved.
Show resolved Hide resolved
👉 You can take a look at this [Filecoin-FFI PR as a reference](https://github.com/filecoin-project/filecoin-ffi/pull/481), which was for network version 24.

Note: one only needs to update `filecion-ffi`'s dependency on `go-state-types` when a network upgrade is introducing new types in `go-state-types` (see [below](#new-types-in-go-state-types)). Otherwise, `filecion-ffi`'s dependency on `go-state-types` is just updated when doing fiinal releases before the network upgrade.

rjan90 marked this conversation as resolved.
Show resolved Hide resolved
## Lotus Checklist

1. To integrate the network skeleton into Lotus, ensure that the relevant releases for ref-fvm, filecoin-ffi, and go-state-types are bubbled up to Lotus.
1. To integrate the network skeleton into Lotus, ensure that the relevant releases for ref-fvm, go-state-types, and filecoin-ffi are bubbled up to Lotus.
- Refer to the [Update Dependencies Lotus tutorial](Update_Dependencies_Lotus.md) for detailed instructions on updating these dependencies in Lotus.
rjan90 marked this conversation as resolved.
Show resolved Hide resolved

1. Import new actors:
Expand Down Expand Up @@ -178,6 +260,12 @@ And you're done! These are all the steps necessary to create a network upgrade s

You can take a look at this [Lotus PR as a reference](https://github.com/filecoin-project/lotus/pull/11964), which added the skeleton for network version 23.

## Special Cases

### New types in go-state-types
Typically it's safe to not upgrade filecoin-ffi's version of go-state-types. The exception is when we add a new type (e.g., a new proof variant). In that case, filecoin-ffi needs to be tracking the dev/rc releases of go-state-types. During network skeleton time, any new `go-state-type` types are likely not known, so even if one needs to bump the `filecoin-ffi` version of `go-state-types` to a development/rc release, that will come later in the network upgrade process.

---
[^1]: Here is system.go template for a simple migration:

```go
Expand Down
23 changes: 16 additions & 7 deletions documentation/misc/Update_Dependencies_Lotus.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,15 @@

This guide will walk through how to update the most common dependencies in Lotus. These are the dependencies this guide currently covers:

- [Ref-FVM](#updating-ref-fvm)
- [Filecoin-FFI](#updating-filecoin-ffi)
- [Go-State-Types](#updating-go-state-types)
- [Builtin-Actors](#updating-builtin-actors)
<!-- no toc -->
- [Updating Ref-FVM](#updating-ref-fvm)
- [Updating Filecoin-FFI](#updating-filecoin-ffi)
- [Updating Go-State-Types](#updating-go-state-types)
- [Updating Builtin-Actors](#updating-builtin-actors)

## Context

Updating these dependencies in Lotus is usually related to network upgrades. See [building a network upgrade skeleton in Lotus context](./Building_a_network_skeleton.md#context) for information on the versions and relationships at play.

## Updating Ref-FVM

Expand All @@ -17,11 +22,15 @@ This guide will walk through how to update the most common dependencies in Lotus

## Updating Filecoin-FFI

0. In Lotus´s [go.mod file](https://github.com/filecoin-project/lotus/blob/master/go.mod), search for `filecoin-ffi` and update the version to your wanted version.
BigLep marked this conversation as resolved.
Show resolved Hide resolved

0. Run `go mod tidy`, and commit your changes.

1. In your `lotus` directory, `cd extern/filecoin-ffi`.

2. `git fetch` to ensure you have the latests changes for *filecoin-ffi*.

3. `git checkout vX.XX.X` to checkout the version you want to update to.
3. `git checkout vX.Y.Z` to checkout the version you want to update to.

4. Then commit the update to your Lotus branch and open a PR for updating Filecoin-FFI.

Expand All @@ -43,10 +52,10 @@ This guide will walk through how to update the most common dependencies in Lotus

1. In your `lotus` directory, `cd build/actors`.

2. Run this script `./pack.sh vXX vXX.X.X-rcX` to pull in the builtin-actors bundle into your Lotus repo.
2. Run this script `./pack.sh vXX vX.Y.Z-rcX` to pull in the builtin-actors bundle into your Lotus repo.

- `vXX` is the network version you are bundling this builtin-actors for.
- `vXX.X.X-rcX` is the builtin-actors release you are bundling.
- `vX.Y.Z-rcX` is the builtin-actors release you are bundling.

👉 Example of a [PR updating Builtin-Actors bundle](https://github.com/filecoin-project/lotus/pull/11682/)

Expand Down
Loading