diff --git a/CHANGELOG.md b/CHANGELOG.md index aed3b6f70ef9..57b444b524bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,17 @@ Removed support for creating Firecracker snapshots targeting older versions of Firecracker. With this change, running 'firecracker --version' will not print the supported snapshot versions. +- [#4230](https://github.com/firecracker-microvm/firecracker/pull/4230): + Changed microVM snapshot format version strategy. Firecracker snapshot format + now has a version that follows semantic versioning and is independent of + Firecracker version. The current version of the snapshot format is v1.0.0. + From now on, the Firecracker binary will define the snapshot format version + it supports and it will only be able to load snapshots with format that is + backwards compatible with that version, according to semantic versioning. + Users can pass the `--snapshot-version` flag to the Firecracker binary to see + its supported snapshot version format. This change renders all previous + Firecracker snapshots (up to Firecracker version v1.5.0) incompatible with + the current Firecracker version. ### Deprecated diff --git a/docs/images/version_graph.png b/docs/images/version_graph.png deleted file mode 100644 index 8b636d3791d7..000000000000 Binary files a/docs/images/version_graph.png and /dev/null differ diff --git a/docs/images/versionize.png b/docs/images/versionize.png deleted file mode 100644 index 4f21be53055c..000000000000 Binary files a/docs/images/versionize.png and /dev/null differ diff --git a/docs/snapshotting/snapshot-support.md b/docs/snapshotting/snapshot-support.md index 301a24c0112d..d4f29e7c91f9 100644 --- a/docs/snapshotting/snapshot-support.md +++ b/docs/snapshotting/snapshot-support.md @@ -174,41 +174,13 @@ The snapshot functionality is still in developer preview due to the following: ## Snapshot versioning -The Firecracker snapshotting implementation offers support for snapshot versioning -(`cross-version snapshots`) in the following contexts: - -- Saving snapshots at older versions - - **DEPRECATED**: This feature is deprecated starting with version 1.5.0. It - will be removed in a subsequent release. After dropping support, Firecracker - will be able to create snapshots only for the version supported by the - Firecracker binary that launched the microVM and not for older versions. - - This refers to being able to create a snapshot with any version in the - `[N, N + o]` interval, while running Firecracker version `N+o`. - - The possibility to save snapshots at older versions might not be offered by - all Firecracker releases. Depending on the features that it introduces, a new - Firecracker release `v` might drop the possibility to save snapshots at any - versions older than `v`. - - For example Firecracker v1.0 and v1.1 adds support for some additional virtio - features (e.g. notification suppression). These features lead the guest - drivers to behave in a very specific way and as a consequence the Firecracker - devices have to respond accordingly. As a result, the snapshots that are - created while these features are in use will not be backwards compatible with - previous versions of Firecracker since the devices that come with these older - versions do not behave in a way that’s compatible with the snapshotted guest - drivers. - - The list of versions that break snapshot backwards compatibility: `1.0`, `1.1` -- Loading snapshots from older versions (being able to load a snapshot created - by any Firecracker version in the `[N, N + o]` interval, in a Firecracker - version `N+o`). - -The design supports an unlimited number of versions, the value of `o` (maximum number -of older versions that we can restore from / save a snapshot to, from the current -version) will be defined later. +The microVM state snapshot file uses a data format that follows semantic +versioning. Each Firecracker binary supports a fixed version of the snapshot +data format. When creating the snapshots it will use the supported data format +version. When loading snapshots, Firecracker will check that the snapshot +version is compatible with the version it supports. More information about the +snapshot data format and details about snapshot data format versions can be +found at [versioning](./versioning.md). ## Snapshot API @@ -293,7 +265,6 @@ curl --unix-socket /tmp/firecracker.socket -i \ "snapshot_type": "Full", "snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file", - "version": "1.0.0" }' ``` @@ -322,12 +293,6 @@ the snapshot. If they exist, the files will be truncated and overwritten. - If diff snapshots were enabled, the snapshot creation resets then the dirtied page bitmap and marks all pages clean (from a diff snapshot point of view). - - If a `version` is specified, the new snapshot is saved at that version, - otherwise it will be saved at the latest snapshot version of the running - Firecracker. The version is only used for the microVM state file as it - contains internal state structures for device emulation, vCPUs and others - that can change their format from a Firecracker version to another. - Versioning is not required for the block and memory files. - _on failure_: no side-effects. @@ -335,11 +300,6 @@ the snapshot. If they exist, the files will be truncated and overwritten. - The separate block device file components of the snapshot have to be handled by the user. -- If specified, `version` must match the firecracker version that introduced a - snapshot version, which may differ from the running Firecracker version. For - example, if you are running on `1.1.2` and want to target version `1.0.4`, you - should specify `1.0.0`. Not specifying `version` uses the latest snapshot - version available to that version. #### Creating diff snapshots @@ -357,7 +317,6 @@ curl --unix-socket /tmp/firecracker.socket -i \ "snapshot_type": "Diff", "snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file", - "version": "1.0.0" }' ``` diff --git a/docs/snapshotting/versioning.md b/docs/snapshotting/versioning.md index b595a524e386..49d4cd8171ec 100644 --- a/docs/snapshotting/versioning.md +++ b/docs/snapshotting/versioning.md @@ -1,44 +1,18 @@ # Firecracker snapshot versioning -This document describes how Firecracker persists its state across multiple -versions, diving deep into the snapshot format, encoding, compatibility and +This document describes how Firecracker persists microVM state into Firecracker +snapshots. It describes the snapshot format, encoding, compatibility and limitations. ## Introduction -The design behind the snapshot implementation enables version tolerant save -and restore across multiple Firecracker versions which we call a version space. -For example, one can pause a microVM, save it to disk with Firecracker version -**0.23.0** and later load it in Firecracker version **0.24.0**. It also works -in reverse: Firecracker version **0.23.0** loads what **0.24.0** saves. - -Below is an example graph showing backward and forward snapshot compatibility. -This is the general picture, but keep in mind that when adding new features -some version translations would not be possible. - -![Version graph]( -../images/version_graph.png?raw=true -"Version graph") - -A non-exhaustive list of how cross-version snapshot support can be used: - -Example scenario #1 - load snapshot from older version: - -* Start Firecracker v0.23 → Boot microVM → *Workload starts* → Pause → - CreateSnapshot(snap) → kill microVM -* Start Firecracker v0.24 → LoadSnapshot → Resume → *Workload continues* - -Example scenario #2 - load snapshot in older version: - -* Start Firecracker v0.24 → Boot microVM → *Workload starts* → Pause → - CreateSnapshot(snap, “0.23”) → kill microVM -* Start Firecracker v0.23 → LoadSnapshot(snap) → Resume → *Workload continues* - -Example scenario #3 - load snapshot in older version: - -* Start Firecracker v0.24 → LoadSnapshot(older_snap) → Resume → - *Workload continues* → Pause → CreateSnapshot(snap, “0.23”) → kill microVM -* Start Firecracker v0.23 → LoadSnapshot(snap) → Resume → *Workload continues* +Firecracker uses the serde crate [1] along with the bincode [2] format to +serialize its state into Firecracker snapshots. Firecracker snapshots have +versions that are independent of Firecracker versions. Each Firecracker version +declares support for a specific snapshot data format version. When creating a +snapshot, Firecracker will use the supported snapshot format version. When +loading a snapshot, Firecracker will check that format of the snapshot file is +compatible with the snapshot version Firecracker supports. ## Overview @@ -61,59 +35,37 @@ emulation, KVM and vCPUs) with 2 exceptions - serial emulation and vsock backend While we continuously improve and extend Firecracker's features by adding new capabilities, devices or enhancements, the microVM state file may change both -structurally and semantically with each new release. The state file includes -versioning information and each Firecracker release implements distinct -save/restore logic for the supported version space. +structurally and semantically with each new release. ## MicroVM state file format -A microVM state file is further split into four different fields: +A Firecracker snapshot has the following format: -| Field | Bits| Description | -|----|----|----| -| magic_id | 64 | Firecracker snapshot, architecture (x86_64/aarch64) and storage version. | -| version | 16 | The snapshot version number internally mapped 1:1 to a specific Firecracker version. | +| Field | Bits | Description | +|-------|------|-------------| +| magic_id | 64 | Firecracker snapshot and architecture (x86_64/aarch64). | +| version | M | The snapshot data format version (`MAJOR.MINOR.PATCH`) | | state | N | Bincode blob containing the microVM state. | -| crc| 64 | Optional CRC64 sum of magic_id, version and state fields. | - -**Note**: the last 16 bits of `magic_id` encode the storage version which specifies -the encoding used for the `version` and `state` fields. The current -implementation sets this field to 1, which identifies it as a [Serde bincode](https://github.com/servo/bincode) -compatible encoder/decoder. - -### Version tolerant ser/de - -Firecracker reads and writes the `state` blob of the snapshot by using per -version, separate serialization and deserialization logic. This logic is mostly -autogenerated by a Rust procedural macro based on `struct` and `enum` -annotations. Basically, one can say that these structures support versioning. -The versioning logic is generated by parsing a structure's history log (encoded -using Rust annotations) and emitting Rust code. - -Versioned serialization and deserialization is divided into two translation layers: - -* field translator, -* semantic translator. - -The _field translator_ implements the logic to convert between different -versions of the same Rust POD structure: it can deserialize or serialize from -source version to target. -The translation is done field by field - the common fields are copied from -source to target, and the fields that are unique to the target are -(de)serialized with their default values. - -The _semantic translator_ is only concerned with translating the semantics of -the serialized/deserialized fields. - -The _field translator_ is generated automatically through a procedural macro, -and the _semantic translation methods_ have to be annotated in the structure -by the user. - -This block diagram illustrates the concept: - -![Versionize]( -../images/versionize.png?raw=true -"Versionize layers") +| crc | 64 | Optional CRC64 sum of magic_id, version and state fields. | + +The snapshot format has its own version encoded in the snapshot file itself +after the snapshot's `magic_id`. The snapshot format version is independent of +the Firecracker version and it follows semantic versioning, i.e. +`MAJOR.MINOR.PATCH`. That means that backwards incompatible changes in the data +format will cause `MAJOR` version number bump, whereas backwards compatible +changes will cause a `MINOR` version number bump. + +For example, that means that a Firecracker binary that supports snapshot data +format `x.y.z` will be able to load all snapshot files with version `x.Y.z` +where `Y <= y`. + +Currently, Firecracker uses the [Serde bincode +encoder](https://github.com/servo/bincode) for serializing the microVM state. +The encoding format that bincode uses does not allow backwards compatible +changes in the state, so essentially every change in the microVM state +description will result in bump of the format's `MAJOR` version. If the needs +arises, we will look into alternative formats that allow more flexibility with +regards to backwards compatibility. ## VM state encoding @@ -132,10 +84,6 @@ Key benefits of using *bincode*: The current implementation relies on the [Serde bincode encoder](https://github.com/servo/bincode). -Versionize is compatible to Serde with bincode backend: structures serialized -with versionize at a specific version can be deserialized with Serde. Also -structures serialized with serde can be deserialized with versionize. - ## Snapshot compatibility ### Host kernel @@ -195,18 +143,8 @@ specifically, the MSRs corresponding to the guest exposed features. ## Implementation -To enable Firecracker cross version snapshots we have designed and built two -crates: - -* [versionize](https://crates.io/crates/versionize) - defines the `Versionize` - trait, implements serialization of primitive types and provides a helper - class to map Firecracker versions to individual structure versions. -* [versionize_derive](https://crates.io/crates/versionize_derive) - exports - a procedural macro that consumes structures and enums and their annotations - to produce an implementation of the `Versionize` trait. - -The microVM state file format is implemented in the [snapshot crate](../../src/snapshot/src/lib.rs) -in the Firecracker repository. -All Firecracker devices implement the [Persist](../../src/snapshot/src/persist.rs) +The microVM state file format is implemented in the [snapshot +crate](../../src/snapshot/src/lib.rs) in the Firecracker repository. All +Firecracker devices implement the [Persist](../../src/snapshot/src/persist.rs) trait which exposes an interface that enables creating from and saving to the microVM state.