Host container migrations #294

vyaghras · 2024-12-04T21:23:25Z

Issue number:

Closes #

Description of changes:
In case of host container upgrade in BoB repo, we need to write migrations. By this PR we are incorperating the concept of weak and strong setting, where a weak setting will be deleted on upgrade and downgrade and default setting will be populated from the settings-defaults.

migrator: remove all the weak setting and settings-generator
apiserver: add version 2 for /tx and /metadata/settings-generator 
apiclient: update README to document version 2 of /tx API 
datastore: support committing metadata from transactions 
models: add strength enum and Settings generator struct 
constants: Use version 2 API to get setting-generators 
openapi: Add version 2 for /tx and /metadata/settings-generator 
settings-committer: change transaction API endpoint to /v2/tx 
sundog: Parse settings generators as a table 
storewolf: enable parsing setting-generator metadata as table

Testing done:
Refer https://gist.github.com/vyaghras/cc7391ade3b276b223d1814a8770eea7

Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

vyaghras · 2024-12-05T01:47:13Z

☝️ Fixed failing tests

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

cbgbt · 2024-12-06T21:11:01Z

Do you mind adding more detail to the PR description about the intent of this changeset? Moving the testing done to a gist may also help to make the PR description a bit less cluttered.

cbgbt

Really nice work!

I'm still working through the final two commits, but have run out of time at the moment. Will follow up with any additional feedback.

sources/api/storewolf/src/main.rs

sources/api/openapi.yaml

sources/api/datastore/src/filesystem.rs

sources/api/sundog/src/main.rs

sources/api/migration/migrator/src/main.rs

sources/api/sundog/src/main.rs

sources/api/datastore/src/filesystem.rs

sources/api/apiserver/src/server/mod.rs

sources/api/apiserver/src/server/controller.rs

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

sources/models/src/lib.rs

cbgbt · 2025-01-27T19:44:56Z

sources/models/src/lib.rs

+            fn visit_map<M>(self, mut map: M) -> Result<Self::Value, M::Error>
+            where
+                M: MapAccess<'de>,


This is fine, but I'm curious if we can use MapAccessDeserializer as is used in the docs here. Could save us having to maintain this.

This works for a struct directly but fails if we pass an array of the struct.

sources/api/datastore/src/constraints_check.rs

sources/api/apiserver/src/server/controller.rs

sources/api/apiserver/src/server/mod.rs

sources/api/storewolf/src/main.rs

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

sources/api/migration/migrator/src/main.rs

sources/api/apiserver/src/server/controller.rs

sources/api/apiserver/src/server/mod.rs

sources/api/datastore/src/lib.rs

The get API on route /tx is used to fetch the pending settings in pending transaction. This is used in setting-commiter to just log the pending settings before committing them. The version 2 of this api will also return the pending metadata.

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

Updated the `commit_transaction` function to enable committing metadata from pending transactions. In commit transaction we will first commit metadata and then pending keys to correctly perform the check to identify if key exists or not. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages. Refer commit: bottlerocket-os/bottlerocket-core-kit@20a435e Refer PR: bottlerocket-os/bottlerocket-core-kit#294

We need to add these to match changes in Bottlerocket-core-kit Refer commit: bottlerocket-os/bottlerocket-core-kit@a72f6bd PR: bottlerocket-os/bottlerocket-core-kit#294

cbgbt

Nice work!

It's a nit, but I might reword the sundog commit message to something like:

sundog: add support for v2 setting generator api

This gets more at the intent of the change and would be easier to find if we were to come looking for the commit later.

sources/api/apiserver/src/server/mod.rs

Updated the `commit_transaction` function to enable committing strength metadata from pending transactions. In commit transaction we will use constraint check function to validate the pending keys and metadata. The commit will proceed once check_constraints function returns success. The strength handling among pending and committed transaction is as: If pending metadata is strong and committed metadata is weak, commit the pending setting. If pending metadata is weak, committed metadata is strong and the setting is already available, do not downgrade from strong to weak. Otherwise add the strength file with value as weak. If pending and committed metadata are the same, no action is performed. Additionally, made minor changes to metadata functions for improved access and flexibility: Introduced a `committed` field to dynamically access metadata in pending transactions. Replaced the hardcoded use of live committed metadata with this committed variable ans pass Committed::Live from previous usages.

For new setting-generator we process following fields from the defaults TOML file: - command - strength - skip-if-populated This needs to be saved as json in the filesystem.

We will remove the weak settings and all the metadata using the migrator of the destination migrator. The storewolf do not repopulate any metadata or setting, if it is already present. As migrator runs before storewolf, if we will delete the weak settings and all metadata in migrator, storewolf can populate the setting-generator from defaults and sundog will populate the new source using the new setting generator from default. We are deleting all the metadata as we populate that using defaults only and also the same is not possible to set the using API.

Earlier we used to have setting generator as a string, but we have now changed it to a struct containing the command, strength and skip-if-populated. Hence after requesting the settings generators from api, we need to change these in to Setting generator struct instance. We will use default strength strong and skip_if_populated true if the setting generator is a string and use what has been provided in the response otherwise. Then after running the setting generator, we will send the weak and strong setting settings separately to process them using api.

- `/v2/tx`: We will also return the pending metadata along with pending settings(that we used to return in version 1). As the return struct is changing, we are doing versioning of the API. - `v2/metadata/settings-generators`: We will also return the settings-generators(that contains strength and are saved as JSON object in datastore). As we just used to return arrays and string earlier as response for this API, returning object may break the existing usage. Hence we need to version this API. - `/settings`(patch and patchkeypair): For both of these we will set strength metadata. The default strength used is strong. - `/tx/commit` and `/tx/commit_and_apply`: We will commit the pending metadata(that just accounts for strength metadata for now) as part of commit. No changes has been done in apply.

vyaghras force-pushed the host_container_migrations branch 2 times, most recently from e997915 to 4d9addf Compare December 5, 2024 01:46

vyaghras mentioned this pull request Dec 6, 2024

Remove host container migration bottlerocket-os/bottlerocket#4324

Open

vyaghras requested review from bcressey and cbgbt December 6, 2024 20:56

cbgbt reviewed Dec 7, 2024

View reviewed changes

cbgbt reviewed Dec 14, 2024

View reviewed changes

vyaghras mentioned this pull request Jan 2, 2025

Enable rendering bootstrap source #335

Draft

7 tasks

vyaghras force-pushed the host_container_migrations branch from 4d9addf to 90a6b85 Compare January 21, 2025 21:57

vyaghras force-pushed the host_container_migrations branch from 90a6b85 to f189f38 Compare January 22, 2025 16:36

vyaghras force-pushed the host_container_migrations branch from 4830f8c to b1fb425 Compare January 27, 2025 17:08

vyaghras requested a review from cbgbt January 27, 2025 18:09

cbgbt reviewed Jan 27, 2025

View reviewed changes

sources/api/storewolf/src/main.rs Outdated Show resolved Hide resolved

vyaghras force-pushed the host_container_migrations branch from b1fb425 to eea3a3d Compare January 28, 2025 17:48

bcressey reviewed Jan 28, 2025

View reviewed changes

vyaghras added 4 commits January 28, 2025 20:38

constants: Use version 2 API to get setting-generators

7e2c79b

apiclient: update README to document version 2 of /tx API

ad686ed

openapi: Add version 2 for /tx and /metadata/settings-generator

cbfc797

vyaghras force-pushed the host_container_migrations branch from eea3a3d to ba2eb64 Compare January 29, 2025 17:19

cbgbt approved these changes Jan 29, 2025

View reviewed changes

sources/api/apiserver/src/server/mod.rs Outdated Show resolved Hide resolved

vyaghras added 4 commits January 29, 2025 21:41

models: add strength enum and Settings generator struct

62e7bf4

storewolf: enable parsing setting-generator metadata as table

378a6dd

For new setting-generator we process following fields from the defaults TOML file: - command - strength - skip-if-populated This needs to be saved as json in the filesystem.

vyaghras force-pushed the host_container_migrations branch from ba2eb64 to e86464a Compare January 29, 2025 21:42

vyaghras added 2 commits January 29, 2025 21:53

vyaghras force-pushed the host_container_migrations branch from e86464a to 9d67072 Compare January 29, 2025 21:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Host container migrations #294

Host container migrations #294

vyaghras commented Dec 4, 2024 •

edited

Loading

vyaghras commented Dec 5, 2024

cbgbt commented Dec 6, 2024

cbgbt left a comment

cbgbt Jan 27, 2025

vyaghras Jan 27, 2025

cbgbt left a comment

Host container migrations #294

Are you sure you want to change the base?

Host container migrations #294

Conversation

vyaghras commented Dec 4, 2024 • edited Loading

vyaghras commented Dec 5, 2024

cbgbt commented Dec 6, 2024

cbgbt left a comment

Choose a reason for hiding this comment

cbgbt Jan 27, 2025

Choose a reason for hiding this comment

vyaghras Jan 27, 2025

Choose a reason for hiding this comment

cbgbt left a comment

Choose a reason for hiding this comment

vyaghras commented Dec 4, 2024 •

edited

Loading