Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement wasi-config #2869

Merged
merged 2 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions crates/expressions/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ edition = { workspace = true }
[dependencies]
anyhow = { workspace = true }
async-trait = { workspace = true }
futures = { workspace = true }
spin-locked-app = { path = "../locked-app" }
thiserror = { workspace = true }

Expand Down
16 changes: 16 additions & 0 deletions crates/expressions/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,22 @@ impl ProviderResolver {
self.resolve_template(template).await
}

/// Resolves all variables for the given component.
pub async fn resolve_all(&self, component_id: &str) -> Result<Vec<(String, String)>> {
use futures::FutureExt;

let Some(keys2templates) = self.internal.component_configs.get(component_id) else {
return Ok(vec![]);
};

let resolve_futs = keys2templates.iter().map(|(key, template)| {
self.resolve_template(template)
.map(|r| r.map(|value| (key.to_string(), value)))
});

futures::future::try_join_all(resolve_futs).await
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am slightly worried about performance - at some point we might want to see if adding some sort of caching mechanism to providers is worth it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably push this down into the provider as e.g. (untested):

trait Provider {
    async fn get(&self, key: &Key) -> anyhow::Result<Option<String>>;

    async fn get_many(&self, keys: impl IntoIterator<Item = &Key>) -> anyhow::Result<impl IntoIterator<Option<String>>> {
        try_join_all(keys.into_iter().map(|key| self.get(key)).await
    }
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think any of our current providers can optimise this beyond running the futures concurrently - all appear to support retrieving only one secret at once - so I propose we defer this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itowlson instead of adding this function, could we use lann's suggestion from get_all

}

/// Resolves the given template.
pub async fn resolve_template(&self, template: &Template) -> Result<String> {
let mut resolved_parts: Vec<Cow<str>> = Vec::with_capacity(template.parts().len());
Expand Down
37 changes: 36 additions & 1 deletion crates/factor-variables/src/host.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use spin_factors::anyhow;
use spin_world::{async_trait, v1, v2::variables};
use spin_world::{async_trait, v1, v2::variables, wasi::config as wasi_config};
use tracing::{instrument, Level};

use crate::InstanceState;
Expand Down Expand Up @@ -37,6 +37,41 @@ impl v1::config::Host for InstanceState {
}
}

#[async_trait]
impl wasi_config::store::Host for InstanceState {
async fn get(&mut self, key: String) -> Result<Option<String>, wasi_config::store::Error> {
match <Self as variables::Host>::get(self, key).await {
Ok(value) => Ok(Some(value)),
Err(variables::Error::Undefined(_)) => Ok(None),
Err(variables::Error::InvalidName(_)) => Ok(None), // this is the guidance from https://github.com/WebAssembly/wasi-runtime-config/pull/19)
Err(variables::Error::Provider(msg)) => Err(wasi_config::store::Error::Upstream(msg)),
Err(variables::Error::Other(msg)) => Err(wasi_config::store::Error::Io(msg)),
}
}

async fn get_all(&mut self) -> Result<Vec<(String, String)>, wasi_config::store::Error> {
let all = self
.expression_resolver
.resolve_all(&self.component_id)
.await;
all.map_err(|e| {
match expressions_to_variables_err(e) {
variables::Error::Undefined(msg) => wasi_config::store::Error::Io(msg), // this shouldn't happen but just in case
variables::Error::InvalidName(msg) => wasi_config::store::Error::Io(msg), // this shouldn't happen but just in case
variables::Error::Provider(msg) => wasi_config::store::Error::Upstream(msg),
variables::Error::Other(msg) => wasi_config::store::Error::Io(msg),
}
})
}

fn convert_error(
&mut self,
err: wasi_config::store::Error,
) -> anyhow::Result<wasi_config::store::Error> {
Ok(err)
}
}

fn expressions_to_variables_err(err: spin_expressions::Error) -> variables::Error {
use spin_expressions::Error;
match err {
Expand Down
1 change: 1 addition & 0 deletions crates/factor-variables/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ impl Factor for VariablesFactor {
fn init<T: Send + 'static>(&mut self, mut ctx: InitContext<T, Self>) -> anyhow::Result<()> {
ctx.link_bindings(spin_world::v1::config::add_to_linker)?;
ctx.link_bindings(spin_world::v2::variables::add_to_linker)?;
ctx.link_bindings(spin_world::wasi::config::store::add_to_linker)?;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want to consider whether we want to make this configurable for users of the factor - i.e., do we want to force all runtimes that use the VariablesFactor to link all three interfaces? We could instead make this linking conditional based on some configuration provided when the factor is constructed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crate feature flag?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the feature is light-weight enough that we could just do a dynamic check instead of a compile time feature flag, but I don't feel strongly either way.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to make a decision on this (and a similar consideration for wasi-keyvalue) fairly soon if we hope to land this in Spin 3.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion is just an API concern for those using this factor as library (in alternate runtimes). Whether Spin CLI turns this feature on, puts it behind a feature flag, or something else, is unrelated to whether the factor allows embedder to turn on wasi config or not.

That all being said, I'm highly in favor of allowing embedders to control what gets linked. I have less strong feelings about how that gets accomplished, but I think I prefer a dynamic check (i.e., a bool, enum, or maybe even bitflags that the embedder passes to the factor to indicate which interfaces should be linked in).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like you're expressing a general principle about allowing embedders to control which interfaces get linked, rather than anything specific to wasi-config, or am I misunderstanding?

Ok(())
}

Expand Down
1 change: 1 addition & 0 deletions crates/world/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ wasmtime::component::bindgen!({
"fermyon:spin/sqlite@2.0.0/error" => v2::sqlite::Error,
"fermyon:spin/sqlite/error" => v1::sqlite::Error,
"fermyon:spin/variables@2.0.0/error" => v2::variables::Error,
"wasi:config/store/error" => wasi::config::store::Error,
},
trappable_imports: true,
});
Expand Down
17 changes: 17 additions & 0 deletions tests/runtime-tests/tests/variables/spin.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
spin_manifest_version = "1"
authors = [""]
description = ""
name = "variables"
trigger = { type = "http" }
version = "0.1.0"

[variables]
variable = { default = "value" }

[[component]]
id = "variables"
source = "%{source=variables}"
[component.trigger]
route = "/..."
[component.config]
variable = "{{ variable }}"
17 changes: 17 additions & 0 deletions tests/runtime-tests/tests/wasi-config/spin.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
spin_manifest_version = "1"
authors = [""]
description = ""
name = "wasi-config"
trigger = { type = "http" }
version = "0.1.0"

[variables]
variable = { default = "value" }

[[component]]
id = "wasi-config"
source = "%{source=wasi-config}"
[component.trigger]
route = "/..."
[component.config]
variable = "{{ variable }}"
8 changes: 8 additions & 0 deletions tests/test-components/components/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions tests/test-components/components/wasi-config/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[package]
name = "wasi-config"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
helper = { path = "../../helper" }
wit-bindgen = "0.16.0"
8 changes: 8 additions & 0 deletions tests/test-components/components/wasi-config/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Variables
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider adding a conformance test for this. The runtime tests here are meant to be Spin CLI specific but this test seems like it should be applied to all Spin compliant runtimes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't do that as part of this PR but I've raised fermyon/conformance-tests#40 to remind us to do so once this merges.


Tests the wasi:config interface.

## Expectations

This test component expects the following to be true:
* Only the variable named "variable" is defined with value "value"
23 changes: 23 additions & 0 deletions tests/test-components/components/wasi-config/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
use helper::ensure_matches;

use bindings::wasi::config::store::{get, get_all};

helper::define_component!(Component);

impl Component {
fn main() -> Result<(), String> {
ensure_matches!(get("variable"), Ok(Some(val)) if val == "value");
ensure_matches!(get("non_existent"), Ok(None));

let expected_all = vec![
("variable".to_owned(), "value".to_owned()),
];
ensure_matches!(get_all(), Ok(val) if val == expected_all);

ensure_matches!(get("invalid-name"), Ok(None));
ensure_matches!(get("invalid!name"), Ok(None));
ensure_matches!(get("4invalidname"), Ok(None));

Ok(())
}
}
1 change: 1 addition & 0 deletions wit/deps/spin@3.0.0/world.wit
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ world http-trigger {
/// The imports needed for a guest to run on a Spin host
world platform {
include fermyon:spin/platform@2.0.0;
import wasi:config/store@0.2.0-draft;
}
30 changes: 30 additions & 0 deletions wit/deps/wasi-runtime-config-2024-09-27/store.wit
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
interface store {
/// An error type that encapsulates the different errors that can occur fetching configuration values.
variant error {
/// This indicates an error from an "upstream" config source.
/// As this could be almost _anything_ (such as Vault, Kubernetes ConfigMaps, KeyValue buckets, etc),
/// the error message is a string.
upstream(string),
/// This indicates an error from an I/O operation.
/// As this could be almost _anything_ (such as a file read, network connection, etc),
/// the error message is a string.
/// Depending on how this ends up being consumed,
/// we may consider moving this to use the `wasi:io/error` type instead.
/// For simplicity right now in supporting multiple implementations, it is being left as a string.
io(string),
}

/// Gets a configuration value of type `string` associated with the `key`.
///
/// The value is returned as an `option<string>`. If the key is not found,
/// `Ok(none)` is returned. If an error occurs, an `Err(error)` is returned.
get: func(
/// A string key to fetch
key: string
) -> result<option<string>, error>;

/// Gets a list of configuration key-value pairs of type `string`.
///
/// If an error occurs, an `Err(error)` is returned.
get-all: func() -> result<list<tuple<string, string>>, error>;
}
6 changes: 6 additions & 0 deletions wit/deps/wasi-runtime-config-2024-09-27/world.wit
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
package wasi:config@0.2.0-draft;

world imports {
/// The interface for wasi:config/store
import store;
}
Loading