Cache env vars with slow exec function calls in templating

[pdecat]

Setting environment variables from commands that are slow affects all prompt executions, e.g. setting:

mise env-vars TEST_MISE="{{exec(command='sleep 1')}}"

which translates into:

[env]
TEST_MISE = "{{exec(command='sleep 1')}}"

will add a 1 second latency at each prompt.

It would be awesome to allow caching such commands.

Note: that's just an example, my actual use case is invoking gopass to set an environment variable with GPG operations depending on my yubikey.

[jdx]

I think me putting caching around templates is too difficult. I have no idea what sort of invalidation logic would work since I have no idea what people might be putting in their files.

Perhaps what we could do though is expose caching inside the templates so you could read/write to the cache instead of performing expensive computations.

[hinricht]

Caching would be great, please have in mind that exec templates can be used to retrieve secrets so env vars shouldn't be cached/stored on disk unencrypted.

i.e.

[env]
TOKEN = "{{exec(command='gopass show --password token')}}"

[hinricht]

Oh just realized this use-case was the original motivation for this FR :)

[pdecat]

I believe the need for caching templates used in environment variables could be eliminated by leveraging the env._.source feature which should only re-execute the configured script if something has changed:

.mise.toml:

[env]
_.source = "./.envrc"

.envrc:

export TEST_SOURCE=test

However, there seems to be an issue with the short circuiting logic as mentioned in #1617 (comment)

Doing a bit of investigation, I've added the following traces in the code:

diff --git a/src/config/env_directive.rs b/src/config/env_directive.rs
index dc995816..9c398a5c 100644
--- a/src/config/env_directive.rs
+++ b/src/config/env_directive.rs
@@ -155,11 +155,11 @@ impl EnvResults {
         };
         let mut paths: Vec<(PathEntry, PathBuf)> = Vec::new();
         for (directive, source) in input.clone() {
-            // trace!(
-            //     "resolve: directive: {:?}, source: {:?}",
-            //     &directive,
-            //     &source
-            // );
+            trace!(
+                "resolve: directive: {:?}, source: {:?}",
+                &directive,
+                &source
+            );
             let config_root = source
                 .parent()
                 .map(Path::to_path_buf)
diff --git a/src/hook_env.rs b/src/hook_env.rs
index 24ccba1e..c5830ad1 100644
--- a/src/hook_env.rs
+++ b/src/hook_env.rs
@@ -25,6 +25,7 @@ pub fn should_exit_early(watch_files: impl IntoIterator<Item = impl AsRef<Path>>
         return false;
     }
     let watch_files = get_watch_files(watch_files);
+    trace!("watch_files: {watch_files:#?}");
     match &*env::__MISE_WATCH {
         Some(watches) => {
             if have_config_files_been_modified(watches, watch_files) {

This revealed that the env._.source script is called before entering should_exit_early:

TRACE  1 [src/config/env_directive.rs:158] resolve: directive: Source("./.envrc"), source: "/home/patrick/workspaces/tests/mise/.mise.toml"
DEBUG  1 [/home/patrick/.cargo/registry/src/index.crates.io-6f17d22bba15001f/xx-1.1.8/src/file.rs:214] glob: "/home/patrick/workspaces/tests/mise/.envrc"
DEBUG  1 [src/cmd.rs:93] $ /usr/bin/bash -c . /home/patrick/workspaces/tests/mise/.envrc
export -p

TRACE  1 [src/config/mod.rs:503] EnvResults {
    env: {
        "TEST_SOURCE": (
            "test",
            "/home/patrick/workspaces/tests/mise/.mise.toml",
        ),
    },
    env_scripts: [
        "/home/patrick/workspaces/tests/mise/.envrc",
    ],
}
TRACE  1 [src/hook_env.rs:28] watch_files: {
    "/home/patrick/.config/mise/config.toml",
    "/home/patrick/.local/share/mise",
    "/home/patrick/.tool-versions",
    "/home/patrick/workspaces/tests/mise/.envrc",
    "/home/patrick/workspaces/tests/mise/.mise.toml",
}
TRACE  1 [src/hook_env.rs:73] config files unmodified
TRACE  1 [src/hook_env.rs:42] early-exit

@jdx Shouldn't the script execution be deferred?

[rmgpinto]

@jdx is there a way to avoid mise re-running the {{ exec(comand='') }} on every prompt?

I have:

[env]
TF_VAR_secret = "{{exec(command='op read \"op://terraform/secret\"')}}"

and was expecting mise to not re-execute the exec portion on every prompt. This issue is old, so I'm wondering if there's a way around this?

Thanks

[hinricht]

❤️

[pdecat]

This is working great, awesome! Thanks @jdx 🚀