Changing recipe in init file doesn't load new recipe for a package

[vyp]

I'm using straight.el (master branch) with use-package and have the following in my init.el:

(setq use-package-always-ensure t)
(use-package general :demand t)

Which works, general gets installed. But now I want to use a different branch for that package, so I change the second line to:

(use-package general :demand t
  :recipe (:host github :repo "noctuid/general.el" :branch "buttercup"))

Now when I quit and relaunch emacs, I don't think I see any new effect, general is still on the default master branch, not buttercup. I may have missed it in the docs, but is this expected behaviour?

[dieggsy]

I think I've noticed this as well - if you call straight-pull-package on general, it will ask to update the remote. I do think this prompt should probably happen on startup, since the idea is that the init.el dictates the "state" of your emacs config.

[vyp]

Yeah exactly. In fact, I don't even think there should be a prompt and it should 'just work'. For example, if I didn't have general installed previously, and just put the second code snippet in my init.el, then general gets installed and is on the buttercup branch. So I expect the same result if I have the same thing in my init.el. But to have a prompt would mean different behaviour based on some previous state (i.e. the fact that I installed general beforehand with default recipe), and so the init file is no longer the sole source of truth.

[raxod502]

This is expected behavior. Otherwise, it would not be possible to switch branches without updating the recipe in your init-file. Furthermore, it would be impossible to guarantee a fast startup if it were necessary to invoke git(1) once for every repository being loaded.

The idea is that your init-file specifies a single source of truth, which may have arbitrary local modifications applied on top of it. To revert those local modifications, use M-x straight-normalize-all. However, it doesn't make sense to revert them automatically because (1) that would lead to data loss, and (2) it would not be possible, performance-wise.

See also #103 (comment) for more discussion.

In your case @vyp, you would want to M-x straight-normalize-package RET general RET in order to switch branches.

I'm open to suggestions on ways in which the user experience or documentation could be improved in this area.

[vyp]

Yeah so after I made my last comment, I was thinking that requiring such a check at every startup would be way too costly in terms of performance, but then if I for example enter a new package in my init.el, then straight does install it at the next startup, so what's going on here? How does straight do this without being slow, is it a simple case of checking if the local repo exists? (I know can just look at the source code, but thought I might ask as well.)

re: local modifications, I thought about this, and I think the ability to load the local state of the directory is nice and like the way you designed it to be used. For example, even if it's not completely deterministic from the init file, it can be made deterministic pretty easily, and plus this way there's no overhead to just experimenting with quick (or long) arbitrary changes, sort of the way emacs is supposed to be.

I'm open to suggestions on ways in which the user experience or documentation could be improved in this area.

So for me it wasn't clear that I had to use normalize at all, since from the readme it seemed as though it would only be useful if I make local modifications. I guess my confusion comes from the fact that entering a new package into the init.el will install it without needing to do anything else at the next startup, but changing a recipe for an already installed package doesn't do anything. That's not to say I don't want straight to install new packages at the next startup, I like the way it works now, but I was just trying to explain how I came to misunderstand the expected behaviour.

I'm not sure how to address that in the documentation though, but does that help? Maybe, a comment somewhere saying normalize is also useful to realize some changes to the init file, such as modifying a recipe..? (I'm happy to close this issue too if you'd like.)

[raxod502]

is it a simple case of checking if the local repo exists?

Yes.

this way there's no overhead to just experimenting with quick (or long) arbitrary changes, sort of the way emacs is supposed to be.

In other words, the whole reason I made this package manager.

So for me it wasn't clear that I had to use normalize at all, since from the readme it seemed as though it would only be useful if I make local modifications.

Agreed, this is not made clear in the README.

entering a new package into the init.el will install it without needing to do anything else at the next startup, but changing a recipe for an already installed package doesn't do anything

It's because there's no way to distinguish a recipe change from a local modification, or some combination of both.

Maybe, a comment somewhere saying normalize is also useful to realize some changes to the init file, such as modifying a recipe..?

Good idea. This should be made obvious, since it only makes immediate sense from the developer's perspective (i.e. mine), and not the user's.

(I'm happy to close this issue too if you'd like.)

Let's leave it open to track the documentation improvements.

[hrehfeld]

Is this actually documented somewhere? I looked/tried for about 10 minutes how to switch from original package to my fork, and ended up just adding the remote manually. I don't really understand the information in this thread either, it seems to me that adding :fork t and then normalizing the package is suggested, but that didn't do anything it seems. (not even fetching afterwards) :/

[raxod502]

Adding :fork t, reloading your init-file, and then normalizing should cause the remote to be added. Doing straight-fetch-package should fetch from your forked remote. You would also have to straight-merge-package to actually update HEAD to the later commit if you have more stuff on the main branch in your fork.

In terms of documentation, I think the information is technically available in the README, but not all in one place like is necessary to understand it holistically. Sorry, I feel bad for creating such a complex system.

[ntc2]

I'm wondering if there's a way to get the best of both worlds here: to maintain the quick startup, but also notice when a recipe changes and update the installed package automatically. I'm not familiar enough with how the local modification scenario works to understand the data loss concern well, but my feeling is that if you manually edit the local repo, and then change the recipe in the init file, you shouldn't be surprised if the local repo gets checked out again and you lose your unsaved local modifications (but this case can be handled more gracefully with a little more work).

My assumption is that network-based Git operations are too expensive to perform in the common, needs-to-be-fast case, but that checking for the existence of files or reading small text files is OK. So, why not have a file -- let's call it straight/recipe-cache.el for now -- that records the recipes for which each package's repo was checked out? Then on startup you just compare each package recipe in the user's init file to the corresponding package recipe in the straight/recipe-cache.el file. This should be quite cheap, since you're just reading one extra file and comparing some small strings. If the init file's recipe disagrees with the straight/recipe-cache.el file's recipe for some package, then you update the package to match the straight/recipe-cache.el file's recipe.

Now, this simple approach will overwrite any local changes the user made to the local package repo, but as I said that I don't think that would be a big surprise, since it's in response to the user changing the recipe. But of course accidental data loss is not fun, and it seems easy enough recognize this case and tell the user: when an updated recipe is detected, via a difference between the user's init file and straight/recipe-cachel.el, just prompt the user to confirm that the package update should happen, warning that any local modifications to that package will be lost. It's up to the user to determine if data would be lost, and they can just backup the local copy if they're not sure.

It would in theory be possible to only warn when the user has actually made local modifications to the local package, but I don't think it's worth the trouble in general, given that you support a lot of different backends, and for non VC backends you might have to go "full nix" to detect these modifications. But in the case of the Git backend, which I assume covers 99% of practical usage, it would be easy to additionally cache which commit was checked out for the old recipe listed in straight/recipe-cache.el, and then it's simple to use Git to check if any local modifications were made (i.e. if the current commit ID is different from the cached commit ID, or if the repo is "dirty" / has uncommitted modifications). In this way, you wouldn't have spurious warnings about the possibility of local data loss for recipes using Git, and so in most cases the automatic package updates in response to users updating the recipes in their init file would "just work" automatically.

[progfolio]

this simple approach will overwrite any local changes the user made to the local package repo, but as I said that I don't think that would be a big surprise, since it's in response to the user changing the recipe.

There are plenty of ways to modify a package recipe for which one would not want to lose their local repo modifications.
For example, changing the :pre-build, :build, or :post-build steps, :files, etc. So I don't think that's a great measure of when to rebuild from scratch.

[raxod502]

What you're suggesting is very plausible without the need for tradeoffs around data loss, because straight.el supports, by design, interactive reconciliation of local changes with what is specified in the recipe and lockfile. It would be similar to running M-x straight-normalize-package when the last known recipe from the build cache is shown to be outdated. The build cache is not guaranteed to be persistent, so some additional work would be needed to handle all cases correctly, but this is very feasible.