straight.el does not find updated recipes when updating&freezing their repos

[asf-stripe]

I've just narrowed down a weird problem I've had with straight.el for a little while: I keep my emacs config in sync between two machines using git & frozen versions via straight. Every so often, I'll add a new package (recently, eglot and deadgrep) on one machine, and then when I check out the configuration on the other machine, emacs will refuse to start up because it can't find the recipe in any of the recipe packages. This seems to happen more often to things that sort alphabetically before straight's recipe repos (melpa, emacsmirror, etc).

Steps to reproduce:

0. Make a note of the version of melpa that's checked out (we'll use that to simulate transporting the emacs config across the network) - let's say the version is 7726fcef0525bb50b313c6e7ef10fa5b16e12101
1. Find a package that's newly released on melpa (that is, it is not in the version of melpa that's frozen) - let's pick deadgrep.
2. Add that package to the init file
3. Freeze both melpa and that package
4. Remove both the new package from the build & repo dirs: rm -rf ~/.emacs.d/straight/{repos,build}/deadgrep/
5. Checkout the old version of melpa in its repo dir & delete its build dir: (cd ~/.emacs.d/straight/repos/melpa && git reset --hard 7726fcef0525bb50b313c6e7ef10fa5b16e12101) && rm -rf ~/.emacs.d/straight/build/melpa
6. Restart emacs

Result:

Emacs fails to load the init file, saying: error: Could not find package deadgrep in recipe repositories: (org-elpa melpa gnu-elpa emacsmirror)

Workaround: M-x straight-pull-package RET melpa RET

Expected result:

Straight should pull melpa and the other recipe repos first to match the frozen versions, then attempt to install new packages / new versions of other packages.

[raxod502]

There are definitely problems with the UX of moving lockfiles between machines (what is really needed is a way to start Emacs with a command-line flag saying "please normalize and update packages as you go, instead of loading the old versions"), but I think that your particular steps for reproduction result in expected behavior. In step 2, how are you adding the package to the init-file if the recipe isn't yet available in that machine's copy of MELPA? If you are writing a lockfile, it should mean that all the packages are currently using correct revisions.

You seem to want recipe repositories to be pulled automatically. I don't think this is the correct solution. Instead, there should be a way to automate the following steps for updating an Emacs configuration using straight.el, which currently have to be performed manually:

1. Fix remotes using M-x straight-normalize-all
2. Fetch new revisions using M-x straight-fetch-all (or M-x straight-x-fetch-all; that function really needs to be merged into the core library)
3. Reset all packages to correct revisions using M-x straight-thaw-versions

Because as you have noticed, if you don't perform all of these steps, then you may find errors during startup. This is really annoying, frankly, and a big problem for UX. But I don't really think automatic updates is the correct solution—after all, other breakage can also result if all the packages aren't reset to the versions in the lockfile.

[asf-stripe]

You seem to want recipe repositories to be pulled automatically.

Oh, ok, I think my repro steps weren't super clear then: I don't want that at all (-:

The way I usually go about pulling in a new package is:

1. Pull the melpa repo (or whichever one it lives in)
2. use-package (or straight-use-package) it in the init file
3. Freeze
4. Commit both of these changes.

That means now my frozen versions have updates to both the melpa and the new package; but since the new package isn't known until melpa can be pulled, I still get an error on startup. So to work around the problem, I myself have to pull melpa to a potentially untrusted version (though in practice I don't usually validate, ugh), then restart emacs and trust straight to update melpa to the frozen version when it comes up.

What I'd like is for straight to update packages from the frozen file in two steps on startup: First it should update to the frozen version those packages that it already has a recipe for; then try to fetch new ones. Does that make more sense?

[raxod502]

I think what you are missing is running M-x straight-thaw-versions on the new machine. That needs to be done before you start Emacs, otherwise your configuration can't be expected to work. To do the work manually, you can just navigate to ~/.emacs.d/straight/repos/melpa and check out the revision listed in your lockfile.

I think that there should be a way for straight.el to be told to do this automatically when appropriate, but for now this is the workaround.

But maybe your real problem is that straight.el doesn't complete startup successfully, which means that Emacs is not fully functional and isn't aware of all of your packages. This is indeed a major problem—and one which really needs to be fixed—but at least it will be aware of MELPA when it errors, so you should be able to have it thaw the version for MELPA, and then restart.

[raxod502]

Maybe you're expecting straight.el to interact with the Git repositories in some way at startup. That simply isn't done, both for performance and because it allows you to make arbitrary local changes without additional steps. You need to run the interactive functions to request changes to be made to the Git repositories (M-x straight-normalize-all, M-x straight-fetch-all, M-x straight-thaw-versions).

[asf-stripe]

Aha, ok, I think that's exactly where I was wrong - this isn't super clear from the docs, and I think my main confusion stems from straight downloading packages on startup that I added to the init file (and failing for the packages that aren't available in the recipe repos that are currently checked out).

I think I can make my workflow work this way, but I've found a separate weird interaction trying out this normalize/fetch/thaw flow:

1. Run M-x straight-normalize-all, M-x straight-fetch-all, M-x straight-thaw-versions in sequence
2. Run M-x straight-freeze-versions
3. Inspect the versions/default.el file

From your description, I would expect a freeze after a thaw to be a no-op. But if I check the diff on the frozen versions file, it shows that freezing reverted back to the versions that were checked out previously (my canary is ace-window which gets re-frozen to a version from January, when the frozen versions previously contained one from June).

I tried restarting emacs a few times between thaw and freeze, but it's the same behavior every time - freeze reverts frozen versions to the versions that were checked out previously.

[raxod502]

What you describe sounds like a bug. I don't see any problem with your steps. Do you think you can provide a minimal init-file and lockfile that reproduce the problem?

[raxod502]

Actually, it would be best if you could open a new issue about that bug, so we can leave this one for discussion of documentation improvements :)