Allow installing packages directly (not using Git)

[raxod502]

See #148 (comment) and #148 (comment).

[raxod502]

My plan:

1. Factor out the :host handling into an extensible system which can have hosts added dynamically.
2. For each host, add a function that generates a tarball download URL for any given revision, or returns nil if direct download is not supported. We could even support things like CGit, I think.
3. Controllable by a user option and/or recipe keyword, download a tarball instead of a Git repo the first time, and write a file .straight-head or something which contains the SHA of the tarball revision.
4. Repository management operations, instead of doing stuff with Git, just download a new tarball when you need to do a checkout. Alternatively you can ask for them to be converted into full Git repositories.
5. Using find-file-hook or similar, files inside tarball repositories are marked as read-only, and if you try to edit them then straight.el first asks to convert them to full Git repositories (in-place).

The main problem with this approach is that I'm not sure how to get the revision SHA when you're cloning master (i.e. initially). Any ideas?

[raxod502]

cc @vyp @JAremko @et2010 @stonejiajia

[SkySkimmer]

The main problem with this approach is that I'm not sure how to get the revision SHA when you're cloning master (i.e. initially). Any ideas?

$ git ls-remote https://github.com/raxod502/straight.el refs/heads/master
4fb25b607b009ee2837e5fccfb19ffb51ef29eed        refs/heads/master

[raxod502]

Amazing! Thanks.

Requiring two network calls is of course not ideal but I think it would still be an improvement, especially when cloning repositories in parallel.

[yantar92]

This feature might not only be useful to reduce disk space usage. I just encountered a situation when Emacswiki package (dired+) in https://github.com/emacsmirror/ is quite outdated in comparison with the wiki version. This problem is likely unavoidable because packages in emacsmirror are updated manually. I would prefer if straight.el provided an option to download Emacswiki packages directly from Emacswiki to ensure the latest version.

[raxod502]

Sorry, but installing packages from Emacswiki will never be supported. See melpa/melpa#5008 for the reasoning. This issue is specifically for installing snapshots of packages that are hosted on GitHub and similar registries.

[yantar92]

Sorry, but installing packages from Emacswiki will never be supported.

To be frank, I thought that straight.el is possibly the safest way to install packages from such untrusted sources. Unlike direct download, I can easily view diff of the package when updating. Hence, I can directly review any suspicious changes in the package.

[raxod502]

What would straight.el be doing to produce such a diff, and how would it differ from a manual download? I don't see how straight.el downloading an untrusted file would be any more secure than you doing the same.

The only way you get a diff is by installing from a Git-based repository. I don't think straight.el can do anything for you here; installing things directly from Emacswiki is just inherently unsafe. What you want is a Git mirror of Emacswiki, which is exactly what Emacsmirror is. Perhaps there is some problem with the synchronization code that Jonas has yet to address, which is causing that package to be out of date. It could be worth looking into.

There's also https://github.com/emacsmirror/emacswiki.org; perhaps something can be extracted from there. But I don't think anything can be done on the straight.el side; the crucial thing that is missing is an up-to-date Git mirror, as that is what would support a secure installation of Drew's packages. Well... "secure" compared to the alternative, at least.

[yantar92]

What would `straight.el` be doing to produce such a diff, and how would it differ from a manual download? I don't see how `straight.el` downloading an untrusted file would be any more secure than you doing the same.

I imagine that straight.el can create local git repo on first download and then automatically create an "update" commit when the package files change. That commit will contain diff from the last downloaded version of the package.

[raxod502]

Hypothetically such a thing could be done, but it would be a totally novel feature---nothing remotely similar is currently supported in straight.el. I'd really rather not go there in straight.el, as it would add significant complexity and we should be discouraging such outdated methods of package management anyway. In my opinion if the user wants to achieve such functionality, they should write their own script, since it is a rather idiosyncratic use case compared to everything else that straight.el supports, and many UI decisions would have to be made.