Ideas to make zap a package manager that did the one job

[misog]

Linux distributions and package managers are not generally very advanced. There is NIX (or Guix) which is step in better direction I think and it solves dependency problems but the CLI usability is bad and it does not support multiple versions of software in simple commands. Possibly, the concept is flawed, hmhm.

Also, in general, users trust software they run (that it will work, that it will not try to hack their machine, ...). So why to add more trusted maintainers and/or trusted infrastructure to the distribution process? (How one man could have hacked every Mac developer (73% of them, anyway)).

Also, does anybody think that maintainers or distributors can effectively check thousands or hundreds of thousands of source code lines for security bugs / backdoor / malware or masked security bugs / masked backdoor / masked malware? No. So in general, every software that is run by user is trusted by the user. Secure distribution system should decrease the number of trusted actors or trusted systems to 1 (just the system of a software producer) and/or pay for audits with money or time (reproducible builds as part of audits).

AppImage is a nice concept because it solved dependencies issues and zap looks like the missing manager to it.

From average user perspective or from ain't nobody got time for that perspective, user wants to:
a) install the most recent version of a package (stable, nightly, lts, ...)
b) install specific version of a package (v5.0.1, v17.4.2, ...)
c) install multiple versions of a package
d) install the same version of a package multiple times (multiple installations)
e) update package to a specific version
f) update all packages to most recent versions
g) skip automatic updates of some packages
h) remove a package
i) manage default package easily (PATH)

What should hold true:
j) dependencies are solved for good
k) update of a package does not affect dependencies of other packages
l) as many as possible operations are instant/fast
m) if ambiguous state or unclear action, ask
n) allow to force action or force yes to asking

CLI examples (ideas/draft)

lpm (theoretical last package manager)

lpm fetch // loads repositories and other remote data (like 'apt update')
lpm package-install firefox // install most recent firefox
lpm package-install firefox krita php nodejs // install multiple packages
lpm package-install firefox@70.0.3 // install specific version of firefox
lpm package-install firefox --disable-updates // install most recent firefox and disables updates
lpm package-update firefox // update default firefox to most recent version
lpm package-update firefox krita php nodejs // update multiple default packages
lpm package-update firefox --force // update default firefox to most recent version, force update
lpm package-update firefox@70.1.0 // update firefox@70.1.0 to most recent version
lpm package-update firefox->firefox@70.1.0 // update default firefox to specific version
lpm package-update firefox@70.0.0->firefox@70.1.0 // update firefox of specific version to specific version
lpm package-update firefox->firefox@70.1.0 krita@4.2.2->krita@5.0.0 // update multiple packages to specific versions
lpm package-update // update all packages to most recent versions
lpm package-config firefox --disable-updates // disable updates of default firefox
lpm package-config firefox --enable-updates // enable updates of default firefox
lpm package-config firefox@70.0.0 --make-default // make firefox@70.0.0 the default package (sets PATH)
lpm package-search firefox --fulltext|--exact|--regex // search remote firefox in fulltext, exact or regex mode
lpm package-locate firefox --fulltext|--exact|--regex // locate local firefox in fulltext, exact or regex mode

// short versions for professionals or speed
lpm pi firefox
lpm pu firefox
lpm pc firefox --md
lpm pc firefox --du
lpm pc firefox --eu
lpm ps firefox
lpm pl firefox

// advanced usage for professional use or mass updates
lpm package-install firefox --new-instance // install new installation/instance of most recent firefox
lpm package-update firefox[id:2] // update firefox installation/instance with id 2 to most recent version
lpm package-update firefox[id:2|id:5] // update firefox installation/instance with id 2 or id 5 (both) to most recent version
lpm package-update firefox[h:b4ac3b80a84a8916d0628b25a70cacbc4b61f07dcf28591a0a1588615c28c1d5] // update firefox installation/instance with the hash to most recent version
lpm package-update firefox[sh:b4ac3] // update firefox installation/instance with short hash b4ac3 to most recent version
lpm package-update firefox[v:70.0.0] // update firefox installation/instance with version 70.0.0 to most recent version
lpm package-update firefox[v:70.0.0,sh:b4ac3] // update firefox installation/instance with version 70.0.0 and short hash b4ac3 to most recent version (only if the installation with hash b4ac3 is 70.0.0)

// maybe it could support <,=,> operators or ^,~,- operators like operators in semantic versioning https://docs.npmjs.com/about-semantic-versioning

The real issue is package availability and authenticity.

1. Do many companies package their software in appimages?
2. Is there a way to verify authenticity of appimage?
3. Is there a way to verify authenticity of the original software which was wrapped to appimage by random community member or distributor?

If companies publish hash sum of every appimage file (or software wrapped in the appimage) and URL of the hash sum (ex. sha512) located at their site with SSL then this is solved without any distribution certificates. Trusted/source-verified package manager (like zap) just checks the hash sum at the URL and compares the hash with locally computed hash of the appimage (or software wrapped in the appimage).
URLs to hashes can be stored online in GIT or community maintained websites. But the process of comparing hashes must be transparent and it has to actually show the source URL of the hash (ex. firefox.org/releases/...) because GIT or community website can not be verified easily and/or every time and can be hacked any time - see this incident.

So, the question is - how to convince companies to publish hash sums for their appimages (or software wrapped in the appimage)? This should be easier than to convince them to provide enough bandwidth to download appimage packages. Appimage packages could be distributed by whatever unsecured systems (ex. non-SSL sites, whatever malwared package repositories) and it would be OK because the hash verification would just work at the user machine.

Here is a proposal of file standard which allows batch downloads of hashes. The idea of appimages is one app = one file. That means one app = one file = one hash. The format is very simple so it is easy to compose and parse by programmers and easy to read by other humans.

# this is a comment
# one file of hashes for one package
# one line (ending \n or \r or combination) = one record
# every record (line) has its version, currently v1
# format v1 is five words separated by one space:
#format_version file_name package_version hash_function hash_of_file
# package_version format is not defined, recommended format is semver (x.y.z) https://semver.org/
# processing party is responsible for detection and handling of version format (no list of formats in v1)
# package_version format should be consistent across records (lines)
# words with spaces must be wrapped in double quotes ""
# empty lines are allowed
# extension is not defined, recommended is .txt
# hash functions are arbitrary but should be SHA-2 (sha256, sha512) or BLAKE2 or another secure and widely available hash functions
# hash_of_file should be hexadecimal string if that is the conventional format of the used hash function
# hash_of_file should be lowercase if letter case does not change hash value
# all words except file_name must contain only 7-bit ASCII characters which as valid in file names in currently supported Linux and macOS and Windows
# file_name must contain only UTF-8 characters which as valid in file names in currently supported Linux and macOS and Windows
# double quote (") is not allowed in any word (allowed only as word wrapper)
# examples:

v1 appimage_filename_whatever.appimage ver3.6 sha512 32cc3e9b2a03d4e4a4875c427f723b7873d991334b3ef92118fd8e75e4f22d8d01b885276e55f8f3e61c6db2f3644f21ed52a7f73485176c92c123ed5c9ccf07
v1 program_v4.5.6_no_extension v6.0.1-dev sha256 c032d47970e81adad5e0eb6dd6351465a7afa8e388df1524ae79aff3406a21e4
v1 "program with spaces" 4.2.8 sha256 9bda22014bdfb1bc8ccec988e77f281c6f5bdfa4a0b3fb90a77f353cfc67f241
v1 "program with spaces" "5.6.9 dev" sha256 2bd1a722fab45ae34f90485dafc0ea2c0f0326fc74a9180523efc9a6127482a5

[srevinsaju]

I agree with you on much of your ideas.

Putting them into checkboxes,

From average user perspective or from ain't nobody got time for that perspective, user wants to:

• install the most recent version of a package (stable, nightly, lts, ...)
• install specific version of a package (v5.0.1, v17.4.2, ...)
• install multiple versions of a package
• install the same version of a package multiple times (multiple installations)
• update package to a specific version
• update all packages to most recent versions
• skip automatic updates of some packages
• remove a package
• manage default package easily (PATH)

What should hold true:

• dependencies are solved for good (shouldn't be a problem for AppImages, since we don't have dependency chains)
• update of a package does not affect dependencies of other packages
• as many as possible operations are instant/fast
• if ambiguous state or unclear action, ask
• allow to force action or force yes to asking

The real issue is package availability and authenticity.

Agreed.

1. Do many companies package their software in appimages?

We can say, 'some' companies still prefer to package their software as appimages. We will need to look for a quantitative value.

2. Is there a way to verify authenticity of appimage?

We can check the authenticity of the appimage, if the appimage is signed using a gpg key

./path/to/appimage.AppImage --appimage-signature

will output the embedded GPG signature, but I have not seen many appimages that are signed. The other method would
probably be to check their md5 and sha256 sums against the AppImages. But, where do we fetch the original sha256sum or md5 hashes to compare against?

3. Is there a way to verify authenticity of the original software which was wrapped to appimage by random community member or distributor?

No.

If companies publish hash sum of every appimage file (or software wrapped in the appimage) and URL of the hash sum (ex. sha512) located at their site with SSL then this is solved without any distribution certificates. Trusted/source-verified package manager (like zap) just checks the hash sum at the URL and compares the hash with locally computed hash of the appimage (or software wrapped in the appimage).

Yes, thats right. But, we don't have anyone willing to maintain such a list, but it would be great if we could have something like that. But I think, the community gotta take an initiative on keeping the source of AppImage and their sha's up-to-date. I am looking at flathub. It would be great if AppImage could inherit something like this - give the recipe, and let the CI build the AppImage. I can guess this would make us closer to a reproducible AppImage!

So, the question is - how to convince companies to publish hash sums for their appimages (or software wrapped in the appimage)? This should be easier than to convince them to provide enough bandwidth to download appimage packages. Appimage packages could be distributed by whatever unsecured systems (ex. non-SSL sites, whatever malwared package repositories) and it would be OK because the hash verification would just work at the user machine.

That's tricky, and I have no answer to that. It is a question I am personally looking for too. In late 2020s, when I was involved with the AppImage community. But, I think Flatpak is something which knows how to handle this properly. When looking at all the points we are talking about, I sometimes wonder if by trying to put all of these features, we are actually trying to build a second Flatpak? It maybe just me, but the idea on Flatpaks, Snaps and AppImages have become closer, except for the fact that the Flatpaks are not executable on its own. Flatpak build system believes in reproducible builds. This might be related issue, AppImage/AppImageKit#297

Here is a proposal of file standard which allows batch downloads of hashes. The idea of appimages is one app = one file. That means one app = one file = one hash. The format is very simple so it is easy to compose and parse by programmers and easy to read by other humans.

Sounds like a plan. We will need to get that idea over here first: https://github.com/AppImage/AppImageSpec. Apps like zap, appimagetool (tool used to build appimages), all follow (or try to follow) this spec. So, if you can get the idea accepted over there, then this issue will become a piece of cake.

[misog]

From average user perspective or from ain't nobody got time for that perspective, user wants to:

* [x]  install the most recent version of a package (stable, nightly, lts, ...)

* [x]   install specific version of a package (v5.0.1, v17.4.2, ...)

* [ ]  install multiple versions of a package

* [ ]  install the same version of a package multiple times (multiple installations)

* [x]  update package to a specific version

* [x]  update all packages to most recent versions

* [ ]  skip automatic updates of some packages

* [x]  remove a package

* [x]  manage default package easily (PATH)

install multiple versions of a package

I think support of multiple versions in OS is useful because sometimes user wants to try new version but also use older version and does not want to reinstall things repeatedly. Another case is - in some version in GIMP or Krita there was a bug in some action that crashed the program without saving file. How could be that solved easily by average user? Simply by choosing which version of program to run for a specific graphic project.

Multiple versions of a package are implemented by design by AppImage format already. However, proper management of AppImage files to prevent the files to be scattered all over the filesystem is missing from the specification. So this is decided by package managers and therefore all handling is let for package managers.

In MacOS, the default location for .app files is Applications. To have more versions available and integrated (app list, search, ...) is easy - user just renames one file and moves it to the default Applications folder.

In Windows OS, users can almost always install multiple versions of software to different folders to have multiple installations, so it is more difficult to have more versions than in MacOS but it is easy. If integration fails, users just clicks and creates a shortcut.

With AppImage, there is expectation that multiple versions works similarly to iOS - that is true, they works similarly at format level. But they do not integrate and do not provide accessible default location to allow manual management. Package managers that will implement integrations and default locations and not implement multiple versions will break expectation of average user to use multiple versions of software easily.

install the same version of a package multiple times (multiple installations)

I agree that this is not so useful in practice now, but I think about it like this: AppImage format supports multiple installations of the same version of package by design. Just filesystem restricts that the same versions must be named differently in one folder or be in separate folders. The installation process by package managers (search+download+verify+integrate) should mirror all existing properties of AppImage file - including the possibility to just have two same program versions in different locations. In the future it could be a common practice to choose multiple partitions to install and manage software. If a package manager supports multiple versions then multiple installations could be for free with proper implementation.

I like that NIX stores software like this:
/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1/

In NIX that hash will not change after uninstall and reinstall, however maybe each installation of AppImages should have own ID or random hash in filesystem path. Then multiple files relevant to the installation can be stored there. For example, logs of that installation, info about SSL certificate that was used during download, hash of the AppImage in read-only file owned by root... In short, installation should exist as entity.

skip automatic updates of some packages

I think this is important because when I install specific version of a software or when I install newest version of a software, sometimes I want to just use that version and do not update automatically. Reasons can vary - for example I am testing just that version or I know that just that version is the last appropriate version for my usecase. Of course, info can/should be output to user about software with disabled updates.

What should hold true:

* [x]  dependencies are solved for good (shouldn't be a problem for AppImages, since we don't have dependency chains)

* [x]  update of a package does not affect dependencies of other packages

* [ ]  as many as possible operations are instant/fast

* [ ]  if ambiguous state or unclear action, ask

* [ ]  allow to force action or force yes to asking

as many as possible operations are instant/fast

I had experience with nix-env -q (like apt search) and it is very slow because it looks like it queries remote database each time. I like the way of apt because it caches content with apt update. It is very small probability than remote content of apt which is also relevant to me gets updated while I am working with apt.

That is my view of ideal package manager.

[misog]

If companies publish hash sum of every appimage file (or software wrapped in the appimage) and URL of the hash sum (ex. sha512) located at their site with SSL then this is solved without any distribution certificates. Trusted/source-verified package manager (like zap) just checks the hash sum at the URL and compares the hash with locally computed hash of the appimage (or software wrapped in the appimage).

Yes, thats right. But, we don't have anyone willing to maintain such a list, but it would be great if we could have something like that. But I think, the community gotta take an initiative on keeping the source of AppImage and their sha's up-to-date. I am looking at flathub. It would be great if AppImage could inherit something like this - give the recipe, and let the CI build the AppImage. I can guess this would make us closer to a reproducible AppImage!

I think that community maintained hashes are much better state than the current state where just software packages are community maintained. Hashes alone (without URLs to hashes) would be very practical to do audits with if git repo is cloned / distributed by community.

However ideal solution would be that hashes are placed at website owned by producer of the software and package manager just downloads the hash when installing software and community distributors just include hash URLs.

I created this issue - I know that is not the repository of specification, but it is big distributor of AppImages AppImage/appimage.github.io#2830

[misog]

Yes, flathub recipes look good because they are configs... But they allow array of post install scripts from I saw and also even Krita is community-maintained on flathub when at the same time Krita authors publish their own genuine AppImage release here https://krita.org/en/download/krita-desktop/

GIT repo just provides some community eyes but it adds one more trusted and attackable infrastructure into the distribution chain.

AppImages are meant to not contain any recipes:

If a third party (the distribution’s “maintainer”) packages the application for inclusion in the distribution, the third party may make unintended changes to the application not authorized by the original author. This has led to some application authors to ask Linux distributions to no longer distribute their software because they did not like the changes.
https://docs.appimage.org/introduction/advantages.html

They should be packaged by software author. The remaining and unsolved problem is how to verify that some random AppImage on torrent or distribution website was really build by its software author.

This is not exclusive to AppImages. If software author offers .deb file or tar.gz2 or source files then the building process to AppImage could be done locally by package manager and the hash URL can point to that .deb / .tar.gz2 / source file zip...

Reproducible builds are good, but very complex for all parties involved. Also they solve a different problem as hashes. Hashes solve that I run the exact same program that software author produced and reproducible builds solve that someone else build open-source software the same way as me.