Application: Package Management and IPFS

[flyingzumwalt]

Application: Package Management and IPFS

Examples:

• npm on ipfs
• linux package managers on ipfs (ie. pacman on ipfs)
• gx
• javascript CDNs

Use Cases:

• I want to download packages from my neighbor.
• I want to download packages ONCE, not a million times...
• I want to download packages faster (dedup)
• I want my package archive to be smaller in size (dedup)
• I want to (personally or org) backup all packages I use + depend on.
• My org wants to use only secure packages (audited, locked to a hash).

Required features:

• sharding (almost done, experimental)
• better transfer stability
• faster blockstore
• big storage scaling
• cluster replication (compare to how distros have multistage mirrors)
• re-synchronization tooling (ie. keep ipfs up to date as a directory changes, rsync parsing)

[johnny-morrice]

Here is an experiment to index, update, and search structured data over IPFS: Godless

I plan to build a demo in the next month, and am thinking of grabbing this task.

The whole package index would be stored in godless. You then run queries against godless to find the app version you want. Or to find all versions of the app. Or to find the names of all the apps. Godless stores a local index but the data is stored on random peers (indices are synced via firing IPFS hashes over pubsub).

The database would store various metadata (versions, names, descriptions, homepages, emails etc). Links to app binaries in whatever form (.dmg, .deb etc) would be stored direct in IPFS.

On signage, stuff in godless is already signed using libp2p crypto keys, but since it's unlikely the package files themselves would be stored in the database, they would not be signed using libp2p stuff.

I think developers would want to sign packages themselves with PGP keys. I don't think you want to get onto imposing on developers how they should sign things so you could support more than one key type. But PGP/GPG is certainly the standard in the Linux/Unix world. I thought of also supporting multiple types of key for godless internal signage too so that people could use GPG, but I have not done this yet. (It's nice to use the same types of key as IPFS)

Now I've been talking about searching for packages over godless/IPFS using a peer to peer database, but the package management app itself should have its own embedded database that expresses its domain model clearly. Loading data in the app would consist of sucking data out of IPFS/godless, checking it fits the form the app expects, and populating the local database.

Godless uses a web of trust model for sharing information already. So a developer running godless could upload her package locally, and then wait for the index to be shared with other hosts. That's possible now, but an app store website should support a more direct approach where you can fire an IPFS hash at it and it will grab the new index and merge it with its own. Naturally for any of that to work, developers would have to exchange public keys. Thankfully developers already do this for exactly this purpose, although, currently they use PGP for that :)

Just a brain dump of how I would do it, what do you think?

Edit: I don't think this should be limited to one type of package, or for a single system. Naturally different systems have different dependency rules, but I am not sure we should get too deeply into dependency management because honestly it is a dreadful dreadful subject. Maybe we could allow dependency metadata and have users execute their own rules using a third-party software. P.S. The way I see systems going is that apps bundle all their own dependencies (harddrives being cheaper than the education needed to understand a dependency graph, and all).

[chrisdebian]

Personal opinion, I wouldn't call it godless, it just sounds like the beginning of a flame-war. Too much religious angst in the world, already. Personal choice, though. Cheers, Chris chris_debian 2E0FRU

…

On Mon, 3 Jul 2017, 18:14 John Morrice, ***@***.***> wrote: I have been working on an experiment to index, update, and search (semi) structured data over IPFS. [ https://github.com/johnny-morrice/godless](It's called Godless) I am wanting to build a demo in the next month, and am thinking of grabbing this task.* The whole package index would be stored in godless. You then run queries against godless to find the app version you want. Or to find all versions of the app. Or to find the names of all the apps. Godless stores a local index but the data is stored on random peers (indices are synced via firing IPFS hashes over pubsub). The database would store various metadata (versions, names, descriptions, homepages, emails etc). Links to app binaries in whatever form (.dmg, .deb etc) would be stored direct in IPFS. On signage, everything in godless is already signed using libp2p crypto keys, but it's since it's unlikely the files packages themselves would be stored in the database, they would not be signed using libp2p stuff. I think developers would want to sign packages themselves with PGP keys. I don't think you want to get onto imposing on developers how they should sign things so you could support more than one key type. But PGP/GPG is certainly the standard in the Linux/Unix world. I thought of also supporting multiple types of key for godless internal signage too so that people could use GPG, but I have not done this yet. (It's nice to use the same types of key as IPFS) Now I've been talking about searching for packages over godless/IPFS using a peer to peer database, package management app itself should have its own embedded database that expresses its domain model clearly. Loading data in the app would consist of sucking data out of IPFS/godless, checking it fits the form the app expects, and populating the local database. Godless uses a web of trust model for sharing information already. So a developer running godless could upload his package locally, and then wait for the index to be shared with other hosts. That's possible now, but an app store website should support a more direct approach where you can fire an IPFS hash at it and it will grab the new index and merge it with its own. Naturally for any of that to work, developers would have to exchange public keys. Thankfully developers already do this for exactly this purpose, although, currently they use PGP for that :) Just a brain dump... what do you think? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/ipfs/ipfs/issues/246#issuecomment-312696939>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARGrd_1LmybnGiU_KmY3xHnI5E0w1Q0oks5sKSFegaJpZM4MgIbO> .

[johnny-morrice]

I didn't think anyone would care (I am not sure how much the p2p community overlaps with the religious community) but I could choose something more PC if people want. It isn't mentioned much in the source code.

Edit: Godless is just a library/server utility. The package manager would get a name that has much more to do with managing packages.

[chrisdebian]

Just a thought, John. Obviously the product is more important than the name. Chris.

…

On Mon, 3 Jul 2017, 18:50 John Morrice, ***@***.***> wrote: I didn't think anyone would care (I am not sure how much the p2p community overlaps with the religious community) but I could choose something more PC if people want. It isn't mentioned much in the source code. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/ipfs/ipfs/issues/246#issuecomment-312703386>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARGrd1CxhuzGtVKgwRoarTLOdZ13QzMFks5sKSn2gaJpZM4MgIbO> .

[johnny-morrice]

You are a very polite man. Have a thumbs up! :D I think I am going to try and track some more IPFS people down over the next bit and point them to my ideas and see if there any objections. If not, cracking on.

[whyrusleeping]

Hey @johnny-morrice that sounds really interesting. It would be nice to figure out a standard format for a 'signed ipfs object'. I want to add some of this logic to the ipfs keystore. If you have thoughts on what that should look like, maybe open an issue in ipfs/specs or ipfs/notes to discuss?

[johnny-morrice]

That sounds like a great idea @whyrusleeping, I'll do that.

[johnny-morrice]

I put some ideas up here ipfs/specs#160

I have made a tiny skeleton tonight for a really, really basic package manager along the lines I described above. There is nothing there at the moment but maybe in a couple of weeks one could click this link and find some useful stuff: https://github.com/johnny-morrice/pkgthing

I am keen to work on another release of godless lib simultaneously. pkgthing would be the first app to use that shiny new system so I'm sure there will be a lot of rough edges to iron out :)

[ozra]

If someone holds a world view where they think they'll "be infected by evil" or "committing a sin", or such, from using a product with an underlying lib called "godless", then I assume they don't use Linux, or derivatives like Android, since these run hordes of daemons. So: no loss of users.

(Note: I fully respect everyone's right to whatever world view they find useful. I also respect everyone's right to name their pets)

Sound like an interesting project!

[whyrusleeping]

note to future commenters, lets try to stay on topic, i don't want to have to lock this thread.

[traverseda]

Do package managers even make sense with IPFS? As a stopgap, sure. In the long term, I'd like to see importing directly from ipfs, with no transformation or compilation step. You already have everything "installed".

[whyrusleeping]

@traverseda Yeah, thats really the dream. I'd love to just have a hash that represents the entire state of my computer and just say "run this hash".

[logicminds]

Similar application #39