[Request] Chaotic-AUR repositories

[FireMasterK]

Chaotic-Aur is an automated building repo for AUR packages, while its not a distro, it includes packages for common aur packages.

[RubenKelevra]

Hey @FireMasterK,

thanks for your request. How much storage does this need?

[Martinligabue]

The official website lists @dr460nf1r3 as the main maintainer. He might know the answer, since I didn't find it online, and might also be interested in this project.

[dr460nf1r3]

The repository size is currently 80GB, potentially increasing when adding new packages 😊
Two notes:

• The preferred way of syncing the repo is Syncthing (all mirrors but our CDN one use this)
• If repo size is a problem we might exclude certain big packages not a lot of people use from syncing easily

As maintainer, I'm also quite interested in this as it opens new possibilities for our users :)

[RubenKelevra]

Hey @dr460nf1r3!

Thanks for the info. Sadly, my main server isn't capable of handling so much data additionally, since the setup requires me to keep a local copy and a copy within IPFS.

If you got a server for this, I'm happy to help to set an additional collaborative cluster up, just for Chaotic-AUR.

Let me know if that's an option :)

[RubenKelevra]

The requirements is a user account and two services on a server which gets already the updates via Rsync or can be included into the rsync "group".

My toolset will then just read the sync log after syncing and push the changes to the local IPFS storage and finally publish the new changes to the ipfs cluster, so other cluster-members can fetch it.

[obj-obj]

@RubenKelevra looking at this issue, people are talking about somehow mounting the directory on IPNS so storage doesn’t need to be duplicated, perhaps that could be used? I’m about to go to sleep, but I’ll try to find some documentation tomorrow.

[RubenKelevra]

@RubenKelevra looking at this issue, people are talking about somehow mounting the directory on IPNS so storage doesn’t need to be duplicated, perhaps that could be used? I’m about to go to sleep, but I’ll try to find some documentation tomorrow.

Hm, no this doesn't work.

There are three issues:

1. IPNS mounts are not writable. I could use this for "dry run" rsync calls and then do the changes in a script. But this is quite error prone and I would need to fetch each file via https which is kinda slow compared to rsync.
2. The 'technology' behind files and folders in IPFS is in its infancy and cannot store any attributes except size. I think I tried this and it makes rsync unusable on those folders, because it can't detect any changes. Maybe there's ways around it, but I'm not sure.
3. Mounting folders is pretty crashy. It's actually far away from beeing usable because of this.

Thanks for the heads up :)

---

Anyway there's another option I used a while ago: ZFS deduplication below IPFS and the storage.

But it takes up a bit IO and memory and limits my choices on the IPFS blocksize.

But I might give this another try.

[RubenKelevra]

@dr460nf1r3

The repository size is currently 80GB, potentially increasing when adding new packages blush Two notes:

I've worked on the server to enable ZFS deduplication again. This gives quite some more room, so I'm happy to add this to the cluster:

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ipfs-data   788G   269G   519G        -         -    42%    34%  1.62x    ONLINE  -

But I need a way to sync via rsync, as my setup requires to parse the rsync logs to get the changes.

[dr460nf1r3]

Hey @RubenKelevra, that's some great news! :)

We do have builds.garudalinux.org as rsync address with two options:

• [chaotic] -> the full repo
• [chaotic-minimal] -> the repo minus quartus, unrealtournament4 and urbanterror (big packages most people won't ever use)

How often would you sync btw? 👀

[RubenKelevra]

Hey @RubenKelevra, that's some great news! :)

We do have builds.garudalinux.org as rsync address with two options:

• [chaotic] -> the full repo
• [chaotic-minimal] -> the repo minus quartus, unrealtournament4 and urbanterror (big packages most people won't ever use)

Thanks :)

How often would you sync btw? 👀

@dr460nf1r3 well, I check via HTTPS for changes on the timestamp files once a minute and start syncing if there are changes.

Would that work for your repo?

[dr460nf1r3]

Yeah, that would be absolutely fine. Thank you! 😊

[dr460nf1r3]

Hey! :) Is there any URL btw? I would like to add this to our website, so people can make use of it 😊

[RubenKelevra]

Hey @dr460nf1r3,

just in the process of adding alhp and chaotic-aur to the cluster (will take an hour or so I guess). The URL will be http://chaotic-aur.pkg.pacman.store.ipns.localhost:8080/$arch.

[dr460nf1r3]

Perfect, just added it to our mirrorlist :)

[RubenKelevra]

@dr460nf1r3 this obviously only works with a locally running IPFS daemon. :)

However you can in theory access it through ipfs.io for example, but it's rather slow and might block it if there are too many requests.

So maybe just link to this project's readme? I'll add the URL there too. :)

[RubenKelevra]

There you go:

https://github.com/RubenKelevra/pacman.store/blob/master/README.md#chaotic-aur

[dr460nf1r3]

So maybe just link to this project's readme? I'll add the URL there too. :)

Already took care of that :D Also added it to the recommendations of our website.

There you go:

Thanks!

[RubenKelevra]

Cool! :)

[RubenKelevra]

@dr460nf1r3 while setting it up, I was wondering if you're interested to have a dedicated cluster instance for just Chaotic AUR.

This way you could join without having the full storage requirements of ALL repos, but only chaotic aur.

It's a bit of work to set up, but doesn't increase my storage requirements, as the data is just linked in two cluster instances, but hold only once in a single IPFS daemon.

So if you're interested in switching over from Syncthing or have this as an additional service, just let me know. :)

[dr460nf1r3]

Thanks for the offer, thats a kind offer. I'll keep it in mind for the future, maybe it'll come in handy one day! Cheers 🥳

[RubenKelevra]

You're welcome! :)