Termux looks for a hosting service

[ghost]

Updates on this issue

• [2021.05.22] We have requested a VPS from FossHost. Request has been approved but it may take up to month for getting the server.
• [2021.06.01] Thanks to @Hero9909 for the providing container on his server. Repository setup is in progress. This issue will be closed once everything is ready.
• [2021.06.02] https://packages.termux.org is now up!
• [2021.06.05] Thanks to FossHost for the server. https://packages.termux.org will be migrated to the new offered machine.

---

Termux is looking for a new package hosting.

Previously we have used Bintray as our primary package hosting, providing support for Debian packages and REST API for repository management as well as donation of 60 TB/month of bandwidth for our project (thanks to JFrog). Though service is now finished and we have to find another one.

Write here if you wish to help us with hosting Termux repositories or have good suggestion of service which could fit our needs. See below on acceptable suggestions.

What we are looking for

Termux needs a primary/seed package host. It is importantly to note that this is not a mirror. Maintainers expected to be granted full access for uploading packages on individual basis as well as possibility of configuring automatic package upload from CI. Depending on the host type, e.g. if VPS, we may need to install a custom software for building & signing the repository. Package repository will be exposed through https://termux.org/ and serve large amounts of data.

Here are suitable variants:

1. VPS, which is not shared with third-party projects.
   Best solution since such server expected to be administrated by @termux team, without any third-party sudoers.
2. SaaS platform for package hosting.
   Do not suggest https://cloudsmith.com/ or https://packagecloud.io/ - they were tried and considered as unsuitable: significant issue is support only of predefined Debian/Ubuntu distributions and thus impossibility of defining a custom apt distribution/component and another issue is inflexibility of Deb package processing (will reject x86_64 packages as it "expects" them to be amd64 instead). Bintray didn't have these problems.
3. S3-like object storage.
   Worst, but acceptable solution as in such case we will have to build repo on CI side which will imply some potential issues.

Regardless of host type, it should have these properties:

• Storage: 25 GB (at least)
• Bandwith: 60 TB/month or unlimited
• Content management over SSH/SFTP or REST API.

It also would be perfectly if service is free, but of course we know that suitable free services are rare thing ;) Services which are paid or require identity verification should be discussed with @fornwall. I'm not going to manage billing nor provide documents on my own.

(!!!) Please do not suggest public shell servers and other shady services which have problems with sharing their (very limited) hardware resources and troublesome to trust. We need to be sure that service will be stable in long term and secure enough to give it our GPG key for signing repository.

What is being used currently

We have set of mirrors currently used to deliver packages. The primary ones are https://grimler.se and IPFS network nodes.

Note that there is already a pending candidate for Termux primary hosting - https://dl.kcubeterm.me. It is managed by @kcubeterm, yet works currently only as mirror.

[theaungmyatmoe]

VPS or IPFS

[ghost]

VPS or IPFS

Regarding VPS: I'm aware about existing providers, i.e. Hetzner, Digital Ocean, Vultr and others, so here question is not which provider to choose. However I'm not going to provide a one on my own (otherwise why my mirrors are on IPFS?).

About IPFS: it is not mentioned in section "What we are looking for" as potential variant and thus not suitable. Moreover, it is used now to provide mirrors. But it is not a permanent solution. Mirror is not a primary host and provide read-only access to clients.

The primary node, host where IPNS (IPFS Name System) keys are stored, is hosted on my PC meaning that repository is up as soon as I still available online. IPNS records expire in 12 hours and should be re-broadcasted at least once within this period.

[eljoest]

Potential sponsorship options:

• Digital Ocean
• fastly (CDN)
• scaleway (credits for their services)
• hetzner and leaseweb appear to have sponsorships as well, but I haven't found dedicated contact information.

[finagolfin]

Why do you want to serve everything from one package host, ie all 60 TBs/month from one server? Fredrik can easily rent an adequate VPS for $10-20/month, which let's you run any software and handle 2-4 TBs/month, and various mirrors like FOSShost could serve the remaining 56-58 TBs/month. The only problem with mirrors is keeping them up to date, but we could have a script monitor them and make sure they're getting updated, and derank them in the pkg installer if they're slow to update.

[kcubeterm]

@buttaface I applied for VPS to fosshost and they approved our project. We will get a dedicated vps soon. They are aware of our requirements since I points this issue on fosshost IRC. RAM size is yet to be discussed but we have numbers of choice in terms of software like nexus, pulp3 even aptly etc. By the way. Could you please share your ssh pubkey.

[ghost]

RAM size is yet to be discussed but we have numbers of choice in terms of software like nexus, pulp3 even aptly etc.

Here is resource stats on processing packages from main, unstable and science repositories - these 3 have the largest packages.

I guess 4 GB RAM will be ok, considering the peak usage was 965156 kbytes.

Command being timed: "aptly publish repo termux-test termux-test"
User time (seconds): 849.26
System time (seconds): 109.75
Percent of CPU this job got: 109%
Elapsed (wall clock) time (h:mm:ss or m:ss): 14:33.64
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 965156
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3563988
Voluntary context switches: 16187830
Involuntary context switches: 5297442
Swaps: 0
File system inputs: 1729848
File system outputs: 2024832
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

---

Why do you want to serve everything from one package host, ie all 60 TBs/month from one server?

Only one host can be primary. For the rest, mirrors are already available.

[SommerEngineering]

Hetzner seems like a good partner to me. I can imagine that they might act as a sponsor. But also their commercial offer seems to fit pretty well: An AX41 NVME server costs about 40 euros (about $49) per month and offers 2x 512 GB NVMe SSD, 64 GB RAM, 100 GB backup storage, 1 GBit/s connection and unlimited traffic. This is then a barebone machine, real hardware, no virtualization, etc.

Link: https://www.hetzner.com/dedicated-rootserver/ax41-nvme?country=en

[librehat]

May I suggest OVH as a potential provider? They have many products including VPS, dedicated server etc. The plans of VPS for example don't limit the traffic (even for the cheapest plan, though the speed is throttled).
I also found they do have quite many sponsorships and partnerships with well-known open-source projects: https://www.ovh.co.uk/news/press/

[finagolfin]

Note that on the "unmetered" connections, which I have experience pushing TBs of traffic through for months at a time, you will not see anything like peak capacity. If somebody pushed 1 GBit/s through that Hetzner server, you would theoretically get 1 Gbps * 3600 s/hour * 720 hours/month / 8 bits/byte * 1000 GB /TB = 324 TB/month. But that's not going to be a dedicated network uplink, so your real throughput could be 1-10% of that, or 3 to 30 TBs/month, depending on how many other servers share that 1 Gbps uplink in the data center.

[Hero9909]

Im in a Team of 3, we rented a server some time ago. currently we are not using all our resources and it looks like we wont use them all in the future. So ive talked to one member and awaiting currently the response of the other. If all goes well, i would share a lxc container with 4 c , 16G of ram and 100G of nvme storage, nested support is available in case of docker. our traffic is unlimited, guaranteed 1G link and currently only around 3-4% in use. keep in mind that as @buttaface said we dont know the structure of the datacentre maybe we share our traffic via overallocation. ill give an update if everything is planned and ready or canceled.

[Hero9909]

Okay, ive got the okay. Next week we are done with our halfe-year-update. After that we could start.
Ill wait until someone of termux-team gets in touch with me as i need an accountable for that.

[ghost]

@Hero9909 That's all nice, however there are questions:

• How long-term this solution could be?
  We expect host to be available at least for 2 years, preferably more.
• Is it like a shared server and our repositories are expected to be packaged as Docker or LXC container?
  Shared server is definitely a cons, which makes hard to delegate repository signing procedure on it. Will have to do that on external CI.
• Would it be accessible over SSH?

For now a VPS from FossHost looks like optimal solution.

[ghost]

@kcubeterm Are there any answers from FossHost?

[Hero9909]

@xeffyr

1. we are planning to keep this as long as we have an accountable for that
2. isnt direct clear to me, correctme if i got this wrong.
   we have full control over the server and offering a lxc container (specs see comment before) with the distro of your choice.
   From there any administration is done via ssh just like a vps.
3. yep

if you like we could talk via discord or ts3 to get a faster q&a talk

[kcubeterm]

Are there any answers from FossHost?

Build is in queue. It may take upto a month. They are really working hard to clear their backlog. I hope they managed to get as far as possible.

[ghost]

we have full control over the server and offering a lxc container (specs see comment before) with the distro of your choice.
From there any administration is done via ssh just like a vps.

Thanks, will consider this as potential solution.

Build is in queue. It may take upto a month.

Ok, continuing look for other suggestions, in case FossHost will unable to provide VPS for some reason.

[librehat]

Just saw Oracle Cloud "Always Free" tier, would that be an option: https://www.oracle.com/uk/cloud/free/

[ghost]

@librehat https://free-for.dev/ - please don't re-provide solutions shown on this page.

Free tier is not same as free service. As stated in #6846 (comment):

Services which are paid or require identity verification should be discussed with @fornwall.

Also Oracle Cloud has a quite problematic registration, I never was able to complete it (have tried 20+ times during several months) for various issues (one of: no email sent, inactive buttons when credit card verification passed, sometimes credit card verification fails, etc).

An example of issue during registration (click to expand)

Registration page shows that credit card verification has been successfully completed. However I can't proceed because confirmation checkbox is locked and button for beginning free tier is inactive.

[librehat]

I am able to register and provision some running VM instances FYI.

But you're right, it requires credit card verification so it needs @fornwall

[ghost]

I am able to register and provision some running VM instances FYI.

So what about egress speed? Many Oracle Cloud free tier reviews consider it as "low" and https://free-for.dev/ specifies it as 5mbps. Is that true?

[librehat]

So what about egress speed? Many Oracle Cloud free tier reviews consider it as "low" and https://free-for.dev/ specifies it as 5mbps. Is that true?

Not 5Mbps. The AMD instances are limited to 0.48Gbps. The Ampere ARM instances are 1 ~ 4 Gbps depending on the configuration. Ampere instances are new to the Always Free Tier (since one or two days ago only) so maybe that's not updated on free-for.dev yet (I don't know where they got the 5Mbps but on my instance page, it's 0.48Gbps for AMD, and 1 Gbps for the ARM one)

Edit: the AMD instance has 50Mbps public Internet bandwidth. However, the ARM instance should have > 1 Gbps bandwidth

[vhqtvn]

I use github release to host packages, and coded a simple repo which redirect deb download to github assets.

https://github.com/vhqtvn/termux-packages/releases/tag/repos-02498ab7d3ae890ea667f06e50cbda0e46c98751

This will work as long as we dont have any package with size larger than 100M. I think this limit can be bypassed by splitting.
Update: https://docs.github.com/en/github/administering-a-repository/releasing-projects-on-github/about-releases it's 2G per file so I think it's more than enough.

[ghost]

On #6846 (comment) I have provided acceptable solutions. This even not the third one which is

3. Some kind of file hosting, preferably something similar to Amazon S3.

If you know what is Amazon S3, you will understand what kind of file hosting is expected. But VPS is still preferred.

So:

• How redirects are configured? Do you have any external server for metadata? - We don't have any server now, which is why this issue is opened.
• Could the packages be easily submitted from CI? As I understand, your solution is not suitable for aptly or other repository management software.
• If using CI, how concurrent uploads could be handled?

termux-packages is managed by multiple people which should be able to submit any amount of packages at any time and without interaction from my side.

Hosting solution must be viable for setting up of F-Droid repository which would be done later as part of finishing https://github.com/termux/termux-app/tree/android-10.

[Hero9909]

@xeffyr @kcubeterm fixed

[ghost]

@Hero9909 I can confirm that now can access it.

Is web server (nginx) is entirely managed by you or you can proxy it inside the container so we will be able to use own config?

[Grimler91]

@Hero9909 could you add a user for

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXFvKMueZ481gw2nXj7QBME8O6LexKh+rFjzP8B7iVeuxbLgbQiWs/6WBMUfdV0a784KOaEJcNz7imarNYbtVkthTAl+2I3DVYMEx7WqRdgEzoUsgK7OmCK9odyeQfkQ2ZT6dnPM8lxGSHw/jhBan1oDPxnjAi70zmFuxI3mz4MSlkuAeyBgV1mHRLoVbaIcu1dKbhDk+eWod2kLMLRy/Pkc8belcMS+o0Rn3X1ZpcaZ9dCNWl8e6zGmlLLcia+xa8ePUOUzy43oXYUjhKkmOXU5onmSqgoAS6l6rQHn6J3lWeMb29ifD0SwbdZ3aCfrPeZbebv7WD2olzBJ8WMT+/6j053itqMTm7SH33RlKOW01m7La+hM1ZZ2+fWoir+BIXpzpLbqGP/TddsN9/iUM6lr6KSI/lCzS28JjBkBeq7onG+I9cSlPQ4dAPnY9Eg/pPgZ+X76dHqhXwvIKWaT5rOkA8clvir6bR4bYCrjGKmDsb9kou5j4P/zQDd8hdUw8= grimler

as well? Thanks!

[Hero9909]

ive already talked to another member about this, we are be abled to setup a seperate server/proxy for the termux.org domain, but well need the certificates, another way is to use the direct mapped ipv6, but this wont work for all users as ipv6 isnt as common as it should be.

[Hero9909]

@Grimler91 Done, please contact me after reading the readme file in your home directory.
Host: tanisorhost.cosmos-ink.net
Port: 2207
User: grimler91

[ghost]

We don't have certificates, but can use Let's Encrypt for obtaining them. This will require automated renew since they expire in 90 days. Other variant will be use of CloudFlare which will also act as proxy (and CDN) and let IPv4 users access the server.

It is expected that container will run these 2 services which should be accessible outside:

• Apt repository (assume just file server)
• Aptly API server which will require reverse proxy and authentication.

So that's why I'm asking whether we will be able to manage the config on our own.

[Hero9909]

im working on a solution, for security reasons i would like to migrate futher detail discussions to a non public place, is telegram okay for you?

[ghost]

is telegram okay for you?

Okay

[Grimler91]

@Hero9909 I am getting permission denied, could you check if the pub key was added correctly?

(Fingerprint of key I posted here matches the private key, so should be correct)

[Hero9909]

try again.

[ghost]

https://packages.termux.org/ is now set up and works fine, so I'm closing this issue.

---

@kcubeterm, @Grimler91, @a1batross Suggesting to reconfigure your mirrors for cloning https://packages.termux.org/.

Main repository:

deb https://packages.termux.org/apt/termux-main/ stable main

Games repository:

deb https://packages.termux.org/apt/termux-games/ games stable

Root repository:

deb https://packages.termux.org/apt/termux-root/ root stable

Science repository:

deb https://packages.termux.org/apt/termux-science/ science stable

Unstable repository:

deb https://packages.termux.org/apt/termux-unstable/ unstable main

X11 repository:

deb https://packages.termux.org/apt/termux-x11/ x11 main

As you can see, paths there different. Though you can try to map them to

termux-packages-24
game-packages-24
termux-root-packages-24
science-packages-24
unstable-packages
x11-packages

to maintain compatibility. If you decide a fresh configuration with new paths, notify me and I will update termux-tools package and information at https://github.com/termux/termux-packages/wiki/Mirrors.

[agnostic-apollo]

Thanks a lot @xeffyr for setting this up and @Hero9909 for providing the container.

[a1batross]

@xeffyr Thank you! Changed paths and added symlinks to maintain compatibility on my side.

[truboxl]

@xeffyr can you update https://github.com/termux/termux-docker as well? Thanks!

[iBug]

Do you provide Rsync service for downstream mirror sites?

[ghost]

No, we don't provide rsync currently.

[iBug]

How long will the IPFS links stay around? Might need some time before I can update our mirror config.

[kcubeterm]

Until any single nodes have that content. If all nodes goes down and neither gateway has cached version. Links won't work. Right now only @xeffyr node is serving, I stopped all of my nodes.

[selurvedu]

I am very glad that my suggestion of Fosshost worked. Thank you all for your efforts!

[finagolfin]

@xeffyr, now that we have a dedicated package server again, can we turn the automated update checker back on on the github CI? I thought that really helped us keep github projects up to date.

[landfillbaby]

maybe we should make a list of packages that aren't automatically updated too so we have an explicit list of what to check manually

[finagolfin]

There is already a list of mostly the inverse, but it's not complete.

[iBug]

Hi @xeffyr, will you update the repository wiki?

I've updated mirrors.ustc.edu.cn to sync from packages.termux.org, with the new directory layout and backwards compatibility with the old ones from Bintray era (just some more symbolic links).

Our current URLs are as follows. Use the edit button to retrieve Markdown source from this comment.

---

Mirrors by University of Science and Technology of China, Linux User Group

Mirror for Chinese users for better ping and download speed.

Repository	sources.list entry
Main	deb https://mirrors.ustc.edu.cn/termux/apt/termux-main/ stable main
Games	deb https://mirrors.ustc.edu.cn/termux/apt/termux-games/ games stable
Root	deb https://mirrors.ustc.edu.cn/termux/apt/termux-root/ root stable
Science	deb https://mirrors.ustc.edu.cn/termux/apt/termux-science/ science stable
Unstable	deb https://mirrors.ustc.edu.cn/termux/apt/termux-unstable/ unstable main
X11	deb https://mirrors.ustc.edu.cn/termux/apt/termux-x11/ x11 main

[ghost]

@iBug Thanks, repository wiki is updated now.

[NatoBoram]

I was rehosting the termux packages on IPFS (ipfs/kubo#4435 (comment)), but it looks like the IPNS stopped resolving. I'm glad that Termux found a suitable hosting service!