feat: overhaul workflows to publish packages to new archive infra #38

khos2ow · 2024-06-22T17:56:52Z

No description provided.

kgilmer · 2024-09-14T16:39:13Z

Hi @khos2ow , with 3.2 out the door the gates are open for the build refactor. Is this still WIP?

khos2ow · 2024-09-15T15:59:07Z

That's great! Yes this is still WIP. I'm gonna start working on it again.

.github/workflows/test-pkg-build.yml

.github/scripts/local-build.sh

kgilmer · 2024-12-15T00:55:53Z

.github/workflows/publish-packages.yml

+          set -x
+
+          ssh-keyscan -H ${{ secrets.KAMATERA_HOSTNAME2 }} >> ~/.ssh/known_hosts
+          ssh root@${{ secrets.KAMATERA_HOSTNAME2 }} "publish-repos.sh \"${{ inputs.packages-path }}\" \"${{ inputs.only-distro }}\" \"${{ inputs.only-codename }}\" \"${{ inputs.only-component }}\""


this makes sense, is obvious. I wonder if you considered running a gh runner on the repo host. Maybe a step in the wrong direction..

I hadn't! That could be interesting. But immediately what I'm thinking is that don't put more things and more responsibility into a server which basically is only designed to "serve an http server."

Making that a runner adds a bit more complexity to this and for us to maintain yet one more thing for it.

We can track it in #44 after getting this merged.

.github/scripts/rebuild-sources.sh

.github/scripts/ext-debian.sh

kgilmer · 2024-12-15T01:03:40Z

.github/scripts/ext-debian.sh

+}
+
+archive_setup_scripts() {
+  # Following allows for internal dependencies


question: can you elaborate on this? What would qualify as an internal dependency?

This part is equivalent of this which I moved from action file to here.

Not entirely sure if it is actually needed or not, but I think if a package that's being built relies on other packages that are maintained by Regolith (and basically only exists in archive.regolith-desktop.com and not Ubuntu or Debian public repos)

It's actually needed 😄 . Presently we have one such case: sway-regolith depends on trawldb. The current approach is to break a build into two phases, for stages that doesn't already have some version of trawldb in the repo (for example, a new OS-codename being added) which result in commits like this and then a revert. Would be really nice to not have to deal with that sort of thing.

Does this mean that, for say an example where 3 packages need to be built, that each build runs in isolation (so that the dependencies are available), or something else? To contrast, the existing build system is kind of "one shot" meaning that any failure fails the whole build, and any dependency to a package build needs to already be available before the build starts. So, it's not possible to say "first build trawldb, then build sway-regolith" because, while it's possible to have trawldb build first, it is not ingested into the package repo until the end of the package build.

That is a great point! I've also thinking about this and I think I might have a way forward to decentralize the process of building and publishing packages. If that works, then dependency management will ultimately boil down to "run the GitHub Action of the dependant package first and then run the Action of the other package."

We can track this further in #45 separately after getting this merged and get the ball rolling for unified archives in new server.

kgilmer · 2024-12-15T01:05:44Z

(still reading code. no concerns yet but need a bit more reading time)

kgilmer · 2024-12-20T01:39:41Z

.github/scripts/ext-debian.sh

-      allow_failing_bin_package="false"
+    short_version="${version%-*}"
+
+    mkdir -p $PKG_PUBLISH_PATH/$DISTRO/$CODENAME/$SUITE


question: can you explain what SUITE models? How is it different than stage?

SUITE is introduced to be able to support multiple Regolith version per release STAGE. Suite is, very loosely, the same as Stage with one big difference. Stage is release stage from the SDLC point of view, but Suite is from the point of view of the published debian package archives.

The corresponding values and mappings are as follow:

Stage Suite Component

experimental experimental main

unstable unstable main

testing testing main

release-current stable main

release-3_2 stable v3.2

release-3_1 stable v3.1

Or in other words, in the scope of deb config line:

deb [arch=amd64] http://archive.regolith-desktop.com/ubuntu/unstable noble main | | | | | | | | | * Component | | | * Distro Codename | | * Regolith Suite | * Distro Name * Architecture

LMK if this is not the case: SUITE is a debian repo concept. We would not necessarily use or need it for other packaging systems.

.github/workflows/build-arch.yml

kgilmer · 2024-12-20T01:42:28Z

.github/workflows/publish-packages.yml

+run-name: Publish Packages ${{ inputs.only-distro }} ${{ inputs.only-codename }} ${{ inputs.only-component }}
+
+on:
+  workflow_dispatch:


question: can you briefly describe at a high level how you envision this workflow to be called?

This is predominantly being triggered as a re-usable workflows inside build-deb-v4.yml actions. I specifically pulled it out to its own standalone action, that it can be triggered manually if there was ever a scenario where build-deb-v4.yml fails, for example, a package source has a conflict, or the build-deb-v4.yml was cancelled, or any other reasons (albeit very slim chance that there ever is a reason).

In that case we have the possibility to fix the issue (e.g. going on publish server and cleanup something, fix other thing, etc) and just triggering this workflow.

But in all reality, I highly doubt we'll ever run this independently. It's just a bit cleaner in the code and makes the, already too big, build-deb-v4.yml a little bit smaller and more readable.

kgilmer

Overall LGTM. Just a few questions to fill out my understanding.

kgilmer · 2024-12-20T01:51:02Z

.github/scripts/local-build.sh

@@ -7,36 +7,151 @@



I tried testing this script locally seeing as you awesomely updated the contrib docs, but got this error. I'm not sure what might be wrong but seems like the command arg parser is not reading --git-repo-path . for some reason..

~/dev/repos/voulage$ .github/scripts/local-build.sh --git-repo-path . --extension .github/scripts/ext-debian.sh --package-name regolith-i3status-rust --package-url https://github.com/regolith-linux/regolith-i3status-rust.git --package-ref debian_source --distro ubuntu --codename jammy --stage unstable mkdir: cannot create directory ‘/pkgbuild’: Permission denied

Nice catch! 6b83419 should fix it.

Worked perfectly

kgilmer

Amazing effort @khos2ow !

khos2ow force-pushed the migrate branch 11 times, most recently from 6f14094 to dd2d0fa Compare November 13, 2024 19:43

khos2ow force-pushed the migrate branch 4 times, most recently from 92b37dd to 272fc97 Compare November 22, 2024 17:57

khos2ow force-pushed the migrate branch 5 times, most recently from 819f1ca to a2ecf79 Compare November 26, 2024 14:59

khos2ow marked this pull request as ready for review November 26, 2024 14:59

khos2ow force-pushed the migrate branch 7 times, most recently from 4e03790 to 0ea7bbc Compare November 27, 2024 20:38

khos2ow force-pushed the migrate branch 2 times, most recently from 9395942 to 1749472 Compare December 14, 2024 22:34

khos2ow requested a review from kgilmer December 14, 2024 22:35

khos2ow changed the title ~~feat: add migration guide to new package archives~~ feat: overhaul workflows to publish packages to new archive infra Dec 14, 2024

khos2ow force-pushed the migrate branch from 1749472 to ffd0b39 Compare December 14, 2024 22:44

kgilmer reviewed Dec 15, 2024

View reviewed changes

khos2ow force-pushed the migrate branch 2 times, most recently from f3b3b24 to 8329568 Compare December 17, 2024 04:03

khos2ow linked an issue Dec 17, 2024 that may be closed by this pull request

Unifying archive package structure #34

Closed

khos2ow mentioned this pull request Dec 17, 2024

Setup dedicated runner on new archive server #44

Open

khos2ow force-pushed the migrate branch from 51a5ed7 to 3eeb630 Compare December 17, 2024 17:04

khos2ow requested a review from kgilmer December 17, 2024 17:04

khos2ow added 6 commits December 17, 2024 17:31

feat: enhance test installable action to use new archives

cb8d6da

feat: enhance local build and test package build action

b13314b

feat: add new action to publish packages

22a22b4

feat: add new action to rebuild package sources

60be3e0

feat: refactor build scripts to publish to new server

80b0d0f

chore: use the new consolidated actions repo

ad4a55a

khos2ow force-pushed the migrate branch from 3eeb630 to ad4a55a Compare December 17, 2024 22:32

khos2ow added 2 commits December 17, 2024 18:44

fix: show the content of package folder after publish

7ac2ce4

fix: make sure Release file exists and not empty

691746c

kgilmer reviewed Dec 20, 2024

View reviewed changes

.github/workflows/build-arch.yml Show resolved Hide resolved

kgilmer reviewed Dec 20, 2024

View reviewed changes

fix: package build and publish paths in local-build script

6b83419

kgilmer approved these changes Dec 23, 2024

View reviewed changes

khos2ow merged commit 422dda2 into main Dec 24, 2024

khos2ow deleted the migrate branch December 24, 2024 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: overhaul workflows to publish packages to new archive infra #38

feat: overhaul workflows to publish packages to new archive infra #38

khos2ow commented Jun 22, 2024

kgilmer commented Sep 14, 2024

khos2ow commented Sep 15, 2024

kgilmer Dec 15, 2024

khos2ow Dec 16, 2024

khos2ow Dec 17, 2024 •

edited

Loading

kgilmer Dec 15, 2024

khos2ow Dec 16, 2024

kgilmer Dec 17, 2024 •

edited

Loading

khos2ow Dec 17, 2024

kgilmer commented Dec 15, 2024

kgilmer Dec 20, 2024

khos2ow Dec 20, 2024

kgilmer Dec 23, 2024

kgilmer Dec 20, 2024 •

edited

Loading

khos2ow Dec 20, 2024

kgilmer left a comment

kgilmer Dec 20, 2024

khos2ow Dec 20, 2024

kgilmer Dec 23, 2024

kgilmer left a comment

Stage	Suite	Component
`experimental`	`experimental`	`main`
`unstable`	`unstable`	`main`
`testing`	`testing`	`main`
`release-current`	`stable`	`main`
`release-3_2`	`stable`	`v3.2`
`release-3_1`	`stable`	`v3.1`

feat: overhaul workflows to publish packages to new archive infra #38

feat: overhaul workflows to publish packages to new archive infra #38

Conversation

khos2ow commented Jun 22, 2024

kgilmer commented Sep 14, 2024

khos2ow commented Sep 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khos2ow Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgilmer Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgilmer commented Dec 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgilmer Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgilmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgilmer left a comment

Choose a reason for hiding this comment

khos2ow Dec 17, 2024 •

edited

Loading

kgilmer Dec 17, 2024 •

edited

Loading

kgilmer Dec 20, 2024 •

edited

Loading