misc: updates and enhancments to host-side management tools

[efahl]

cleaner

• add a tool to wipe out unused public/store/ entries

update_all_targets

• extract configuration data from settings
• add a mechanism to optionally flush the redis database
• add a filtering mechanism so users can reduce footprint and bandwidth use on their local servers

More bits produced by my local server journey...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.55%. Comparing base (5e65dec) to head (d3fc006).
Report is 122 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1027      +/-   ##
==========================================
+ Coverage   80.75%   90.55%   +9.79%     
==========================================
  Files          15       14       -1     
  Lines         977     1080     +103     
==========================================
+ Hits          789      978     +189     
+ Misses        188      102      -86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[aparcar]

Hi this is nice but I think a bit over engineered.

Regarding removal I use a simple find with -delete command which cleans up builds older than a week. It does the job reliably and I don't think it needs anything more advanced.

For reloading, this only happens once and is then triggered via web hooks. I think a solution for smaller installations would be to add a reload job and respond 202 while the information is still missing. Once available it either continues with the building or returns an error. The build status while 202 could be something like "updating_metadata"

[efahl]

Hi this is nice but I think a bit over engineered.

Of course! 😀 I should have labelled this RFC, we can kill it.

Most of this was really just me learning about how everything fit together and what happens when you change things.

The update script in that form is very useful to me (probably only me) because of the way I'm testing things. I use owut quite extensively to test changes to ASU and vice versa, I use the local server to test changes in owut. So, that means I restart the server a lot, I'm guessing I did over 50 cold restarts of my local server today alone (podman stop, wipe public/*, podman build and restart, run update script...).

[efahl]

triggered via web hooks

Can you point me to where the webhooks are fired off, from the buildbots or somewhere? I've dug through https://github.com/openwrt/openwrt/tree/main/.github/workflows but can't find anything that would fire off the update request...

[Neustradamus]

@efahl: I think that you can add 24.10 part?

[efahl]

add 24.10

That's in #1066

[Neustradamus]

@efahl: Oh yes, good!

This part must be reviewed/merged before ^^