feat(vacuum): add vacuum management for cleaning expired packages #3442
Conversation
| // Optionally stores the package information in vacuum DB if vacuum controler is instantied and vacuum Enabled | ||
| if is.VacuumCtrl != nil && is.VacuumCtrl.IsVacuumEnabled(logE) { | ||
| logE.Debug("store package in vacuum") | ||
| if err := is.VacuumCtrl.Vacuum(ctx, logE, vacuum.AsyncStorePackage, []*config.Package{pkg}); err != nil { | ||
| logE.WithError(err).Error("store package in vacuum during install") | ||
| } | ||
| } |
There was a problem hiding this comment.
This is important in context of exec
Async operations are dropped if the target to exec is launched.
(Maybe i need to add this to Install controller too ?)
| @@ -770,6 +778,7 @@ func InitializeCopyCommandController(ctx context.Context, param *config.Param, h | |||
| installpackage.NewCargoPackageInstallerImpl, | |||
| wire.Bind(new(installpackage.CargoPackageInstaller), new(*installpackage.CargoPackageInstallerImpl)), | |||
| ), | |||
| provideNilVacuumController, // vacuum controller is not used so we provide nil but it is required by installer and installpackage | |||
There was a problem hiding this comment.
This permit to not enable vacuum in context of Copy command
| // handleStorePackage processes a list of configuration packages and stores the first package in the list. | ||
| func (vc *Controller) handleStorePackage(logE *logrus.Entry, vacuumPkg []*ConfigPackage) error { | ||
| if len(vacuumPkg) < 1 { | ||
| return errors.New("StorePackage requires at least one configPackage") | ||
| } | ||
| defer func() { | ||
| if err := vc.close(logE); err != nil { | ||
| logE.WithError(err).Error("Failed to close vacuum DB after storing package") | ||
| } | ||
| }() | ||
| return vc.storePackageInternal(logE, []*ConfigPackage{vacuumPkg[0]}) | ||
| } | ||
|
|
||
| // handleStorePackages processes a list of configuration packages and stores them. | ||
| func (vc *Controller) handleStorePackages(logE *logrus.Entry, vacuumPkg []*ConfigPackage) error { | ||
| if len(vacuumPkg) < 1 { | ||
| return errors.New("StorePackages requires at least one configPackage") | ||
| } | ||
| defer func() { | ||
| if err := vc.close(logE); err != nil { | ||
| logE.WithError(err).Error("Failed to close vacuum DB after storing multiple packages") | ||
| } | ||
| }() | ||
| return vc.storePackageInternal(logE, vacuumPkg) | ||
| } |
There was a problem hiding this comment.
Theses functions have been implemented to benchmark and evaluate which are better to use with aqua.
I think async is the best one but we need to use Close function in parent controller to ensure process are terminated like in Exec controller.
|
Thank you for your contribution! |
|
Thank you for your great work! According to the benchmark test, it looks like the overhead is really small. Command interfaceI don't think e.g. aqua vacuum show [--expired]
aqua vacuum runThe command DB schema
{
"LastUsageTime": "2025 ...",
"PkgPath": [""]
}The key is not good because we can change package name freely. I think I think we can add metadata such as type, name, etc to the value. Code review
When we return an error or output error or warn log, definitely any operation fails.
CheckWe should confirm if the treat of locale has no problem. func (vc *Controller) isPackageExpired(pkg *PackageVacuumEntry) bool {
const secondsInADay = 24 * 60 * 60
threshold := int64(*vc.Param.VacuumDays) * secondsInADay
return time.Since(pkg.PackageEntry.LastUsageTime).Seconds() > float64(threshold)
} |
| @@ -294,6 +294,7 @@ type Param struct { | |||
| Installed bool | |||
| PolicyConfigFilePaths []string | |||
| Commands []string | |||
| VacuumDays *int // When defined, vacuuming is enabled | |||
There was a problem hiding this comment.
| VacuumDays *int // When defined, vacuuming is enabled | |
| VacuumDays int // When defined, vacuuming is enabled |
I don't think this should be a pointer.
| @@ -26,13 +27,14 @@ type Controller struct { | |||
| fs afero.Fs | |||
| policyReader PolicyReader | |||
| enabledXSysExec bool | |||
| vacuumCtrl *vacuum.Controller | |||
There was a problem hiding this comment.
This should be an interface.
Then we can replace it with a mock when vacuum is disabled.
| // If the closing vacuum db failed, we should not stop the process | ||
| // so we log the error and continue the process. | ||
| // Updating vacuum db will be retried next time. | ||
| logE.WithError(err).Error("close the vacuum db failed") |
There was a problem hiding this comment.
| logE.WithError(err).Error("close the vacuum db failed") | |
| logerr.WithError(logE, err).Error("close the vacuum db failed") |
logerr.WithError extracts logrus.Fields from err properly.
|
I’ll revisit my draft and get back to you, taking your feedback into account! |
Check List
Require signed commits, so all commits must be signedIntroduce Vacuum Feature for Expired Packages
Addresses #2942 and Discussion #3086.
This pull request introduces a new feature for vacuuming expired packages.
Workflow
Environment Variable Requirement
The feature activates only when the environment variable
AQUA_VACUUM_DAYSis set with a positive integer greater than 0.Implementation in
InstallPackageexecorinstall(but notaqua updateoraqua cp, which currently useprovideNilVacuumControllerin their controller initialization), package-related information is added or updated in a BoltDB (vacuum.db) located inROOT_DIR.Expiration Criteria
AQUA_VACUUM_DAYS. Expired packages become eligible for "vacuum."Execution of
aqua vacuumCommandvacuumcommand performs the following operations:ROOT_DIR/pkgs/type.../pkg/version.Optional - Not Implemented: Automatic Vacuum Execution
aqua installcommand to trigger vacuum expiration automatically, enabling "auto-cleaning."Visualizing Managed Packages
The
aqua vacuumcommand includes two flags,-listand-expired:-listlists all packages.-expiredlists only expired packages.These flags enable the use of a fuzzy finder in the console to explore the
vacuum.dbdatabase and retrieve package-related information:Coverage
A high latence is observed for test du to tests of failling access to vacuum.db
Performance Considerations
Significant effort has been made to minimize the overhead introduced by this feature on Aqua commands. Benchmark tests in the
execcontroller attest to this.Three methods for adding packages to the database were implemented (storePackage, storesPackages, and asyncStorePackage). However, the asynchronous method (asyncStorePackage) shows the least overhead and is preferred. I think i'll remove all others.
Benchmark when adding 100 packages at the same time :
The asynchronous method is the best option in all cases, especially when adding one package at a time (the typical use case for Aqua).
Benchmark of exec command via hyperfine :
Main impact of storing packages is the method PackageInfo.PkgPaths, but i think the best choice to have all Paths concerned by each package. There are used to remove package from system during vacuum operation.