Filter between mirrors

[mswart]

Fixes #357

Description of the Change

Distributions contain a huge set of packages, mirroring them requires a large amount of storage and bandwidth. To mitigate this issues aptly provides an filter feature to reduce the set of mirrored packages. Especially it supports to limit the downloaded packages to a set of packages and their needed dependencies as well.

As this filter is self-contained within each mirror, it has two limitations:

1. its difficult to partial mirror a security or updates mirror to match a filtered main repository. Update and security repositories contain (newer) packages for a (small) subset of the whole distribution to fix bugs/security issues. As the repository contains not all packages, filtering breaks due to cuts within the dependency tree.

2. integration of packages from other sources. It is not possible to filter for all dependencies of a package if this package itself is not part of the mirror repository but instead published within another (local) repository.

In both cases all dependencies between repositories/mirrors have to be included manually in the mirror filter.

This commit introduces -deps-from-mirrors and -deps-from-repos options for mirrors. They are comma-separated lists referencing mirror/repository names.

If -filter-with-deps is activated all dependencies of packages from the references mirrors/repositories that are part of the current mirror are included in addition to the filter result itself. This resolves 2.

If in addition -dep-follow-all-variants is activated, all packages that are included in the referenced mirrors/repositories and present in the current mirror (matched by package name only) are selected to download, too. This resolves 1.

Limitations / Questions

• I am not confident about the option names, but I lack ideas for shorter, more to the point ones.
• The implemented approach differs from the proposals in -filter-with-deps across mirrors #357 - each mirror is updated on its own. If both mirrors have (many) packages with dependencies only in other mirrors, it might take some update rounds (update mirror 1, update mirror 2, update mirror 1 ...) to pick up all needed dependencies. I observed a similar behavior for experiments with xenial-mirrors and its security/update repositories. It took 2-3 updates to reach a stable filter point. I am currently unsure about the cause, I wound investigate further if the feature / fundamental implementation design is approached.
• I am unsure how to best implement functional tests.

Checklist

• unit-test added (if change is algorithm)
• functional test added/updated (if change is functional)
• man page updated (if applicable)
• bash completion updated (if applicable)
• documentation updated
• author name in AUTHORS

[smira]

Thanks, looks interesting!

[mswart]

Nice. I have rebased the PR to fix the merge conflicts.
I am happy to make further adjustments or addition to your feedback.

[belm0]

I tried the patch with my use case and it worked well-- looking forward to the feature.

It's been a while since the last rebase, would you run again?

[erikoest]

Is this patch available for aptly version 1.5.0? I tried to merge the mswart:filter-between-mirrors branch over aptly-dev:v1.5.0 and got a few conflicts, mainly in the api code.

[neolynx]

Hi !

It looks like this was never merge, just closed probably..

should we reopen this ?

[erikoest]

It's up to the aptly-developers, I guess. And I think a new merge req is needed since this one won't fit with newer versions of aptly. I'm using the filter-between-mirrors functionaliy actively, so I'll need it in later versions. But first, I can ask mswart if he would like to maintain his branch and make it compatible with v1.5.

[neolynx]

that would be great... let's have a look at how good it rebases...

[neolynx]

rebased...

[neolynx]

does not seem to build anymore on master

[neolynx]

@erikoest
I read about some issue here: #357

were they fixed, could they still be an issue ?

[erikoest]

I'm not familiar with issue #357. I don't think we've seen this problem in our systems. Maybe you should consult @mswart who wrote the patch.

[neolynx]

I managed to fix the build and the tests :)

coverage is low, so we need more tests: 38.52% of diff hit (target 74.84%)

@mswart I know it has been years, would you still be interested in taking this further ?