bulk: allow prefetching binary packages from official repos

[bapt]

As soon as we have the entire list of packages to build but before
the sanity check, pre fetch all the binary packages that we can
from the official repositories.

The sanity check running after than will probably destroy some
packages for which the confiration will be different or any reason
suitable for the sanity check to destroy packages.

When listing the packages to fetch only list the one not already
existing to avoid pkg fetch to redownload and overwrite packages
we have already customized

[brooksdavis]

This looks super helpful for incremental builds on cloud-based poudriere servers.

[brooksdavis]

It would be nice if there was a way to specify a full URL rather than hardcoding pkg.freebsd.org. Supporting latest|quarterly|<URL> seems viable.

[bdrewery]

-b doesn't make me think fetch at all. In bulk.sh it is -b branch which makes me think 'branch' rather than 'binary'; I'm saying that at a quick glance of the manpage I might think this is saying which branch to build from rather than anything to do with fetching. I don't really have a better idea.
We're getting to the point of needing to use things like --fetch. I had been thinking -s[eed] but -f -F -S are taken already and I had a plan to flip -S into -s IIRC.

Like Brooks comment I think this inevitably ends in someone asking for REPO_URL to be configurable so this flag interface may need to forward think about that.

[bdrewery]

1. Need this to not run for non-build commands (like pkgclean) and to not run when resuming, and to still respect -c and -C. So I think it makes sense higher up (Github review won't let me comment on untouched lines.) :

if [ -n "${PACKAGE_BRANCH}" -a "${resuming_build}" -ne 1 ]; then
		download_from_repo
fi
# existing code
if [ ${JAIL_NEEDS_CLEAN} -eq 1 ]; then
			msg_n "Cleaning all packages due to newer version of the jail..."
			rm -rf ${PACKAGES}/* ${cache_dir}

2. Then there is the question of what should -n (dry run) do? Should it fetch so that a proper accounting of what will build is done or should it just say "will try fetching pkgA pkgB ..."? I have no opinion here.

[bdrewery]

I think this is fine for now but ultimately we may want to inspect more what will be downloaded to avoid delete_old_pkg from nuking the same packages every time for things like different options or flavors.

[bdrewery]

Remaining items I am fixing before merge:

• pkg bootstrapping; pkg is fetched but then the build fails because symlink packages/Latest/pkg.txz is missing.
• bulk -n (dry mode) still is fetching
• bulk -C needs to only fetch dependencies and not the packages that are listed, or a second pass to delete listed ports
• bulk -c: It is unclear what this should do. Should it just mirror the remote or still build listed ports?
• testport support
• poudriere.conf always-on support (documented)

Items for after:

• Incremental build changes #822 is likely to be a problem here too due to out-of-sync remote and local tree. I've seen llvm* building when it probably could be skipped.
• Need to compare options before fetching or every build will likely fetch and delete which wastes time and bandwidth. 1 differing option could cause thousands of unneeded packages to be fetched.
• ignored ports are fetched
• we fetch packages that have different dependencies. Such as using ports openssl locally and remote does not, we still fetch the packages and then delete them.

[bdrewery]

A major pitfall is that local patches, overlays, EXTRA_PATCH_TREE, etc, are not considered. If it finds a remote package with the same options it will use that and never build with the custom patches. IMHO this is enough to remove the feature entirely as it is prone to cause security and POLA issues for users.

I could auto detect EXTRA_PATCH_TREE dirs and overlays dirs, but not directly dropped files. A blacklist could be used too, but there is no concise way to tell a user the feature is dangerous with custom patches, and implementing all of these detections is more work.