Makefile.inc1: allow real-update-packages to be called independently #1445

svmhdvn · 2024-10-02T01:51:21Z

To perform an incremental update of a pkgbase repo, you would call 'make update-packages', which will stage, create, and incrementally choose newer package versions to sign as part of a pkg repo. However, this forces you to stage the kernel and source packages along with the world packages. For a jail-only installation of FreeBSD, these packages are generally not required.

This patch separates the 'update-packages' target from the 'real-update-packages' target to allow choosing world, kernel, and/or source individually at the command line like so:

Jail-only installation:

make [...] -j$(nproc) buildworld
make [...] PKG_VERSION='15.snap<date>' \
    stage-packages-world create-packages-world \
    real-update-packages

Other minor changes are included to fix the PKG_VERSION variable definition logic and order of operations.

svmhdvn · 2024-10-02T01:52:50Z

Makefile.inc1

@@ -571,7 +571,6 @@ VERSION=	FreeBSD ${_REVISION}-${_BRANCH:C/-p[0-9]+$//} ${TARGET_ARCH} ${SRCRELDA
 .endif
 MAJOR_REVISION=	${_REVISION:R}

-.if !defined(PKG_VERSION)


The below variables should always be defined regardless of what PKG_VERSION is set to, since it is used to compare two repos for incremental updates.

svmhdvn · 2024-10-02T01:53:20Z

Makefile.inc1

@@ -593,9 +592,8 @@ EXTRA_REVISION=	.${BRANCH_EXT}.${_TIMENOW}
 BRANCH_EXT=	p${_BRANCH:C/.*-p([0-9]+$)/\1/}
 EXTRA_REVISION=	${BRANCH_EXT}
 .endif
-PKG_VERSION:=	${_PKG_REVISION}${EXTRA_REVISION:C/[[:space:]]//g}
-.endif
-.endif	# !defined(PKG_VERSION)


This line was wrongly commented as # !defined(PKG_VERSION).

svmhdvn · 2024-10-02T01:53:40Z

Makefile.inc1

 PKG_VERSION_FROM!=/usr/bin/readlink ${REPODIR}/${PKG_ABI}/latest
 PKG_VERSION_FROM_DIR=	${REPODIR}/${PKG_ABI}/${PKG_VERSION_FROM}
 BRANCH_EXT_FROM=	${PKG_VERSION_FROM:C/^[^[:alpha:]]+p?([[:alpha:]]*)[0-9]+$/\1/}
-.else


These should be left undefined instead of empty for better logic below.

svmhdvn · 2024-10-02T01:55:44Z

Makefile.inc1

@@ -2054,8 +2047,8 @@ PKGMAKEARGS+=	PKG_VERSION=${PKG_VERSION} \
 packages: .PHONY
 	${_+_}${MAKE} -C ${.CURDIR} ${PKGMAKEARGS} real-packages

-update-packages: .PHONY
-	${_+_}${MAKE} -C ${.CURDIR} ${PKGMAKEARGS} real-update-packages
+update-packages: stage-packages .PHONY


The update-packages target continues to have the same behaviour as before. The only difference is that the real-update-packages target does not force a stage and creation of world, kernel, AND source package lists.

NOTE: stage-packages is listed in the prerequisites because it does not depend on the value of the PKG_VERSION variable, whereas create-packages does.

svmhdvn · 2024-10-02T01:56:44Z

Makefile.inc1

-	@echo "==> Bootstrapping repository, not checking for new packages"
-.else
+real-update-packages: .PHONY
+.if defined(PKG_VERSION_FROM_DIR)


The logic is now inverted for better flow.

Does the inversion depend on your other changes or can it be merged standalone?

svmhdvn · 2024-10-02T01:57:16Z

Makefile.inc1

 	${PKG_CMD} -o ABI=${PKG_ABI} repo -o OSVERSION="${SRCRELDATE}" \
 		-m ${WSTAGEDIR}/meta \
 		-o ${REPODIR}/${PKG_ABI}/${PKG_VERSION} \
 		${REPODIR}/${PKG_ABI}/${PKG_VERSION} \
 		${PKG_REPO_SIGNING_KEY} ; \
 	cd ${REPODIR}/${PKG_ABI}; \
-	ln -s ${PKG_OUTPUT_DIR} latest
+	ln -sf ${PKG_OUTPUT_DIR} latest


This avoids deleting the latest symlink when the PKG_CMD invocation fails.

bsdimp · 2024-10-04T18:52:17Z

Can you look at this @evadot @bapt @emaste ?

bsdjhb · 2024-10-04T18:54:06Z

Makefile.inc1

-	@echo "==> Bootstrapping repository, not checking for new packages"
-.else
+real-update-packages: .PHONY
+.if defined(PKG_VERSION_FROM_DIR)


Does the inversion depend on your other changes or can it be merged standalone?

bsdjhb · 2024-10-04T18:54:33Z

Makefile.inc1

 .endif
-	${_+_}@cd ${.CURDIR}; \


Using make -C here I think could be its own commit.

I'm happy to iteratively land the trivial changes out of the set, as well.

Also, I had this question on the freebsd-dev IRC channel at some point, but I don't remember getting a definitive answer on it. Why is the ${_+_} prefix scattered around everywhere? As I understand, the manpage states that the + prefix can be used to force a command to always run, even when -n is used at the make command line. So why does it need to be expanded as a variable surrounded by _ characters?

I was curious. It's a local oddity to make the -N option not recurse. See 56585ab and it's predecessor acabf29

bsdjhb · 2024-10-04T18:55:06Z

Makefile.inc1

@@ -2349,14 +2340,13 @@ real-sign-packages:	_pkgbootstrap .PHONY
 .if ${PKG_BIN_VERSION} < 11700
 	printf "packing_format = \"${PKG_FORMAT}\";\n" >> ${WSTAGEDIR}/meta
 .endif
-	@[ -L "${REPODIR}/${PKG_ABI}/latest" ] && unlink ${REPODIR}/${PKG_ABI}/latest; \


Likewise, this hunk to use a forced ln -s to remove the previous link can be its own commit I think.

svmhdvn · 2024-10-04T22:51:00Z

I broke the commit into three separate commits:

bugfix for the latest symlink
non-functional style changes
main change to make real-update-packages independent

Is this easier to review/merge incrementally now?

svmhdvn · 2024-10-10T15:48:10Z

Updated the PR on top of 458dc7f

…xists

To perform an incremental update of a pkgbase repo, you would call 'make update-packages', which will stage, create, and incrementally choose newer package versions to sign as part of a pkg repo. However, this forces you to stage the kernel and source packages along with the world packages. For a jail-only installation of FreeBSD, these packages are generally not required. This patch separates the 'update-packages' target from the 'real-update-packages' target to allow choosing world, kernel, and/or source individually at the command line like so: Jail-only installation: 1. make -j`nproc` buildworld 2. make PKG_VERSION='15.snap<date>' \ stage-packages-world create-packages-world \ real-update-packages

svmhdvn · 2024-10-31T16:29:29Z

Rebased the PR on top of 8549e3c, which brought in some of the changes from the first commit in my patch series. Any updates on reviewing this PR?

svmhdvn commented Oct 2, 2024

View reviewed changes

svmhdvn force-pushed the update-packages branch from c9445e5 to fdfe444 Compare October 2, 2024 22:14

bsdimp added the changes-required label Oct 4, 2024

bsdjhb reviewed Oct 4, 2024

View reviewed changes

svmhdvn force-pushed the update-packages branch from fdfe444 to 7d4b28c Compare October 4, 2024 22:49

svmhdvn force-pushed the update-packages branch from 7d4b28c to 77d7ee7 Compare October 10, 2024 15:47

svmhdvn added 3 commits October 31, 2024 12:27

Makefile.inc1: define PKG_VERSION_FROM only if $REPO/latest symlink e…

e20564e

…xists

Makefile.inc1: minor style changes

d6bf1ab

svmhdvn force-pushed the update-packages branch from 77d7ee7 to 02d0f0f Compare October 31, 2024 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Makefile.inc1: allow real-update-packages to be called independently #1445

Makefile.inc1: allow real-update-packages to be called independently #1445

svmhdvn commented Oct 2, 2024

svmhdvn Oct 2, 2024

svmhdvn Oct 2, 2024

svmhdvn Oct 2, 2024

svmhdvn Oct 2, 2024

svmhdvn Oct 2, 2024

bsdjhb Oct 4, 2024

svmhdvn Oct 2, 2024

bsdimp commented Oct 4, 2024

bsdjhb Oct 4, 2024

bsdjhb Oct 4, 2024

emaste Oct 4, 2024

svmhdvn Oct 4, 2024

brooksdavis Oct 4, 2024

bsdjhb Oct 4, 2024

svmhdvn commented Oct 4, 2024

svmhdvn commented Oct 10, 2024

svmhdvn commented Oct 31, 2024

Makefile.inc1: allow real-update-packages to be called independently #1445

Are you sure you want to change the base?

Makefile.inc1: allow real-update-packages to be called independently #1445

Conversation

svmhdvn commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsdimp commented Oct 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svmhdvn commented Oct 4, 2024

svmhdvn commented Oct 10, 2024

svmhdvn commented Oct 31, 2024