updating feeds doesn't update to latest

[brianjmurrell]

I'm trying to build a package using your packaging building script (which is awesome BTW). I notice my luci feed is really quite old though, compared to the current tip of the openwrt-19.07 branch.

As you can see here scripts/feeds update -f -a only updates the luci feed to openwrt/luci@adbbd5c:

+ .../sdk-19.07.1-ar71xx-nand/scripts/feeds update -f -a
Updating feed 'base' from 'https://git.openwrt.org/openwrt/openwrt.git;v19.07.1' ...
Already up to date.
Create index file './feeds/base.index' 
Updating feed 'packages' from 'https://git.openwrt.org/feed/packages.git^1ec76dd7ca647f76a75c4f88b3b70d154221e7e9' ...
Create index file './feeds/packages.index' 
Updating feed 'luci' from 'https://git.openwrt.org/project/luci.git^adbbd5cf690cf993e0de1e56266ac6fd2b2dfe91' ...
Create index file './feeds/luci.index' 
...

Yet, the openwrt-19.07 branch of luci is currently at openwrt/luci@a24f198 which is quite a ways ahead of where I am being updated to: openwrt/luci@adbbd5c...openwrt-19.07

The result is that the package I end up building: luci-app-mwan3_git-20.029.45734-adbbd5c-1_all.ipk is quite a bit older than what the imagebuilder wants to install currently:

Installing luci-app-mwan3 (git-20.061.59608-041863a-1) to root...
Downloading http://downloads.openwrt.org/releases/19.07.1/packages/mips_24kc/luci/luci-app-mwan3_git-20.061.59608-041863a-1_all.ipk

Is there a step in the SDK that your script is missing to get the feeds up to their most recent commits?

[Andy2244]

The default feeds are taken from the SDK, which locks them to the specific point in time. Not sure what you trying to-do?

You can just grab whatever packages you want and put them in a subdir and directly add them to the config file via:

## Local feeds (src-link) have to-be in the script dir, the path is than given relative from there, without './'
## 'my-feed' dir exists in the script dir
#FEED_4="src-link test my-feed"
#FEED_4_PACKAGES="test_package"

PS: For luci packages just change the include to:
include $(TOPDIR)/feeds/luci/luci.mk

[brianjmurrell]

which locks them to the specific point in time

Oh. Hrm. That seems less than helpful. Wouldn't most people want their SDK installation to default to building the same version of any given package they'd get out the OpenWRT package repos?

IOW, updating the feeds on the SDK should march them up to whatever is the most current git hash for the SDK's OpenWRT version/branch, so that it's always building the current version (not the locked-in-time) version of a package for the OpenWRT version. Did that make sense?

As for what I'm trying to do, I'm just trying to build a local version of luci-app-mwa3 with:

### ('18.06.0', '18.06.1', '18.06.2' 'snapshots')
VERSION="19.07.1"
TARGET="ar71xx"
SUBTARGET="nand"

# build from official feeds
PACKAGES="luci-app-mwan3"

DEBUG=1

# disable signing for snapshots
DISABLE_SIGNING=1

and hoping to get a luci-app-mwan3_git-20.061.59608-041863a-1_all.ipk build, just like is current at https://downloads.openwrt.org/releases/19.07.1/packages/mips_24kc/luci/ right now.

[brianjmurrell]

Ahhh. I see:

$ cat sdk-19.07.1-ar71xx-nand/feeds.conf.default
src-git base https://git.openwrt.org/openwrt/openwrt.git;v19.07.1
src-git packages https://git.openwrt.org/feed/packages.git^1ec76dd7ca647f76a75c4f88b3b70d154221e7e9
src-git luci https://git.openwrt.org/project/luci.git^adbbd5cf690cf993e0de1e56266ac6fd2b2dfe91
src-git routing https://git.openwrt.org/feed/routing.git^839ea37939ad1c6cbb4ac543962bbb66be4a2a3f
src-git telephony https://git.openwrt.org/feed/telephony.git^6f95d6ab3f359ee2ce81c20522700937424d1591

I (wouldn't everyone?) really want to do a /luci.git/s/^.*/;openwrt-19.07/ in sdk-19.07.1-ar71xx-nand/feeds.conf then, don't I? I would argue that most people would want this project to do the same and equivillent for all feeds so that they are always building the current version of packages for their OpenWRT version.

I suppose I could set FEED_1 to be src-git luci https://git.openwrt.org/project/luci.git;openwrt-19.07 to effect the same thing. Seems counter-intuitive though.

[Andy2244]

As for what I'm trying to do, I'm just trying to build a local version of luci-app-mwa3 with:

My builder was actually created for building external custom feeds, local modified packages or reconfigure a specific package. As you noted you can always just grab a official package from the official download sources.

I think its possible to just strip the commit numbers, but the idea is mainly to build a external/custom package against the official SDK, so kernel modules and all dependencies match the SDK version.

PS: I actually use it to quickly verify my new versions against older (none snapshot) builds, without the need to setup a full build environment for each subversion.

[brianjmurrell]

reconfigure a specific package

That's what I am trying to do. But of course, I want my modified build to be what is current for my OpenWRT version plus my modification. Speaking of local modifications, I am using the tarball format that

openwrt-package-builder/build.in

Lines 383 to 408 in 07982d0

	function apply_patches {
	local patch_files="$(ls "$SCRIPT_DIR"/patches-*-${VERSION}.tar.gz 2>/dev/null )"
	if [[ -z $patch_files ]]; then
	return 0
	fi
	local file
	for file in ${patch_files[@]}; do
	if [[ -f $file ]]; then
	local feed=$(basename "$file" | cut -d '-' -f 2 | tr -d '"[:cntrl:]')
	# if (sha256sum --quiet -c "$file.sha256"); then
	# continue
	# fi
	if [[ -d $SDK_DIR/feeds/$feed ]]; then
	echo_yellow "Applying patches from $file to $SDK_DIR/feeds/$feed/"
	tar -xf "$file"
	find "$(pwd -P)" -maxdepth 1 -type f -name '*.patch' -exec patch -r - --strip=1 -d "$SDK_DIR/feeds/$feed/" -Ns -i {} \;
	rm -f *.patch
	# sha256sum -b "$file" > "$file.sha256"
	else
	echo_red "Could not apply patches from $file, since $SDK_DIR/feeds/$feed/ does not exist!"
	fi
	fi
	done
	# need to remove patch files, so if host removes them we don't reapply or use outdated!
	rm -f patches-*.tar.gz
	} 2> "$ERR_OUT" 1> "$STD_OUT"

implements. I'm wondering about the usability/workflow of this method and why it wants to delete the patch tarball when it's done? So what is the expected workflow? Where is that tarball of patches supposed to live and how is it supposed to get into $SCRIPT_DIR, given that it needs to be copied to the $SCRIPT_DIR for each run because it will be removed after the run?

I'll push some PRs once I have what I am looking for. Of course, all new features with switches so as to not change existing behaviour.

[Andy2244]

So what is the expected workflow? Where is that tarball of patches supposed to live and how is it supposed to get into $SCRIPT_DIR, given that it needs to be copied to the $SCRIPT_DIR for each run because it will be removed after the run?

The actual patches live in a folder structure inside the script dir, example:
.patches/packages/18.06.0/mypatch.patch

You have to package them via ./builder.sh bundle_patches. I personally just either use direct github PR# or copy the package in a local custom_feed directory and than use the source_link feed logic to build it from there.

PS: build.in is what runs inside the container so yes that would be the location to stick the sed into to use updated official feeds.

[brianjmurrell]

Ahh. I see. Is there a reason to use a tarball between the two? Why can't build.in just find and apply the patches in the $SCRIPT_DIR/.patches/ directory directly?

[Andy2244]

The container has no special logic for different SDK versions, that's handled outside. The patches are SDK specific so thats how i tested stuff, but as noted i personally never really use this logic, so feel free to improve it.

[brianjmurrell]

But build.in is perfectly aware of SDK version.

Looking at build.in I'm still not seeing why it cannot handle .patches/ directly rather than having to have them tarballed up.

I will try add some logic to handle .patches/ directly, in the format described to see what issues arise. It just seems to me that building an existing package in a different way (i.e. so needing to apply a patch to its' feed) is probably one of the most straightforward/common use-cases.

I could be biased. :-)

[Andy2244]

Looking at build.in I'm still not seeing why it cannot handle .patches/ directly rather than having to have them tarballed up.

Sure, to-be honest i can't really remember why i did it this way. Maybe just so i can control when/what patch versions are applied.