WSL2 file permission issues cause Buildroot and Yocto build failures.

[edsyl]

This file permission issue manifests itself in many ways. For embedded Linux development, the ability to run and configure Yocto and Buildroot is mandatory. Unfortunately neither build tool completes due to inexplicable file permission errors. Doing exactly the same on Mint19.3 completes with no issues.
I have tried the metadata options to no avail. I'm afraid that wsl2 is anon-starter for Embedded Linux devs on Win10.
Too bad because wsl2 is great in other ways.
Easy to reproduce. From bootlin, download the buildroot lab manual and follow the instructions. Have verified this with multiple buildroot branches as well.

https://bootlin.com/doc/training/buildroot

[rmsilva1973]

@therealkenc I've been able to reproduce the issue. Here are the steps (it's waaay long):

wget https://bootlin.com/doc/training/buildroot/buildroot-labs.tar.xz
tar xvf buildroot-labs.tar.xz
cd buildroot-labs
sudo apt-get install sed make binutils gcc g++ bash patch gzip bzip2 perl tar cpio python unzip rsync wget libncurses-dev
git clone git://git.busybox.net/buildroot
git checkout -b felabs 2020.02
make menuconfig
<select 'Target Type (i386)'>
<select 'ARM (little endian)'>
<enter Target Architecture Variant'>
<select 'cortex-A8'>


<select 'External Toolchain'>


<select 'Linux Kernel (NEW)'>


<select 'OK'>
<enable 'Build a Devie Tree Blob (DTB)'>
<enter 'In-tree Device Tree Source file names'>
<type 'am335x-boneblack-wireless'>
<select 'OK'>
<enable 'Needs host OpenSSL'>


<enable 'U-Boot'>
<enter 'U-Boot configuration (Using an in-tree board defconfig file)' menu>
<select 'Using a custom board (def)config file' option>

<type 'am335x_evm'>
<enter 'Custom make options (NEW)' menu>
<type 'DEVICE_TREE=am335x-boneblack'>
<enter 'U-Boot binary format' menu>
<select 'u-boot.img'>


<enable 'Install U-Boot SPL binary image' option>
<select 'Save' option and accept the default location>
<select 'Exit' until get back to prompt>
make 2>&1 | tee build.log

The build fails at one point with the following error:

_xzcat /home/moutinho/buildroot-labs/buildroot/dl/toolchain-external-arm-arm/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz | tar --strip-components=1 -C /home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12 --exclude='usr/lib/locale/' -xf -
rm -rf /home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain
mkdir -p /home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain
mv /home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/ /home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain/
mv: cannot move '/home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/arm-none-linux-gnueabihf' to '/home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain/arm-none-linux-gnueabihf': Permission denied
package/pkg-generic.mk:193: recipe for target '/home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/.stamp_extracted' failed
make[1]: *** [/home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/.stamp_extracted] Er
build.log.gz
build.log.gz

ror 1
Makefile:84: recipe for target '_all' failed
make: *** [all] Error 2

I'm attaching the full build.log, just in case

[edsyl]

Yes, this was the first incarnation of the issue I ran into. I tried another branch of Buildroot and then encountered a mv that failed permissions even though the destination directory was created in the previous line of the script.There are some serious issues with scripts and permissions.

[rmsilva1973]

@therealkenc With some help from the guys from buildroot (https://buildroot.org/) development team, I was able to include an strace just before the mv command that fails.. I'm uploading the strace with the system calls here. But I'm affraid the syscall which seems to be failing isn't more enlightening:

rename("/home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/arm-none-linux-gnueabihf", "/home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain/arm-none-linux-gnueabihf") = -1 EACCES (Permission denied)

Oddly enough, if I run this same mv command AFTER the build fails, it works.

build-with-strace.log.gz

[edsyl]

Yes I'm sure there is a file permission clash between wsl2 and the Linux FS. It makes me wonder how many other users of wsl2 are using large build programs for other applications as I'm sure they would encounter this issue as well. Outside of the shell environment the commands all work, but of course that is not useful for us.RegardsED

[rc-matthew-l-weber]

Does it happen to work after you test disabling windows 10 User Account Control (UAC) ?

I never noticed this in WSL1.

[edsyl]

Could never get it to build under wsl1 due to IPC/fakeroot issues. There were some hacks to get around that but they ceased to work ( according to reports) with wsl2. Unfortunately that error came at the very end of the Buildroot make and as stated, I cannot get the make to get past the permission errors. I disable UAC on my desktops. I also tried changing the wsl2 metadata settings and that did not fix it either. RegardsEd

[rc-matthew-l-weber]

IPC/fakeroot issue has been fixed in WSL1. I haven't attempted a build in WSL2 yet

[therealkenc]

rename("/home/moutinho/buildroot-labs/buildroot/output/build/toolchain-external-arm-arm-2019.12/arm-none-linux-gnueabihf", "/home/moutinho/buildroot-labs/buildroot/output/host/opt/ext-toolchain/arm-none-linux-gnueabihf") = -1 EACCES (Permission denied)

Oddly enough, if I run this same mv command AFTER the build fails, it works.

On WSL1 that's /dupe #1529 et al. You want WSL2.

[ghost]

Hi! We've identified this issue as a duplicate of another one that already exists in this repository. This specific instance is being closed in favor of tracking the concern over on the referenced thread.

Thanks for your report!

[onomatopellan]

$ ll output/images/
total 13620
drwxr-xr-x 2 onoma onoma    4096 May 28 09:50 ./
drwxr-xr-x 6 onoma onoma    4096 May 28 09:50 ../
-rw-r--r-- 1 onoma onoma   60107 May 28 09:50 am335x-boneblack-wireless.dtb
-rw-r--r-- 1 onoma onoma 8007680 May 28 09:50 rootfs.tar
-rwxr-xr-x 1 onoma onoma  107444 May 28 09:38 u-boot-spl.bin*
-rw-r--r-- 1 onoma onoma  554701 May 28 09:38 u-boot.bin
-rwx------ 1 onoma onoma  765568 May 28 09:38 u-boot.img*
-rw-r--r-- 1 onoma onoma 4435856 May 28 09:50 zImage

Following similar steps I can't reproduce on Windows 10 version 2004 using a brand new WSL2 Ubuntu 20.04 distro.

There shouldn't be file permissions problems in WSL2 since it uses a lightweight VM with real ext4 filesystem. Maybe the problem happened in a converted (WSL1 -> WSL2) distro.

[therealkenc]

2004 using a brand new WSL2 Ubuntu 20.04 distro

That is because dupe #1529 does not manifest on WSL2 (except on 9p). The OP was on WSL1. [Though technically only by assumption, because the template was deleted.]

[edsyl]

There should not be I agree. But I'm not the only one to report shell script permission  issues with wsl2. It was just the fact that I was using Buildroot and ran into these issues.  I do not use UAC on m win10 version at all. I was using Ubuntu 18.04 LTS WSL2. Since this was a wsl2 issue it should NOT be marked as a dupe as the overall wsl2 architecture has changed dramatically over wsl1

[edsyl]

How can this issue be labelled a dupe when the overall architecture has changed so drastically? It does not address the reported issue and the problem remains!!!

[therealkenc]

If you're on wsl2 then this isn't #1529.

[gkenaley]

How to reproduce permissions error under WSL 2 running Ubuntu 18.04 on Windows 10 2004:
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential git bison python2 flex diffstat gawk
$ mkdir -p ~/Projects/rpi4
$ cd ~/Projects/rpi4
$ git clone https://github.com/WindRiver-Labs/wrlinux-x.git
$ ./wrlinux-x/setup.sh --machine bcm-2xxx-rpi4 --dl-layers --distro wrlinux
$ . ./environment-setup-x86_64-wrlinuxsdk-linux
$ . ./oe-init-build-env
Edit ./conf/local.conf to allow downloads: BB_NO_NETWORK ?= '0'
If necessary comment out sanity test in ../layers/oe-core/meta/conf/sanity.conf by
commenting out this line: INHERIT += "sanity"
$ bitbake wrlinux-image-std
Note that the build (if it would finish) would take maybe 5 hours, vs. 2.5 hours on a Hyper-V VM and 35 minutes on a Xeon/nVME server.

[therealkenc]

These are not actionable test cases.

[gkenaley]

Thanks for the quick turnaround!
I had actually run this test case on Ubuntu 20.04, but I didn't want to burden you with the small patch required to make bitbake build on 20.04 (a mistake on my part). If you are up for it, here are the slightly modified instructions and attached tiny patch to make it work on Ubuntu 20.04. If you still can't make it fail then it must be machine-sensitive(?).

{Note: before the Win10 2004 update my Ivy Bridge Core i-7 PC had been running WSL1.
Earlier in this thread there is speculation that upgraded WSL installations are susceptible.
Is there a procedure to purge WSL 1 or 2 from Windows 10 and then re-install WSL 2 from scratch? Would this result in Ubuntu 20.04 using a newer kernel than 4.4?}

How to reproduce permissions error under WSL 2 running Ubuntu 20.04 on Windows 10 2004 build 19041.264 and WSL as per the attached jpg image (note it insists on using an old kernel):
:
Download meta-hamman.tar.zip and extract the meta-hamman.tar.xz from it to ~/Downloads.
meta-hamman.tar.zip
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential git bison python2 flex diffstat gawk
$ mkdir -p ~/Projects/rpi4
$ cd ~/Projects/rpi4
$ git clone https://github.com/WindRiver-Labs/wrlinux-x.git
$ ./wrlinux-x/setup.sh --machine bcm-2xxx-rpi4 --dl-layers --distro wrlinux
cd layers
tar -xvf ~/Downloads/meta-hamman.tar.xz
cd ..
$ . ./environment-setup-x86_64-wrlinuxsdk-linux
$ . ./oe-init-build-env
Edit ./conf/local.conf to allow downloads: BB_NO_NETWORK ?= '0'
If necessary comment out sanity test in ../layers/oe-core/meta/conf/sanity.conf by
commenting out this line: INHERIT += "sanity"
$ bitbake wrlinux-image-std
The warnings in yellow are the same ones I got, and are not unusual on any host.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
wruser@BULLITT:~$ uname -a
Linux BULLITT 4.4.0-19041-Microsoft #1-Microsoft Fri Dec 06 14:06:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

[therealkenc]

What I meant was these are not reproduction steps that have a high degree of probability of leading to a WSL actionable. There are too many moving parts for you (sic) to identify a diverge even if there is one.

Which isn't to suggest the effort isn't appreciated. I am a sucker for actual CLI repro steps. Unfortunately your layer thing doesn't seem to take with the steps above. But I managed to patch the stime thing manually and restart the build.

Earlier in this thread there is speculation that upgraded WSL installations are susceptible.

He means upgrading the distro (Ubuntu). You can wipe a distro from the start menu. Right click on "Ubuntu 20.04 LTS", then uninstall menu item. Then wsl.exe --set-default-version 2 from the command prompt, and then re-install Ubuntu from the store. Doing that will probably be of questionable efficacy in this instance, but who knows. [Mostly it is an exercise in starting afresh, which cures all manner of ills, both distro upgrade to WSL2 related, and unrelated to WSL entirely.]

Would this result in Ubuntu 20.04 using a newer kernel than 4.4?

No, and that is sort of the point. For all we know you could have been chasing a kernel bug, and for all we know still might. That is almost never the case. But an embedded Linux build would not be an especially effective way to find out, if it were. Or maybe you're chasing a 20.04 glibc bug. Or anything else that differs from 18.04 and 20.04.

Hypervisor bug? Always possible, in extremis. But not likely if the premise here is you need 20.04 to trigger. That's userspace.

If you still can't make it fail then it must be machine-sensitive(?).

Takes a lotta RAM, that's fer sure.

FWIW The chances this is actually an EACCES is being overestimated. That is going to be a symptom of some earlier (or simultaneous) sequence of events going sideways.

[onomatopellan]

@gkenaley If your uname -a shows 4.4.0-19041-Microsoft then you are running a WSL1 instance. For a running WSL2 instance it should show 4.19.104-microsoft-standard.
Running wsl.exe -l -v there is no doubt which version is running:

:~$ wsl.exe -l -v
  NAME                 STATE           VERSION
* Ubuntu-20.04         Running         2
  Ubuntu-20.04.wsl1    Stopped         1
  Alpine               Stopped         2

[therealkenc]

Guh I didn't even see that (looked right through it). Thanks Onomatopellan. EACCES on a directory rename WSL1 is just #1529.

[gkenaley]

@onomatopellan Thank you - that got me actually running WSL 2. I did read/google substantial documentation when upgrading from WSL 1, but it's pretty sparse. Additionally, Powershell prompted me to install the kernel update "Windows Subsystem for Linux Update" 4.19.104 when I ran "wsl.exe --set-default-version 2".
wruser@BULLITT:~$ wsl.exe -l -v
NAME STATE VERSION
Ubuntu-20.04 Running 2
I will proceed in testing on both domain and non-domain PCs. I hope the Linux build speed is competitive with a Hyper-V VM, at least.

[edsyl]

As an aside, I turn UAC off on my win10 machines. I did try changing the WSL2 permission options as well. Made no difference. On two differing win10 machines Buildroot craps out for the permission issues. I am going to try a Yocto build. I suspect I will see something similar.RegardsEd

[therealkenc]

I am going to try a Yocto build.

You can, but the probability of you being able to turn "yocto build doesn't work" into a tight repro that would get dev attention is... very small. That is because you are on a real Linux kernel on a real ext4 filesystem (on a fake hard drive). [Without implication that doing so is impossible. Because anything possible.]

What would be constructive is a screencap with the mv: cannot move fail (or equivalent) and a cat /proc/version, lsb_release -r, cmd.exe /c ver right below it. That would allow this thread to at least continue as a me2-with-screencap landing zone.

[onomatopellan]

@edsyl At OP you are talking about metadata options and that's a thing only for /mnt/* folders. If your projects are stored in /mnt/* folders then that would explain your problems even in a WSL2 instance.
So make sure you are running in a WSL2 instance (wsl.exe -l -v) and your project is stored in a /home/user folder.

[c-goes]

@edsyl At OP you are talking about metadata options and that's a thing only for /mnt/* folders. If your projects are stored in /mnt/* folders then that would explain your problems even in a WSL2 instance.
So make sure you are running in a WSL2 instance (wsl.exe -l -v) and your project is stored in a /home/user folder.

I hope that this will be possible also with external drives. Not everyone has enough space to do builds in /home/user. It's also limited to 256 GB by Windows itself. For very large projects /home/user isn't an option.
I don't see why metadata couldn't work here, it seems to build properly aside from the rename bug.

[therealkenc]

aside from the rename bug

#1529 for that.

Closing because the OP issue isn't going to make it to an actionable repro. There is a similar (but different) #2665 issue (message on down) which is instructive if one wanted to pursue this further and find an upstream work-around to building on /mnt/c or WSL1.

[Pixelpunker]

WSL2 tries to do some translating between Linux and Windows filesystem conventions that apparently isn't perfect yet.
This gave me the idea that mounting a real ext4 file system as scratch space for buildroot might solve the issue. However out of the box WSL2 can't mount neither virtual disc images nor ext4 partitions.

But there's a trick by onomatopellan that uses a second minimal linux distro installed via WSL2: How to automount an external vhdx file in WSL2

After that you will have a mount point /mnt/wsl/Alpine. Make sure that all your working directories are located there. After the build is done a post-build script can copy the output to /mnt/c/whatever which will be accessible from the Windows side.

[c-goes]

@Pixelpunker To have a second distro is the best workaround so far. Works also with official distros available here: https://docs.microsoft.com/en-us/windows/wsl/install-manual
Just extract the appx, inside there is a install.tar.gz that can be imported with wsl.exe. The paths given to wsl.exe import can be on another drive.

wsl.exe --import builddistro A:\builddistro\ .\install.tar.gz