wp-env: No longer working on Linux

[ockham]

Describe the bug
For the past couple of days, I haven't been able to run wp-env on my Linux box.

To reproduce
On Linux, in your Gutenberg source directory, run npx wp-env start. It'll fail with the following error:

✖ .IOError: [Errno 13] Permission denied: '~/.wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/docker-compose.yml'

(I got the same error with npm -g i @wordpress/env && wp-env start.)

Desktop (please complete the following information):

• OS: Linux (Ubuntu 18.04.3 LTS)

Additional context
This might correlate with #20002 being merged, but I'm not sure it's caused by that PR. /cc @noisysocks

Can we harden Gutenberg against issues like this? For the past ten days, of the time I spent working on Gutenberg, an estimated 50% went into trying to get it to run, figuring out that docs were outdated, updating them, and, just as I thought that things were running smoothely, getting stuck again with this new issue. That doesn't feel like a good use of my time 🙁

[simison]

Sorry to hear about the trouble.

Note that build scripts use wp-scripts so that's what gets run in CI.

Would a good first step be to add simple dummy wp-env run into CI and then start working on migrating package.json scripts to it?

[noahtallen]

This might correlate with #20002 being merged, but I'm not sure it's caused by that PR.

It does sound as if that PR caused the issue. Instead of downloading the required files to cwd /wordpress and then putting a docker-compose override there, the new setup creates the docker-compose file and other dependencies at ~/.wp-env/. It sounds like the Linux issues mostly stem from filesystem permissions.

Can we harden Gutenberg against issues like this

I think yes. :) The wp-env tool is new enough that use cases like your own are still being uncovered. Indeed, your first issue dealing with the @wordpress/scripts env command was essentially the first point that @wordpress/env was really mentioned as the best way to start a local environment, so you're really on the bleeding edge of things here! 😁 I've been fixing other issues I'm finding along the way (#20158, #20157, #20113), and hopefully we can make this a robust tool for local wordpress environment. (It is already working much faster than the old @wordpress/scripts based solution when it works :))

If you like, you could try to fix the issue. I could see it being hard to replicate it since I don't have a linux box. I think the problem happens at these lines:

gutenberg/packages/env/lib/env.js

Lines 215 to 227 in a59a5d9

	async function initConfig() {
	const configPath = path.resolve( '.wp-env.json' );
	const config = await readConfig( configPath );
	await fs.mkdir( config.workDirectoryPath, { recursive: true } );
	await fs.writeFile(
	config.dockerComposeConfigPath,
	yaml.dump( buildDockerComposeConfig( config ) )
	);
	return config;
	}

You can run the source code directly while developing (no need to build anything)

$ path/to/gutenberg/packages/env/bin/wp-env start

In the short term (and I hate suggesting this), I wonder if sudo npx wp-env start would work?

[ramonjd]

I could see it being hard to replicate it since I don't have a linux box

I run Ubuntu on Virtual box. It's like wading through slag in a treacle body suit, but it's good enough for testing.

[noisysocks]

Sorry you're running into issues @ockham! Like @noahtallen says, wp-env has just undergone a major rewrite and so there will definitely be some kinks to work out! I intend to continue work on it over the next month or two and make it hard enough for us to migrate completely to wp-env.

I suspect that your user doesn't have write permission for the new environment temporary directory (~/.wp-env). Does running this fix the problem?

sudo chown -R username ~/.wp-env

Replace username with your username. You can find this out by running the whoami command.

We may need to look at using a different directory for wp-env's temporary folder on Linux systems, or allowing the user to specify a different directory using e.g. an environment variable.

[ockham]

Sorry you're running into issues @ockham! Like @noahtallen says, wp-env has just undergone a major rewrite and so there will definitely be some kinks to work out! I intend to continue work on it over the next month or two and make it hard enough for us to migrate completely to wp-env.

I suspect that your user doesn't have write permission for the new environment temporary directory (~/.wp-env). Does running this fix the problem?

sudo chown -R username ~/.wp-env

Replace username with your username. You can find this out by running the whoami command.

I'm afraid that doesn't fix the issue. ~ is my user's home folder -- it's basically the one folder that's guaranteed to have all the necessary permissions for my user 🙃 I've deleted the ~/.wp-env folder in between runs, so it's newly created by the wp-env tool each time. It must be indeed the tool itself that assigns weird permissions to some subdirectory or file.

[ockham]

Possibly related: https://stackoverflow.com/questions/53344380/errno-13-while-running-docker-compose-up

(I'm doing this inside of my home directory tho, so the answer about adding it to some AppArmor config don't seem to apply 🤔 )

[ockham]

AFAICT, the error is happening on this line:

gutenberg/packages/env/lib/env.js

Line 57 in 9d67638

dockerCompose.upOne( 'mysql', {

Since docker-compose works fine for me with other projects, my hunch is that the reason for my issue might be somewhere in the docker-compose npm that this tool uses.

Specifically, I think that prior to #20002, dockerCompose.oneUp wasn't used (only dockerCompose.run), so I'm going to look into that command's implementation.

[ockham]

Nah, nothing to do with the docker-compose npm implementation. It's really just about the docker-compose.yml location. The following patch fixes the issue for me:

diff --git a/packages/env/lib/env.js b/packages/env/lib/env.js
index 94421544d2..ace4502a51 100644
--- a/packages/env/lib/env.js
+++ b/packages/env/lib/env.js
@@ -216,6 +216,8 @@ async function initConfig() {
        const configPath = path.resolve( '.wp-env.json' );
        const config = await readConfig( configPath );
 
+       config.dockerComposeConfigPath = path.resolve( 'docker-compose.yml' );
+
        await fs.mkdir( config.workDirectoryPath, { recursive: true } );
 
        await fs.writeFile(

[ockham]

There's a chance that the issue I'm seeing is related to the docker Snap package for Ubuntu that I'm using, which might have an impact on how binaries from a given Snap can access files. (The solutions described at https://stackoverflow.com/questions/53344380/errno-13-while-running-docker-compose-up didn't work for me.)

I think this is a rather widespread and recommended way to install Docker on Ubuntu (and other Linux distributions), plus it seems to be overall well-maintained and provides up-to-date versions. I think it's also the simplest way to install Docker on a number of Linux distributions; compare e.g. with these instructions.

In conclusion, it'd be good to make wp-env work with those Snaps 🙂

[noahtallen]

The following patch fixes the issue for me:

Interesting, so docker cannot run a docker-compose.yml from ~/.wp-env/whateverhash/docker-compose.yml, but it can from your source code 🤔

As I mentioned in #20002, what do you think about having an environment variable to change ~/.wp-env/ to something else? Along with that, I wonder if there is a general directory on linux that could work for this location.

[ockham]

FWIW, I reported the problem (Docker snap being unable to access file in a hidden directory) here: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1863604

[anonymouse64]

The docker snap cannot access directories that start with a "." in the home folder. This is a generic security mechanism for all snaps. I'd recommend moving the directory out of ~/.wp-env to somewhere like ~/wp-env or allow configuring this somehow.

[ockham]

Thanks @anonymouse64!

So @noahtallen I guess it'll be good to proceed with #20253. Is there any chance we can make the default a non-hidden directory, so wp-env works OOTB on Ubuntu? /cc @noisysocks @epiqueras

[epiqueras]

Yes, we need #20253 as an additional measure, but I'd prefer to also remove the . from ~/.wp-env so that this works by default. @noisysocks Do you have any reservations against that? I don't think it's a problem if this folder is not hidden.

[noahtallen]

The only thing I can think is that pretty much all developer dev tool folders like this are hidden by default. Not sure why, but it is the common practice:

@ockham fantastic, I'll update the path in that PR as well. Maybe just on linux to start (if possible), but could also do it overall.

[epiqueras]

Just on Linux would be better. Having it hidden is still nice, if possible.

[noisysocks]

Yes, we need #20253 as an additional measure, but I'd prefer to also remove the . from ~/.wp-env so that this works by default. @noisysocks Do you have any reservations against that? I don't think it's a problem if this folder is not hidden.

Would something like ~/Library/Application Support/wp-env work?

[epiqueras]

Does Linux also have that directory?

[noisysocks]

I searched briefly and it looks like ~/.whatever is the conventional name for a support directory on Linux systems 😅

Oh well. I don't much mind. A hidden directory is preferable since the contents of .wp-env aren't exactly user-friendly (all of the directories inside it have hash names), but gotta do what you gotta do!

[epiqueras]

Let's track https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1863604 and remove it ASAP.

[sainthkh]

https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1863604 is closed with won't fix tag. They said they won't fix it for security.

[epiqueras]

What’s the security concern?

[sainthkh]

@epiqueras Below is the copy of the answer.

The docker snap cannot access directories that start with a "." in the home folder. This is a generic security mechanism for all snaps. I'd recommend moving the directory out of ~/.wp-env to somewhere like ~/wp-env or allow configuring this somehow.

[noahtallen]

I'd recommend that one installs the latest major version

Definitely :) I was mostly trying to test wp-env behavior at the time, which is why I used master (from the other plugin that I develop in)

[ockham]

I ran npx wp-env start --debug to track down why wp-env isn't working for me on second try. Here's the log leading up to the error:

ℹ NodeGit: Cloning or getting the repo.
ℹ NodeGit: Repo already exists, get it.
ℹ NodeGit: Fetching the specified ref.
ℹ NodeGit: Checking out the specified ref.
⠦ Downloading WordPress.
  - core: 100/100%Creating network "6ac6b03bd54e5e922f0d180f6d7ff25d_default" with the default driver
✖ EACCES: permission denied, open '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'
[Error: EACCES: permission denied, open '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'
}

Not sure that's very useful tho 😅

[sainthkh]

After #20648 is merged, I can run the sudo npx wp-env start on Ubuntu.

NOTE: sudo is required here.

For some reason, it fails at the first attempt like below:

sainthkh@sainthkh-All-Series:~/v/gutenberg$ sudo npx wp-env start
[sudo] password for sainthkh: 
✖ Error while running docker-compose command.
Starting aac851dfb8c3a8cb59795ad0b12671c7_mysql_1 ... done
Starting aac851dfb8c3a8cb59795ad0b12671c7_wordpress_1 ... done
mysqlcheck: Got error: 1049: Unknown database 'wordpress' when selecting the database

But after that, it works without any problem:

sainthkh@sainthkh-All-Series:~/v/gutenberg$ sudo npx wp-env start
✔ WordPress started. (in 87s 64ms)
sainthkh@sainthkh-All-Series:~/v/gutenberg$ sudo npx wp-env stop
✔ Stopped WordPress. (in 8s 583ms)

If it's the case for everyone, we might need to close this issue and focus on solving why mysqlcheck: Got error: 1049: Unknown database 'wordpress' when selecting the database doesn't work at the first attempt.

[noahtallen]

Is sudo required just the first time or also on all the subsequent runs?

[sainthkh]

@noahtallen every time.

[ockham]

I realized that the Docker env was working, even though I was still getting

Error: Could not process the 'wp-config.php' transformation.
Reason: wp-config.php is not writable.

The latter has just been fixed by #22682, so I'll close this issue.

(I'm still getting an Error: Stylesheet is missing., but I guess that's unrelated. Media upload is also still impossible. I might file another issue for that.)

[noahtallen]

Error: Stylesheet is missing

wait, what's this error!? 😋

@sainthkh Is the DB connection issue still problematic? Also, regarding sudo, is that normally needed for docker-compose/docker on linux?

[sainthkh]

Actually, things became worse after I upgraded to Ubuntu 20.04. It just fails with the segmentation fault message even with sudo. I don't know what I misconfigured.

Because of that, I'm currently working on Windows.

[noahtallen]

That's too bad. Could you open a new issue with that specific error message?

[sainthkh]

@noahtallen Opened an issue at #22731.

[noahtallen]

Thank you! didn't want to clutter this one up too much more, since it has already covered two or three different issues 😅

[danieliser]

On WSL 2 (Ubuntu 20.04), and ran into this recently.

The sudo chown -R username ~/.wp-env command didn't quite do it, because the folder was actually ~/wp-env. Corrected that and it worked perfectly. Finally I can restart wp-env without destroying and rebuilding.

[danieliser]

@ockham - Was this patched and just not released yet? I'm on the latest release and this is still happening for me. I'm having to do the chown pretty much every time I restart my computer or the VM or stop & start the wp-env again.

[noahtallen]

@danieliser there was another (similar) issue I fixed in #23797. Can you paste the output of the error you see when running wp-env a second time so that we can see if the error is the same?

Was this patched and just not released yet

This specific issue should have been released, but the one I mentioned above has not yet been released :/

[noahtallen]

If you're running wp-env from Gutenberg, though, you should be referencing the local source code (e.g. npx wp-env or npm run wp-env), and shouldn't see issues developing the gutenberg project specifically.

[danieliser]

@noahtallen - Not using it for Gutenberg, just in general plugin development and testing. Hasn't happened today yet, but I will post the error.

It was this same one though:

✖ EACCES: permission denied, open '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'
[Error: EACCES: permission denied, open '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/home/me/wp-env/6ac6b03bd54e5e922f0d180f6d7ff25d/tests-WordPress/wp-config.php'
}

Error: Could not process the 'wp-config.php' transformation.
Reason: wp-config.php is not writable.

[noahtallen]

Does it happen every time you run wp-env start if it is already running?

Btw, this issue should fix this altogether and make filesystem permissions in wp-env work a lot better: #22515. But it needs some eyes and work to make progress

[danieliser]

@noahtallen No it seems to be related to either restarting the host OS (Windows 10) or the rebooting of the WSL vm.

This one came up today after a reboot:

✖ EACCES: permission denied, open '/home/daniel/wp-env/a539d783750b32e7f42cbafde69608aa/tests-wordpress-latest/wp-content/plugins/fill-data.php'
[Error: EACCES: permission denied, open '/home/daniel/wp-env/a539d783750b32e7f42cbafde69608aa/tests-wordpress-latest/wp-content/plugins/fill-data.php'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/home/daniel/wp-env/a539d783750b32e7f42cbafde69608aa/tests-wordpress-latest/wp-content/plugins/fill-data.php'
}

After running the command to fix it I still got this:

✖ Error while running docker-compose command.
Starting a539d783750b32e7f42cbafde69608aa_mysql_1 ... done
Starting a539d783750b32e7f42cbafde69608aa_tests-wordpress_1 ... done
Error: Could not process the 'wp-config.php' transformation.
Reason: wp-config.php is not writable.

[noahtallen]

🙃 By the way, do you use sudo to run docker & wp-env commands? I've had to do that myself on ubuntu, and I hear that is normal for docker on linux

[danieliser]

No, I haven't had to do that.

[nextgenthemes]

@danieliser I guess you have made special post setup docker group setup because in fact @noahtallen is correct that you can not run any docker command without sudo with a normal basic installation. I am just thinking about what to do because I end up with Could not connect to Docker. Is it running? even though I can run sudo docker run hello-world and it works, just not without sudo.

https://docs.docker.com/engine/install/linux-postinstall/

Enabling it without sudo opens a security hole, running docker without root seems to have many restrictions and requires complicated extra setup. But running npm script with sudo is a total nogo, that it probably the biggest security risk. I guess I will go with the docker group thing. I have not tried it, but I guess it would work with sudo just like direct docker commands.

[noahtallen]

I set up Gutenberg development on a new Linux box recently, and doing the postinstall steps to add my user to the docker group was a requirement to get wp-env working.

I agree with you that requiring sudo to use wp-env would probably be even worse.

[apeatling]

I'm running into this issue with a Raspberry Pi. I've run all post install steps in the Docker docs. Everything works fine the first time I start wp-env, but any subsequent times I restart it, I run into either the EACCES issue or wp-config.php is not writable issue.

It seems the files it's complaining about are attached to the root user, instead of your current user that has been added to the docker group.