Adding a docker-based dev env

[kraftbj]

Testing moving development commands to within docker to avoid local variances or require specific versions of things to be installed locally.

Proposed changes:

• Adds a new js-package for a new jetpack cli.
• Adds a docker setup for a new dev env.

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

None yet.

Does this pull request change what data or activity we track or use?

No.

Testing instructions:

Testing the new CLI

• Ensure you have Docker Desktop (Mac, if applicable), git, and node installed.
• On Mac, it seems you have to go to the Docker Desktop settings->Resources->Proxy->check "Manual Proxy Configuation" [otherwise the wordpress container can't reach wordpress.org, haven't dug into that]. Restart Docker if applicable.
• npm i -g @automattic/jetpack-cli
• cd /tmp (or somewhere else without Jetpack)
• jp -v (confirm it is installed, 1.0.0-beta1 is the current version)
• jp --help -- it should report it can't find a monorepo checkout and invite you to use jp init
• jp init -- follow prompts
• cd jetpack (or whatever dir)
• git checkout try/dev-in-docker to get on this branch.
• jp --help -- this should pull the new dev container (I manually pushed it to docker hub for the sake of testing)
• jp build plugins/jetpack --with-deps (does it work?)
• jp docker up -d -- the CLI output will not be the expected "visit localhost". it'll say it's done before it's ready (need to fix).
• Visit localhost ensure it is the WP install.
• jp docker install -- should run the usual install process, localhost should show a regular site.
• jp docker down

Testing the existing setup.

• In a working dev environment, e.g. jetpack commands work.
• Do nothing else.
• Run jetpack --help, jetpack build plugins/jetpack --with-deps
• Run jetpack docker up -d
• etc
• NOTE: You can switch to the new CLI in your existing checkout if you delete the node_modules dir. If you want to go back and forth between the two, you'll need to nuke the node_modules each time you switch.
  Does everything still work as expected? If not, does clearing the docker containers/images resolve it? Please note that.

The intent of the PR as this point is create a new, rough, beta way to dev within docker without impacting the existing jetpack commands running locally for devs who have things working. I hope to quickly iterate to the dev-in-docker is a reliable way for anyone to jump into, but the local way of doing it should still work too (and for the foreseeable future).

Known issue: Git Hooks do not work on a fresh install using only jp for dev. It'll commit, push, etc without running the hooks.

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WordPress.com Simple site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin, and enable the try/dev-in-docker branch.

  • For jetpack-mu-wpcom changes, also add define( 'JETPACK_MU_WPCOM_LOAD_VIA_BETA_PLUGIN', true ); to your wp-config.php file.

• To test on Simple, run the following command on your sandbox:

  bin/jetpack-downloader test jetpack try/dev-in-docker
  

  bin/jetpack-downloader test jetpack-mu-wpcom-plugin try/dev-in-docker
  

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Choose a review path based on your changes:
   • A. Team Review: add the "[Status] Needs Team Review" label
     • For most changes, including minor cross-team impacts.
     • Example: Updating a team-specific component or a small change to a shared library.
   • B. Crew Review: add the "[Status] Needs Review" label
     • For significant changes to core functionality.
     • Example: Major updates to a shared library or complex features.
   • C. Both: Start with Team, then request Crew
     • For complex changes or when you need extra confidence.
     • Example: Refactor affecting multiple systems.
3. Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!

[jeherve]

I wanted to test this from a fresh computer, so I tried to start from a computer running Ubuntu, where nothing was installed yet.

Unfortunately, I run into an error when I run BUILD_LOCAL=1 jp build plugins/jetpack --with-deps:

ERROR: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

I'm not sure what's causing this, and I'm not familiar enough with Ubuntu to debug this I'm afraid.

[anomiex]

Gave it a quick once-over.

[kraftbj]

In addition to testing on my two production Apple Silicon (M1, M2) devices, I tested it on a clear Intel Mac using brew install node and downloading Docker Desktop. I only tested the new version on the clean machine since the intent here is to make it fast for someone new to start.

[anomiex]

Other than all the doc changes to point at the new jp command and hide jetpack, I don't see anything that will impact use of jetpack anymore at a quick read-through.

[anomiex]

Random thought: Would it be more or less confusing to also name this tool jetpack?

• Pro: Whether you're using this new tool or have done jetpack cli link and are using the old one, almost all the commands you'll run are the same.
• Con: We'd need to invent terminology to clearly talk about tools/cli jetpack versus js-packages jetpack.
• Con: If there are functionality differences between the two tools, that could be really confusing when someone runs into it.

[kraftbj]

Random thought: Would it be more or less confusing to also name this tool jetpack?

• Pro: Whether you're using this new tool or have done jetpack cli link and are using the old one, almost all the commands you'll run are the same.
• Con: We'd need to invent terminology to clearly talk about tools/cli jetpack versus js-packages jetpack.
• Con: If there are functionality differences between the two tools, that could be really confusing when someone runs into it.

TBH, my original thought was this would be the "new" jetpack command and had started on having both a jp and jetpack to run the new dev container, with a migration script if the old jetpack code was being fired.

But I didn't move on that to try not to add too many areas that could break things at once -- as in, today in theory, you can use one or the other. If you use "existing" jetpack, it'll just keep working the same, even if the new ones has some bugs. And, then, just having it be a wrapper of the existing commands, I was confusing myself trying to remember "do I mean the new jetpack or the pnpm jetpack, etc.

tl;dr: I'm open to this becoming the jetpack CLI, but I figure that could be a migration done in the future.

[kraftbj]

@anomiex Would you mind reviewing this for merging with the blocking criteria being things that would break the existing jetpack command for those who already have things working?

I would say the jp command is beta and not feature-complete, but more folks can start testing it for "regular" development with other branches, PRs, etc.

[anomiex]

Personally I'd rather keep using the existing non-Docker jetpack command. And anything that does pnpm jetpack in CI will need to keep working somehow, so completely replacing the tools/cli jetpack might be tough.

[anomiex]

Left a few inline suggestions, but from the perspective of "shouldn't break existing monorepo stuff" this LGTM. The suggestions could even be left for a followup.

[anomiex]

🤔 Interesting...

This means it won't be runnable directly from a monorepo checkout. But maybe you could somehow point npm -g at the monorepo checkout as a source when testing?

And hopefully eslint doesn't get confused by missing imports, although I suppose we could probably just disable the relevant rules.

[kraftbj]

Nod, with having it in the workspace, pnpm cli-setup would pick up on it and globally link jp to the copy in that monorepo. Then, when you nerf node_modules (or delete that checkout totally) and want to use jp init to start again, no joy.

It might be something we can adjust in another way, but if we keep jp to be relatively stable (most everything it does fires code within the monorepo's pnpm install), it might not be horrible to have it be annoying to test changes within the monorepo? Or, you can npm install within the projects/js-packages/jetpack-clidir and run node projects/js-packages/jetpack-cli/bin/jp.js [cmd] (how I did most of my testing).

[anomiex]

I wasn't too worried about the potential difficulty in testing. But documenting how to test it in the js-package's README.md might be a good idea.

[anomiex]

Probably no tests ever, since installing jest won't be possible with it being excluded from pnpm-workspace.yaml.

In which case, again, deleting this so CI will be able to skip it entirely might be good.

[anomiex]

Might be worth deleting this placeholder file.

[anomiex]

Might be worth deleting this placeholder file.

[anomiex]

As mentioned above, with this project being excluded from pnpm-workspace.yaml, you won't be able to have jest installed so having a jest config is probably useless.

[kraftbj]

Thanks! I'll clean up those placeholders in the next PR.

[robertsreberski]

Hey @kraftbj @anomiex this way of setting up Jetpack feels like a breeze!

What's stopping us from making this way of managing env the default one, and describing it in the README?

Happy to help coordinate actions, if needed.

[anomiex]

#40934 (comment) above discusses it.

In short: If we rename this jp to jetpack, then we can avoid having to update documentation everywhere to say jp instead of jetpack, plus we can keep the option of using tools/cli jetpack as an "advanced" option for people who don't want to globally-install the new package.

But doing all that at once seems risky, so let's take it in some steps.

[kraftbj]

Two things that are on my mind:

1. git hooks don't work if you don't have things setup locally, which is just something I haven't explored ways to resolve.
2. It's actually pretty slow for some operations over direct on local (e.g. jp install --all when there's a lot of IO activity)

I'm going to write a p2 post today to announce is as a public beta asking a12s to let us know anything unexpected about it.

I haven't tried every jetpack command as a jp command yet either. I know the basic jp docker ones work, but that's the most interesting part of the new setup (since I couldn't get the new docker to communicate with the host to spin up other containers to work, so jp docker commands selectively are executed on the host machine or the dev env container).