-
Notifications
You must be signed in to change notification settings - Fork 29.3k
Sanity Check
About
Getting Started
Client Testing Steps
Server Testing Steps
CLI Testing Steps
VS Code has an extensive release process, called Endgame, that contains multiple testing processes to ensure that we deliver a product of the highest quality to our users. Sanity testing is the final verification process a build must pass before being released.
NOTE: This testing is done for each platform build we ship.
Sanity testing is a manual process and is the opportunity to have a final set of eyes on a build before it is released to users. This testing ensures that the VS Code stable release candidate (Stable RC) installs and runs as expected.
We are looking for errors, failures, or anything else that is not desired behavior. Examples are things not loading or errors when trying to open the build in the first place.
- On the most basic level: install, launch, then open a file or the about dialog
- Other options to test a bit more:
- Click through all the items on the sidebar
- Try some command from the command palette
- Open / click around in the terminal
- Install and use an extension
During sanity testing, sections are assigned by operating system. Some operating systems along with the sections that they can test are as follows:
Operating system | Sections that it can test |
---|---|
Mac x64 | Mac x64, Windows x64, Linux x64, Linux Server, Linux CLI |
Mac ARM | Mac ARM, Windows ARM, Linux Server, Linux CLI |
Windows x64 | Windows x64, Linux x64, Linux Server, Linux CLI |
In addition to those operating systems, any web browser on devbox.microsoft.com can test Windows x64, Linux x64, Linux Server, and Linux CLI sections.
Each section on the endgame plan lists specific builds to test. Some of the builds are as follows:
- Client: the following group of builds
- Installer: on Windows, the system and user executables are installers that install VS Code to a system or user directory, respectively. With these executables, a user can launch VS Code after installing it.
- Archive: a zip or tar.gz archive. On Windows and Linux, these builds do not have an installer and can be run automatically after extracting them. On macOS, the builds can be run after extracting them and moving the application file to the
/Applications
folder. - Universal Archive: an archive specifically for macOS that supports both Intel and Apple Silicon chips by bundling two binaries into a single product.
- Debian, RPM, and Snap packages: Linux packages that require differing steps per package to install. Steps for each package are listed in the following sections.
- Server: the VS Code server. Steps for each platform are listed in the following sections.
- CLI: the VS Code command line interface that comes bundled with VS Code. During sanity testing, we use the CLI to test remote tunnels. Steps for each platform are listed in the following sections.
Once sections are assigned, follow the steps for your assigned section and/or builds below, then check off your assigned section and/or builds on the endgame plan.
- On macOS you can get a Parallels license from the team admin to spin up Windows and Linux VMs.
- On Windows (virtualized or not), you can use the sanity testing WSL images created by @sbatten.
- Dev Box, thanks to @lszomoru, allows us to spin up and access Windows x64 VMs from any browser.
- An Ubuntu Desktop VM can run the Debian, Archive, and Snap builds. A Fedora Workstation VM can run the RPM and Archive builds.
The Windows system and user executables are installers. After installing VS Code, it can be launched from the start menu.
The Windows archive can be run by extracting the archive and double-clicking on the extracted executable.
The macOS archives can be run by extracting the archive, moving the app to the /Applications
folder, and launching it.
The Linux archive can be run by extracting it, opening a terminal to that directory, and running ./code
.
The Linux packages require differing commands to install and remove:
Package | Install steps | Remove steps |
---|---|---|
Debian | sudo dpkg -i <file>.deb |
sudo dpkg -r code |
RPM | sudo dnf install <file>.rpm |
sudo dnf remove code |
Snap | sudo snap install --classic --dangerous <file>.snap |
sudo snap remove code |
After installing a Linux package, run VS Code by running code
in the terminal.
Note: You can run this test from Linux, Mac or Windows.
- Install and run Docker Desktop
- Install Dev Containers
- Install VS Code at the exact version (commit) that needs sanity checking. https://builds.code.visualstudio.com/builds/stable
- Run the following commands. They register the QEMU hardware emulator and start 5 kinds of containers:
docker run --privileged --rm tonistiigi/binfmt --uninstall '*'
docker run --pull always --privileged --rm tonistiigi/binfmt --install all
docker run -d amd64/ubuntu sleep inf
docker run -d arm64v8/ubuntu sleep inf
docker run -d arm32v7/ubuntu /bin/sh -c 'apt update && apt install -y libatomic1 && sleep inf'
docker run -d amd64/alpine sleep inf
docker run -d arm64v8/alpine sleep inf
- Check that you can connect to each of the containers using the "Attach in New Window" button for each container in the Remote Explorer. For each container:
- Check which platform you are on by running
uname -m
from the integrated terminal. (Expect:x86_64
for amd64,armv7l
for arm32 andaarch64
for arm64) - Alpine Linux runs on
x86_64
andaarch64
, checkcat /etc/os-release
shows Alpine as the distro.
- Check which platform you are on by running
- Use the Remote Explorer to remove the containers. Note that the current window's container cannot be removed, so use a new window instead.
You can sanity test the CLI and the server at the same time, by using the CLI to start a tunnel and connecting to it from a browser, specifying the right commit hash for the server version under test.
- Download and extract the CLI
- Run it with
./code tunnel
- The CLI will print a
vscode.dev
link. Open that link, but add?vscode-version=<commit hash>
to the end of the URL, where<commit hash>
is the hash of the build that is being tested. You can find this on the builds page. - Navigate to that URL. This will cause the CLI to download the specified version of the server and open a remote session in your browser over the tunnel.
This is a second option for sanity testing the Windows server. Use the Remote-SSH extension to connect from any client platform to a Windows remote. You can connect to localhost on your own Windows machine, a parallels VM, or another machine. You'll need to set up and start OpenSSH services, such as the OpenSSH SSH Server service and the OpenSSH Authentication Agent service.
Note: if you are an AAD user on your Windows machine, you may have to apply this workaround.
Here is an example localhost config that can be added to your SSH config
file:
Host localhost
HostName localhost
User <username>@microsoft.com
Where <username>
can be found by running whoami /user
on cmd.exe.
When connecting, the password is the same as the user password (which might be the Active Directory password, but definitely not the login pin).
Set the undocumented setting "remote.SSH.force32bitWindows": true
to force a 32-bit VS Code server to be installed on a 64-bit Windows host. If you have previously connected to this host with this version of VS Code, you will first want to run the command "Kill VS Code Server on Host" to remove the previously installed server.
Use the Remote-SSH extension to connect from any client platform to a macOS remote (connecting to localhost
is ok). To start the SSH server, you just have to enable Remote Login as described here.
- Download the CLI archive and extract it
- Run it with
./code tunnel
and make sure it starts and you see the license notice
Note: You can run this test from Linux, Mac or Windows.
- Install Docker Desktop
- Look up the build's commit id and replace
<commit>
with that value below. - Run the following commands one-by-one. They register the QEMU hardware emulator and start 5 Linux builds of the CLI:
export COMMIT="<commit>" # Bash
$env:COMMIT='<commit>' # PowerShell
docker run --privileged --rm tonistiigi/binfmt --uninstall '*'
docker run --pull always --privileged --rm tonistiigi/binfmt --install all
docker run -e COMMIT -it --rm --platform linux/amd64 mcr.microsoft.com/devcontainers/base:latest /bin/sh -c 'apt update && apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-x64/stable" -O- | tar -xz && ./code tunnel'
docker run -e COMMIT -it --rm --platform linux/arm64 mcr.microsoft.com/devcontainers/base:latest /bin/sh -c 'apt update && apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-arm64/stable" -O- | tar -xz && ./code tunnel'
docker run -e COMMIT -it --rm arm32v7/ubuntu /bin/sh -c 'apt update && apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-armhf/stable" -O- | tar -xz && ./code tunnel'
docker run -e COMMIT -it --rm amd64/alpine /bin/sh -c 'apk update && apk add musl libgcc libstdc++ && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-alpine-x64/stable" -O- | tar -xz && ./code tunnel'
docker run -e COMMIT -it --rm arm64v8/alpine /bin/sh -c 'apk update && apk add musl libgcc libstdc++ && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-alpine-arm64/stable" -O- | tar -xz && ./code tunnel'
- For each CLI test:
- Make sure the program starts and you see and can agree to the license
- Connect to the tunnel via
https://vscode.dev/tunnel/<name>?vscode-version=COMMIT
- Close the tab, then hit Ctrl+C on the server.
Project Management
- Roadmap
- Iteration Plans
- Development Process
- Issue Tracking
- Build Champion
- Release Process
- Running the Endgame
- Related Projects
Contributing
- How to Contribute
- Submitting Bugs and Suggestions
- Feedback Channels
- Source Code Organization
- Coding Guidelines
- Testing
- Dealing with Test Flakiness
- Contributor License Agreement
- Extension API Guidelines
- Accessibility Guidelines
Documentation