New VS Code extension

[lucbeaufils]

Introduction

IDE tooling is regularly brought up on the devenv Discord channel. Most of today's developers expect a code editor which embeds code navigation and linting, debug launch configurations and project management. Visual Studio Code is currently the most popular code editor supporting C. As such there already has been several attempts at configuring VS Code's C/CMake environment for Zephyr (See References bellow). Yet, those tools and documents are sparse and the Zephyr project as a whole doesn't provide streamlined answers to users who frequently seek help on the Zephyr Discord server to set up these extensions.

My company, Savoir-faire Linux, has developed the official VS Code extension for the Yocto Project, another very popular build system for embedded open source products. The Yocto project's community has been praising this addition to the ecosystem. It improved the ease of use for expert developers, while providing a shiny experience for new users. Since maintenance is another key aspect of such a project, we have also created CI/CD tools that allow us to ensure the tool will keep being relevant with updates. We'd like to organize an official VS Code support with the Zephyr community.

Problem Description

Setting up a Zephyr project in VS Code can be complex and is lacking an official guide. This complexity may deter IDE users from engaging with Zephyr development. Many features also remain out of reach except from the command line and users might be missing on them.

Proposed Solution

As a first and quick step, we would like to provide an official guide within the Zephyr documentation to set-up the basic functionalities through existing and well maintained extensions (C, CMake, Cortex debug). This is akin to the existing configuration guides like the one for CLion.

We'd then like to contribute the Zephyr extension's ecosystem in order to provide one extension, or a bundle of extensions that could become the officially published and maintained Zephyr VS Code experience. There already exist a dozen of Zephyr related extensions, with varying features. However most of them are isolated side projects from single developers. They may work well in some cases, but it's currently very unlikely that new users end up fully leveraging them, and it's hard to know if they'll still be maintained in a few years. Our aim would be to coordinate and build an official solution on top of these projects in the following states:

1. Get in touch with the most promising existing extensions (See References)
2. Discuss the stability or flexibility of their features
3. Develop non regression CIs and CD tooling to help with maintenance
4. Write a complete documentation guide
5. Put together those features as an official extension or bundle
6. Add new features and maintain them based on users' feedback

The resulting VS Code experience should cover the following features:

• Project Initialization and Management
  • Initialize a new project or open an existing one
  • Download the Zephyr SDK
  • Manage different project topologies
  • Create tasks for west commands
• Configuration
  • Configure the C extensions' linter
  • Configure the CMake extension to access its features
• Development Tools
  • Use menuconfig to manage defconfig and .config
  • Manage Device Tree Compiler (DTC) overlays
  • Support build and pristine build commands
  • Run advanced west commands like spdx
• Testing
  • Show Twister test suites reports
  • Run and debug Twister tests individually
• Flashing
  • Provide board-agnostic flashing commands
• Debugging
  • Debug kernel and userspace execution via the VS Code interface
  • View memory
  • Integrated serial monitor
• Internal Development
  • Non-regression tests to check against Zephyr/VS Code updates
  • CD and developer documentation to encourage contribution and maintenance

References

Existing GitHub issue: #21119

Existing extensions:

• Zephyr IDE : https://marketplace.visualstudio.com/items?itemName=mylonics.zephyr-ide by @@rijesha
• Zephyr Tools : https://marketplace.visualstudio.com/items?itemName=circuitdojo.zephyr-tools
• Cortex-Debug : https://marketplace.visualstudio.com/items?itemName=marus25.cortex-debug
• DeviceTree for the Zephyr Project : https://marketplace.visualstudio.com/items?itemName=trond-snekvik.devicetree
• Kconfig for the Zephyr Project : https://marketplace.visualstudio.com/items?itemName=trond-snekvik.kconfig-lang

VS Code existing configuration example:

Zephyr VS Code Example : https://github.com/beriberikix/zephyr-vscode-example

Some Discord users who may be interested:

• @jonathanberi (zephyr-vscode-example)
• @trond_snekvik (DeviceTree & Kconfig for the Zephyr Project)
• @royac6

Some Github users who may be interested:

• @moh60
• @deribaucourt
• @jerome-lamy
• @kartben

Yocto Project Bitbake extension: https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake

[ajarmouni-st]

Other Zephyr-related extensions:

• EditorConfig for VS Code https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig
• nRF DeviceTree https://marketplace.visualstudio.com/items?itemName=nordic-semiconductor.nrf-devicetree
• nRF Kconfig https://marketplace.visualstudio.com/items?itemName=nordic-semiconductor.nrf-kconfig
• LinkerScript https://marketplace.visualstudio.com/items?itemName=ZixuanWang.linkerscript

[Thalley]

As a Zephyr contributor, what I am look for in such an extension is (on top of my head)

• Ability to build and flash the project of the current selected file (if any) - Useful for debugging/testing multiple samples when testing a new feature
• Ability to build and run twister on the current selected test file for same reasons as above
• Ability to build and run a specific babblesim test easily (may require additional changes to the source code)
• Ability to run compliance checks locally

The above points is something I've solved locally with a few VSCode tasks, but they are somewhat limited

From a Intellisense and linting perspective

• Linting and reverse lookup of Kconfig options (both the nRF and non-nRF Kconfig extension only does this so-so with limited configuration and options)
• Intellisense of C/C++ code - The cpptools and clangd extensions already does this, but neither of them really does it well and/or requires significant configuration, nor really supports all the build-time generated files we have in Zephyr
• Include-what-you-use IWYU support (the clangd extension supports this)
• Autoformatting according to the rules we apply in CI (can mostly be done with clang-format)
• Lookup a Github PR based on a commit SHA (maybe the github extension already supports this?)

As a Zehyr contributor I don't really care for / need support for autodownloading the SDK nor creating new projects.

[Thalley]

The "problem" with compile_commands.json, is that it's from a specific build, i.e. a specific application on a specific board. When developing for (and not on) Zephyr, you want to to support many applications and boards and configurations.

I have a very broad application that I build for this purpose, but it doesn't cover anything from the shell, the unit test framework or the babblesim, so when just using compile_commands, you are not getting all the header files that you may want, nor the Kconfig options.

[JPHutchins]

Yes, that's quite hard to solve. I see this as well if I am looking through unbuilt source (from other boards for example), or perhaps looking through test files. I'm not sure that VSCode really supports using a different compilation database per file. Seems like it may require something clever with workspaces.

[deribaucourt]

Instead of compile_commands.json, one could use the CMake extension as a compiler flags provider ("configurationProvider"). If we could properly setup the CMake extension for Zephyr's environment, it could manage unbuilt files for us. The CMake extension also allows to switch between configurations (that would be board/applications for us) with a status bar selector.

AFAIK there's no configuration guide for this, but that's one of the first things we'd like to look into according to this plan.

[Thalley]

Another challenge when using Zephyr is that we have quite a few header files that are generated at build time, e.g. the autoconf.h which contain all the Kconfig values.

[ddavidebor]

Intellisense works quite well in the extension Nordic made !

[JPHutchins]

Debugging

• Debug kernel and userspace execution via the VS Code interface
• View memory
• Integrated serial monitor

The debugger should use the SVD files so that cortex-debug can watch and fiddle with peripherals directly. This is already working, but people don't necessarily know about it. It should also load the Zephyr RTOS JLink plugin (if applicable) so that there is nice thread view.

We always create cortex debug targets for bootloader + app so that we can debug either. As well as targets for on target unit tests so that you can step through those from VSCode.

[kartben]

FWIW I just had a quick look at "Zephyr IDE" VS Code extension by @rijesha and it seems pretty promising -- hopefully there can be some common ground between what's been done there already and any future "official" Zephyr extensions :)

[rijesha]

It would be great to have an official extension. Zephyr IDE could have a lot of improvements to make it a more polished and full featured experience. There are some changes in west that would help in app development. An example of this would be the ability to report back what KConfig and overlay files would be used in a build. So having the Zephyr Project backing/developing an extension would be great.

[RoyAc6]

I also see that the lack of a "universal IDE" for Zephyr is an important setback. From our experience in developing the previous IDE, SW4STM32, we see that the most important thing is simplicity and making everything work with a few clicks.

The IDE should target users are not familiar with the sophisticated and complex setup environment that Zephyr requires.
The IDE should be able to do basic stuff easily and be robust. Complex features, which are also important, can always be managed by a hardcore Zephyr programmer in one way or another.

The IDE should be able to:

1. Install the native host tool: Today, I think this is a very important point, as each OS needs a different setup and each user has a different configuration which leads to a lot of conflicts.
2. Import, set up Zephyr, and create new applications.
3. Debug: This part should be simplified as much as possible: 1-click.

Other interesting features can be added later once we have a robust tool.
Regarding these 3 points:

1. I already have some scripts that can install everything (host dependencies) automatically and set up the environment. You can check them at this repo Here is a demo to see how they work.

• There are different approaches when dealing with these tools, ours is to use "portable/standalone" tools as much as possible.
• The main purpose is not to "pollute" the host's computer with many additional tools that will lead to many conflicts and make updates more complex.
• In these scripts, even Python is installed as a portable tool, we can eventually make it global if needed

2. Zephyr IDE is a nice tool that can do this. We have also already developed a similar tool with webviews for convenience, but both can coexist, each with different ways of managing projects.
3. We need to figure out a way to simplify debugging.

[trampas]

From my experience we need "one click setup". Specifically if someone gets a URL to a repo there should one install they do to build/debug code.
For example they install VSCode (maybe extension too) then type in URL of zephyr project and then it should be able to the west init/west update and get everything setup to build the code for the board the origin repo was designed for. Basically make it as close to one click setup, install, build and flash/debug. Additionally the build should use the correct versions of GCC and build tools (see xPack https://xpack.github.io/), with the idea the build on different machines create binary exact hex files.

The best thing would be to have some way in the readme.md of a project where there is a button (one click) they click on and it will install all the tools, dependencies and IDEs needed to build the project. I do not care if this one click is done locally on a machine, or in the cloud. The point is to have some way another person can get up and running with an existing zephyr project quickly and easy.

[deribaucourt]

One click setup sounds really nice and helpful for new users. However it's also helpful only for the first steps of working with a Zephyr project. I wonder if it should be among the first workflows to address in our backlog? Maybe it's more important to have the actual features first, like build/flash/debug and code navigation, through manual configuration.

I think it would be nice in general if we have an overview of the features that should be prioritized according to the community. To anyone reading this, make sure to upvote the comments in this issue with the features you think are most helpful.

[trampas]

I would agree in general, except for my personal experience over 20 years of embedded development. Specifically it is not sane or reasonable but when you have a code base which is very hard for people to get started, they never start. I cursed Zephyr for my first week, because it was hard to get IDE and build system setup. Multiply this by number of developers using Zephyr and it starts being a lot of time and money.

I use to argue that setting up development environment should be a barrier to entry. That is if you are not smart enough or determined enough to setup the development environment you should not be messing with the code. In fact in my projects that used bare metal and eclipse the bring up process of installing eclipse and tools was 24 pages long. I found that I spent most of my time hand holding people's hand as they setup the dev environment. Then as I started doing contract work I realized that clients did not have the internal skillset to set up development environment, this is why they hired me. As such they needed a simple one click solution. They needed to feel they could replicate the binaries to know they had all the code and scripts.

I have a network of developers/contractors and one guy said they they gave up and went to segger embedded studio as it was closer to one click. They said the IDE was god awful but was worth it for the one click setup. He said with 4 developers on his team, it saved him a lot of time and headaches. At the time I thought this was insane, but learned that he was correct. That is if no one can replicate your binaries then the reality is you have created very little if any value. To understand this consider giving someone a binary file that does everything they want with no source code. They get what the binary gave them and nothing else. This is the same has having source you can not easily compile to get the binaries. This is why most here are in the open source projects, we want more than just a binary drop.

Additionally what I have found is that it is rare these days where you are the sole developer on a product. This is especially true when using Zephyr or any RTOS. As such keeping everyone synced on the same development tools is critical. Again the one click setup and keeping things sync gets compounded by number of developers on a project.

As far as debugging (running debugger) goes, it is very rare we use that or step through code much anymore. Specifically, with Zephyr RTOS and multi core parts (like Nordic BLE parts) it is very hard to step through any code. The moment you hit break point BLE will disconnect and you are hosed going forward. I personally find that using logging to be far more powerful and long term useful than stepping through code. Now stepping through code does have it place, and I personally use it once a week, but not on a daily level. Again this sounds strange but here again good logging practices make the log far more powerful, as the log will tell you where things went bad when done properly. Additionally logging to flash on released devices is powerful to fix field issues.
Additionally most bugs we run into with RTOS are timing and synchronization of threads, again here the debugger and break points rarely help in diagnosing these issues.

Building and Flashing is a big concern and here again the one step setup is to make this work. I do hate how slow zephyr projects are to build and flash, however I tolerate it with a fast computer and getting a cup of coffee when I hit build. But I would rather have slow correct builds over problems. I still would rather have one click setup than faster build/flash times, but it is a close race.

So I would still say the one click setup is the highest priority. Also note that the community of current zephyr users might bias any voting. That is you will not get votes from new zephyr users and often seasoned Zephyr users will have forgotten how hard the startup process was.

At the end of the day, your source code is useless unless someone else can replicate the binaries. Hence the need for "one click" project setup and compiling. Even if you have CI/CD tools, the source code is still of little value without the "one click" IDE.

[KozhinovAlexander]

I've also created my own Zephyr vs code workspace configuration (see my repos, take it, since it is Apache 2.0) which works well for stm32.
It works for me in every aspect, but "works for me" is not a quality measurement :)
If somebody familiar with vs code extension creation will start such a project, I'll be happy to help with Debugger embedding, West tasks creation, openocd integration, system configuration.
Thumbs up from me for the idea and great description with the overview.

[joelspadin]

I've been using Nordic's nRF connect extension for a while when working on ZMK, and my biggest issue with it is that I cannot configure it to tell it where to find things. For example, ZMK uses devicetree files to define keyboard layouts, and whenever I open one of these files using nRF Connect, the whole file lights up in errors because it didn't find all the necessary headers, and it didn't recognize the file as an overlay instead of a root board file.

My one recommendation for an official Zephyr VS Code extension would be to have defaults that match a conventional project layout, but to also support settings or other customization points to tell the extension how to work with your project.

[butok]

Please look at the MCUx VSCode extension: https://github.com/nxp-mcuxpresso/vscode-for-mcux/wiki/Working-with-Zephyr
It can be a good candidate for the Universal Zephyr IDE.
It is well supported. New release every month .

[deribaucourt]

That one seems to combine a lot of features from the other ones we have cited so far! However it doesn't seem to be opensource. So probably not suitable as an official Zephyr solution. Could be interesting to get in touch with the maintainers at NXP and see if they'd want to go open source.

[butok]

That one seems to combine a lot of features from the other ones we have cited so far! However it doesn't seem to be opensource. So probably not suitable as an official Zephyr solution. Could be interesting to get in touch with the maintainers at NXP and see if they'd want to go open source.

Yes, it has a proprietary license.
But you may influence it as a Zephyr user, submitting issues in GitHub or in the VScode:

[ajarmouni-st]

Cc @fpistm

[Ac6Embedded]

Hi everyone,

We have released the first (beta) version of the VSCode extension: Zephyr Workbench.

We've addressed several points mentioned in this discussion. Our main goal was to create a powerful tool that remains user-friendly, for both hardcore developers and beginners.
Here are some of the features:

• Install native Host Tools (Python, CMake, etc.)
• Import West workspaces from a remote repository using West, or from a local folder
• Parse West workspace data from the config file
• Create application projects for specific boards from samples
• Configure application KConfig
• Build/Flash application
• Install debuggers
• Support for Windows and Linux (macOS coming soon)

This is still in beta, and many aspects can and will be improved. The code will soon be released on our GitHub.
We look forward to your feedback and suggestions!

[hakanjansson]

I really like the original proposal in this discussion and the idea of an official Zephyr VS Code extension, or bundle, published and maintained by the Zephyr Project. Is that idea actively being pursued?

[RoyAc6]

We've been working hard over the last couple of weeks and have added many new features to Zephyr Workbench. The most important one is the "Debug Manager," a feature that allows users to create a debug session integrated with the "west debugserver." It also provides links to install locations for different runners. Tested on STM32(OpenOCD, PyOCD, Jlink) and NXP (link server), other runners are coming soon.

This is the closest we've come to achieving a "one-click and it works" solution, tailored specifically for Zephyr and west debug implementations while remaining flexible and generic.

The tool is still in beta, and we're currently working on tutorials, documentation and supporting MacOS. We would love to hear your feedback or welcome any contributions!

[zafersn]

I was wondering ST engineers thoughts on this topic and if the st teams are working on their own extension or are they going to support an open source approach? ST has one of the biggest communities here and has zephyr users. Hi @ajarmouni-st I can only see you here from the st, Will you be able to give some thoughts on this?

[erwango]

@zafersn for now ST plan is to support open source approach on this topic.