Add support for environment variable substitution in launch.json

[SonnyDinnetz]

Relating to this issue: #717

I've added code which checks for optional properties "envFile" and "environment" in launch.json at the beginning of resolveDebugConfiguration. I tested on Windows 10 only as I don't have access to a Linux machine rn.

1. It first checks if the user has defined an envFile and if so parses it for key/value pairs.
2. After looking at the file it then appends anything it found to the "environment" property.
3. It then compares each value for each key in launch.json for values that contain ${env:key_name} and if it finds them and there was a corresponding key in either "environment" or the envFile it replaces ${env:key_name} with the associated value.

My primary reason for doing this was to make automating my build process easier. My build system organizes different builds under ./build/{software version}/{gcc output} with my makefile handling everything involved there. I've grown really fond of building/flashing/debugging all with one button click in vs code, but to do that I've had to keep the file paths in my launch.json in sync with the software version defined in my makefile, but that's rather inconvenient.

With this I can create a .env file in my .vscode folder and define any values that are unique to my setup and/or the current version in it, then .gitignore the .env file so that my coworkers can maintain their own unique environment. This shouldn't have any effect on any projects which don't use the "envFile" or "environment" properties as it only makes changes to the launch config if those values are present.

Formatting for the key/value pairs:

• Any file path string which resolves to a file that can be parsed.
• File cannot begin with a comment or the regex on line 308 of configprovider.ts won't find matches (line.match(/^([^#\s]+)=(.*)$/))
• Cannot use ${workspaceFolder} to locate the file due to this function being called before VS code does it's substitution (if we wait until afterwards then we lose the placeholders from the value strings in launch.json i.e. ${env:KEY})

Example launch configuration:

{
    "name": "Build & Debug - J-Link", 
    "envFile": "./.vscode/.env",
    "cwd": "${workspaceFolder}",
    "type": "cortex-debug",
    "executable": "${workspaceFolder}/build/${env:software_revision}/${env:software_revision}_DEBUG.elf",
    "request": "launch",
    "servertype": "jlink",
    "serverpath": "${env:JLINK_GDB_PATH}",
    "gdbPath": "${env:ST_GDB_PATH}",
    "device": "STM32G431KBTx",
    "runToEntryPoint": "main",
    "svdFile": "${env:SVD_PATH}",
    "svdPath": "${env:SVD_PATH}",
    "liveWatch": {
        "enabled": true,
        "samplesPerSecond": 4
    },
    "v1": false,
    "serverArgs": [
        "-speed", "8000",
        "-vd"
    ],
    "interface": "swd",
    "ipAddress": "192.168.1.50",
    "preLaunchTask": "Make Debug"
}

Example .env file (with fake key obv):

software_revision="0.01.05A"
ST_GDB_PATH="C:/ST/STM32CubeCLT_1.15.0/GNU-tools-for-STM32/bin/arm-none-eabi-gdb.exe"
JLINK_GDB_PATH="C:/Program Files/SEGGER/JLink_V796s/JLinkGDBServerCL.exe"
SVD_PATH="C:/ST/STM32CubeCLT_1.15.0/STMicroelectronics_CMSIS_SVD/STM32G431.svd"

AES_PRIVATE_KEY="ABCDEFGHIJKLMNOP"

[haneefdm]

I am a bit confused about the envFile and what other debuggers do with it. It is generally meant for environment variables for the debugger or the target and not for replacements of other stuff in launch.json. The launch.json is just a VSCode thing. I the package.json, the documentation is inadequate for what envFile/environmet does and the precedence rules. See

https://code.visualstudio.com/docs/python/environments#_environment-variables
https://code.visualstudio.com/docs/cpp/launch-json-reference#_environment

Using the same names (envFile/environment) for a totally different purpose can raise support issues and is confusing. Even to me because I cross boundaries between embedded and non-embedded environments.

I haven't done a deep dive on what other do but reading from their docs the meaning does not seem to be the same.

How does work with preLaunchTask if it modifies the envFile? See microsoft/vscode-cpptools#5060 (comment)

The whole reason people were asking for this is so that their build can populate the envFile. The way this is done, how can it work? Did you test it? What does cpptools do?

[haneefdm]

Your lint is failing. Please run npm run lint to make sure. Also do a npn run package so there are no build issues and that it packages properly

[haneefdm]

This algorithm is O(n*m) isnt int? We can do this in O(n+m). Imagine a 100 env. vars and a 100 properties in the launch config.

[SonnyDinnetz]

I've been thinking about how to make this more efficient and wasn't sure how to get to O(n+m). Do you have an idea you can point me at?

My new strategy is to check each key in launch.json for ${env:*} noting them down as I go. Then in a separate loop check if they're in the envFile and replace them then. Should reduce the number of comparisons I'm doing a lot but I don't know if it actually changes the bigO.

[haneefdm]

So, who handles the actual environment variables? Maybe I missed it but are you merging the process.env with what we find in the launch config? I would start with (a copy of) process.env as the initial environment.

I am assuming VSCode uses process.env as well. And have clear/documented precedence rules.

[haneefdm]

What do you know of case sensitivity/preservation nature of env vars on Windows VSCode? What Windows does may not matter. What matters is what does VSCode do on Windows. We have to emulate whatever that behavior is.

[SonnyDinnetz]

What do you know of case sensitivity/preservation nature of env vars on Windows VSCode? What Windows does may not matter. What matters is what does VSCode do on Windows. We have to emulate whatever that behavior is.

Wasn't able to find documentation that outright said whether or not VSCode was intended to obey case sensitivity for env variables but I did check and at least for the ${workspaceFolder} substitution it did so I went ahead and made sure that my search considered case before replacing.

[SonnyDinnetz]

I am a bit confused about the envFile and what other debuggers do with it. It is generally meant for environment variables for the debugger or the target and not for replacements of other stuff in launch.json. The launch.json is just a VSCode thing. I the package.json, the documentation is inadequate for what envFile/environmet does and the precedence rules. See

I'll get more and clearer docs made up. I'll also get something prepared for the wiki too so I can go into details without cluttering the package.json too much.

https://code.visualstudio.com/docs/python/environments#_environment-variables https://code.visualstudio.com/docs/cpp/launch-json-reference#_environment

Using the same names (envFile/environment) for a totally different purpose can raise support issues and is confusing. Even to me because I cross boundaries between embedded and non-embedded environments.

I haven't done a deep dive on what other do but reading from their docs the meaning does not seem to be the same.

The documentation for other implementations of these are super lacking and it seems like they all handle it differently. For example:

https://code.visualstudio.com/docs/python/environments#_environment-variables
This implementation seems to treat envFiles almost like lumps of python constants to swap out. In the example they have I think of swapping dev.env for prod.env in the launch config the same as changing the import itself.

https://code.visualstudio.com/docs/cpp/launch-json-reference#_environment
Then in this one they give next to no explanation at all but the syntax they use is completely different. Defining the key and value entirely separately using the keys "name" and "value" to define each key and value.

https://code.visualstudio.com/docs/reference/variables-reference
This was what I was using as my inspiration for environment variables. They of course don't come out and say it but they treat USERNAME here as if they were drawing it from the system variables ala powershell>echo ${env:USERNAME}. My thought process is that I don't want to define a ton of system variables on my machine when I really only want them to exist project to project so I create a file which can be dedicated to a single project or even launch config.

How does work with preLaunchTask if it modifies the envFile? See microsoft/vscode-cpptools#5060 (comment)

I played around with placing the whole routine in resolveDebugConfigurationWithSubstitutedVariables instead of resolveDebugConfiguration but that runs into supercession issues with the actual system variables. Essentially, some time between resolveDebugConfiguration and resolveDebugConfigurationWithSubstitutedVariables VSCode is doing it's own substitution which would replace any instance of ${env:*} that it doesn't find with an empty string making the searches and swaps done in my function always fail. What this guy microsoft/vscode-cpptools#5060 (comment) is saying fits what I'm seeing. In my implementation preLaunch tasks should have no effect on the envFile or environment since by the time they run they should have already been parsed and swapped out.
I'm definitely open to trying to find a solution for this though, maybe a different key than "${env:*}" that way windows doesn't touch it before we do?

The whole reason people were asking for this is so that their build can populate the envFile. The way this is done, how can it work?

My use case is entirely in swapping out aspects of the launch configurations. Moreso using the envFile to populate aspects of my builds than the the build populating the envFile.

Did you test it?

I tested a bunch on windows, I'm planning to do some runs on a linux box this weekend to make sure it acts like I expect it to.

What does cpptools do?

As far as I can tell it doesn't interact with cpptools at all. What drove me to do this in the first place is that when I have cortex-debug selected as the type for a launch config it won't allow me to use the "environment" or "envFile" keywords at all in the launch configs. I think it warrants some digging though so I'll find some time to build a blinky project that uses the cpptools "environment" keyword and see if maybe I'm missing something and if it integrates well with the changes I'm asking for.

[haneefdm]

What does cpptools do?

As far as I can tell it doesn't interact with cpptools at all

That is not what I meant. Of course cpptools will not be involved if the debug type is cortex-debug. What I meant is that they have a similar feature -- how to they process it. It could be good to use their model if applicable. I would not create an embedded program to test. I would use a normal desktop C program.

I know it is easy to simply do what is needed for "my" use case. But, we have to look at the general picture. We are kind of doing this backwards. We are deriving specifications/requirements while developing. Even when I want to do something, I submit a proposal (using an issue here) and solicit feedback before writing any code.

Wasn't able to find documentation that outright said whether or not VSCode was intended to obey case sensitivity for env variables but I did check and at least for the ${workspaceFolder} substitution it did so I went ahead and made sure that my search considered case before replacing.

Welcome to my world. Most of the time, I have to reverse engineer stuff like this. I would not use ${workspaceFolder} to test something like this. The rules are specific to ${env:...}

I don't want to end up with two sets of rules and two processors. We need to deal with ALL env vars. No half and half.

And, thanks for taking the initiative. I knew it would not be as simple and if I could achieve the true goal of WHY people were asking for it.