How do I test that extension code is showing error/warning/info messages?

[DanTup]

I'd like to write tests for my extension that tests code that calls window.showErrorMessage (and friends). For example, if I don't find a required SDK, I display an error message with a button that lets the user locate the SDK:

	const locateAction = "Locate SDK";
	const downloadAction = "Download SDK";
	let displayMessage = `Could not find a ${sdkType} SDK. ` +
		`Please ensure ${sdkType.toLowerCase()} is installed and in your PATH (you may need to restart).`;
	const selectedItem = await window.showErrorMessage(displayMessage,
		locateAction,
		downloadAction,
	);

Is there some way that in a test I can verify that window.showErrorMessage was called?

I thought I could just wrap the function in my test setup with something that counts when it gets called; however I was unable to make this work (the method has overloads, so I was unable to figure out how I could easily wrap it).

I've tried looking for other extensions that might be testing things like this, but so far turned up nothing.

[HaaLeo]

I do not know if the built in mocha test-framework can do that. However you can do it with the spies of the jasmine framework. Because those enable you to check how often and with which args the window.showErrorMessage method was called:

// Arrange: set up the spy
const vscodeSpy = spyOn(vscode.window, 'showErrorMessage').and.callThrough();
const locateAction = "Locate SDK";
const downloadAction = "Download SDK";
let displayMessage = `Could not find a ${sdkType} SDK. ` +
	`Please ensure ${sdkType.toLowerCase()} is installed and in your PATH (you may need to restart).`;

// Act: Add your actual function call here
// something like testObject.showMessage();

// Assert: evaluate your spy
expect(vscodeSpy.calls.count()).toBe(1);
expect(vscodeSpy.calls.argsFor(0)[0]).toEqual(displayMessage);
expect(vscodeSpy.calls.argsFor(0)[1]).toEqual(locateAction);
expect(vscodeSpy.calls.argsFor(0)[1]).toEqual(downloadAction);

But to use the jasmine framework you need to write your own testRunner.

[bpasero]

@DanTup currently not possible unless extensions would get API to query the currently active when conditions, because when a toast shows up, I set the notificationToastsVisible context key.

@jrieken @alexandrudima is it possible from an extension to query the currently active context keys?

[jrieken]

@jrieken @alexandrudima is it possible from an extension to query the currently active context keys?

no

[bpasero]

Closing and extracting into #45566

[DanTup]

@bpasero @jrieken I was actually hoping for more than just testing the presence of the prompt, but checking what it is (for example, checking which SDK we're telling the user they don't have). Apparently Mocha has "spies" as described above, I could give that a shot and see if it lets me handle this (the solution would also be more generic, whereas the proposed solution probably only works for specific ones).

[bpasero]

@DanTup there is no access from extensions to anything that goes on in the renderer, so how would "spies" help? Unless you would spy on the extension host API calls maybe.

[DanTup]

@bpasero I haven't tried it yet and I'm not familiar with it, but I presume the spies mentioned above effectively wrap a function and log it's invocation. If my tests and my extension get the same vs.window.showInformationMessage and the test is able to "overwrite" it with the wrapped function, maybe that would work?

(I'm not familiar with the architecture, so it's possible this is nonsense, I'm just thinking out loud!)

[bpasero]

@DanTup got it, yeah as long as you are patching in the extension host process it should be fine.

[DanTup]

Looks like it kind works:

const spy = sinon.spy(vs.window, "showErrorMessage");
console.log(spy.called); // false
vs.window.showErrorMessage("Test");
console.log(spy.called); // true
// console.log(spy.arguments);
spy.restore();

It errors when trying to read arguments (TypeError: 'caller' and 'arguments' are restricted function properties and cannot be accessed in this context.) but I I don't believe that's a Code issue. So hopefully I can manage what I need :-)

[DanTup]

For anyone else looking to do this, using sinon I was able to both check the functions were called and also provide return values for things like showOpenDialog for testing. As noted above, I can't access .arguments but it doesn't seem required for really doing what I needed.

const showInputBox = sinon.stub(vs.window, "showInputBox");
showInputBox.resolves("my_test_flutter_proj");

const showOpenDialog = sinon.stub(vs.window, "showOpenDialog");
const tempFolder = getRandomTempFolder();
showOpenDialog.resolves([vs.Uri.file(tempFolder)]);

// Intercept executeCommand for openFolder so we don't spawn a new instance of Code!
const executeCommand = sinon.stub(vs.commands, "executeCommand");
const openFolder = executeCommand.withArgs("vscode.openFolder", sinon.match.any).resolves(null);
executeCommand.callThrough();

await vs.commands.executeCommand("flutter.createProject");

assert.ok(showInputBox.calledOnce);
assert.ok(showOpenDialog.calledOnce);
assert.ok(openFolder.calledOnce);
assert.ok(fs.existsSync(path.join(tempFolder, "my_test_flutter_proj", FLUTTER_CREATE_PROJECT_TRIGGER_FILE)));

showInputBox.restore();
showOpenDialog.restore();
executeCommand.restore();