Problem with test dependency

[alekdavisintel]

I just noticed a weird issue. I have a number of CRUD tests with dependencies: get some values necessary for item creation, then check if the item with the same name already exists, if so, delete the item, then create the item, do a couple of gets, do a patch, another get, and finally delete. It seems to work fine, if I have a pending account that I need to clean up, but if there is no cleanup account, then I skip the cleanup deletion step and even though the rest of the tests execute as expected (based on the logs), the icons of all of the subsequent tests show as non-tried:

In my particular case, only the Clean up existing non-employee acount test must show as skipped. Other tests must show as either successful or failed (a couple of them failed). Here are the entries from the console log:

INFO: ✓ status == 200
INFO: ✓ sourcesJson isArray
INFO: ✓ sourcesJson.length == 1
INFO: ✓ sourcesJson[0].id isString
INFO: ✓ status == 200
INFO: ✓ status == 200
INFO: ✓ managersJson isArray
INFO: ✓ managersJson.length == 1
INFO: ✓ managersJson[0].id isString
INFO: ✓ status == 200
INFO: ✓ accountJson.id isString
INFO: ✓ accountJson.accountName == Alek.Davis.Test2
INFO: ✓ status == 200
INFO: ✓ nonEmployeeIdentitiesJson isArray
INFO: ✖ nonEmployeeIdentitiesJson.length > 0 (AssertionError [ERR_ASSERTION]: nonEmployeeIdentitiesJson.length (0) > 0 - c:\Users\MYUSERNAME\.vscode\extensions\anweber.vscode-httpyac-6.14.0\dist\extension.js:192:46875)
INFO: ✖ nonEmployeeIdentitiesJson[0].id isString (TypeError: Cannot read properties of undefined (reading 'id') - Object.userJS (c:\PROJECTS\POC\Tests\Identities\identities-CRUD.http:115:49)
INFO: ✖ nonEmployeeIdentitiesJson[0].name == {{accountJson.accountName}} (TypeError: Cannot read properties of undefined (reading 'name') - Object.userJS (c:\PROJECTS\POC\Tests\Identities\identities-CRUD.http:116:49)
INFO: ✓ status == 200
INFO: ✓ validatedAccountsJson isArray
INFO: ✖ validatedAccountsJson.length > 0 (AssertionError [ERR_ASSERTION]: validatedAccountsJson.length (0) > 0 - c:\Users\MYUSERNAME\.vscode\extensions\anweber.vscode-httpyac-6.14.0\dist\extension.js:192:46875)
INFO: ✖ validatedAccountsJson[0].id isString (TypeError: Cannot read properties of undefined (reading 'id') - Object.userJS (c:\PROJECTS\POC\Tests\Identities\identities-CRUD.http:134:45)
INFO: ✖ validatedAccountsJson[0].name == {{accountJson.accountName}} (TypeError: Cannot read properties of undefined (reading 'name') - Object.userJS (c:\PROJECTS\POC\Tests\Identities\identities-CRUD.http:135:45)
INFO: ✓ status == 200
INFO: ✓ patchedAccountJson.firstName == Alexander
INFO: ✓ patchedAccountJson.email == alek.davis.test2+updated@example.com
INFO: ✓ patchedAccountJson.phone == +1 9166666666
INFO: ✓ status == 200
INFO: ✓ updatedAccountJson.firstName == Alexander
INFO: ✓ updatedAccountJson.email == alek.davis.test2+updated@example.com
INFO: ✓ updatedAccountJson.phone == +1 9166666666
INFO: ✓ status == 204

[alekdavisintel]

I did a bit more testing and can confirm that there are a few issues with the VSCode extension's handling of the cancelled request. Scenario: there are 3 tests in an http file, the first of which should be cancelled, but the other two should run normally. Here is what happens:

• If I run an individual uncancelled test (by pressing the green arrow icon on the left side of the test region), the request gets made but the icon indicates that the test was skipped (the icon issue is consistent between the project tree view and test results for this and the other cases).
• If I attemt to run tests using the Send Selected (and selecting all tests) or Send All command, all of tests get skipped.
• If I run tests by executing the folder from the Testing view, the two uncancelled tests do run, but the icons show them as skipped.

I suspect this is an issue with the VSCode extension (and probably more than one).

[AnWeber]

@alekdavisintel I introduced a bug with the last release. I underestimated the scope of the $cancel. Unfortunately, it survives the context and this causes subsequent problems. All testRuns after this run, will be interpreted as skipped.. I reverted the change with the commit.
But apparently there are still a few errors in the context of disabled or $cancel. CLI skips all regions after first disabled httpregion. The concept I have there does not work. But I realize that it doesn`t work the fast way and I should think about it.

[alekdavisintel]

I have not tested CLI, yet, but from what I see in VSCode, it works fine if I run tests from the Testing tree view (either by file or by folder). Actually, the @disabled condition does not work at all, but it did not work before the last update, so no change here, but the $cancel logic works fine (again, only when I use the Testing view). The only problem is the icons after the test run which should reflect the completed (successfull or failed, but not skipped) test results (they show all as skipped). But the test do run.

I do not know the difference between the test runs between the invocation of the Testing view and the Code Lens Send All and Send Selected options and why the behavior is different. Logically, if I test a file from the Testing view or run the Send All/Send Selected (with all tests in the file selected), I'd expect the behavior to be the same, but it's not. Maybe the logic of different test invocation options is different.

[AnWeber]

Would you please test the new release? I would have added several optimizations here on how to deal with skipped and errors. This should solve your problem.

[alekdavisintel]

I will do more extensive testing next week (assuming I still have a job, don't know if you followed the tech news), but a quick test looks nice, at least I see the correct icons upon the test run completion (when I run it from the Testing window). A really minor thing (and I'm not sure if it is intentional), but in the test results tree the order of tests seems wrong unless you meant to group cancelled tests at the end of the run intentionally.

Testing tree view:

Test results tree view:

There seems to be a few issues when I run tests using the send all codelens option:

• After a cleared test results and used send all the Test Results tab did not appear.
• The httpyac - Request window is blank and in the httpyac - Log window, I get multiple errors:

2024-08-04 17:48:01.261 [error] symbol for "" (Line 3-3) could not be provided name must not be falsy
Error: name must not be falsy
	at f.validate (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:152:18116)
	at new f (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:152:18411)
	at zN.toDocumentSymbol (c:\Users\username\.vscode\extensions\anweber.vscode-httpyac-6.15.0\dist\extension.js:347:37855)
	at zN.toDocumentSymbol (c:\Users\username\.vscode\extensions\anweber.vscode-httpyac-6.15.0\dist\extension.js:347:37992)
	at zN.toDocumentSymbol (c:\Users\username\.vscode\extensions\anweber.vscode-httpyac-6.15.0\dist\extension.js:347:37992)
	at zN.provideDocumentSymbols (c:\Users\username\.vscode\extensions\anweber.vscode-httpyac-6.15.0\dist\extension.js:347:37708)
	at async f.provideDocumentSymbols (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:154:98908)

If I run a single test using the green arrow on the left margin of the source file, I get the same error but the Test Results tab appears and I get the request log output. So something fishy is going on here, but execution from the Testing menu seems to work fine. I am only using the $cancel variable. Not sure if you want me to try the @disabled meta variable (I do not use them now except permanently disabled tests, but I will try if you made any changes to it).

Many thanks for the improvements.

P.S. Love the output that now goes into in the Test Results tab. Very helpful.

[AnWeber]

@alekdavisintel I haven't heard the news yet. Stupid bullshit. I hope it turns out well for you.

[alekdavisintel]

I tested the @disabled metatag handling just in case, and looks like it has not been fixed, yet. Not a big deal, since the same functionality can be accomplished via $cancel.

[AnWeber]

After a cleared test results and used send all the Test Results tab did not appear.

send all command is not related to test results. This command does not trigger a test run and that is why the test tab is not displayed. The command is also older and was created before the Test Explorer integration of vscode.

The error is generated when building the DocumentSymbol for vscode (Outline View) and has more to do with the parsing of the symbols but nothing to do with the execution. It would be interesting to see what the file looks like, as each symbol must actually have a name, but this is apparently not the case.

update: maybe you use outputRedirection or an empty header value. There is a typo

P.S. Love the output that now goes into in the Test Results tab. Very helpful.

Yes, I skimmed through your Pain Points. That was a quick win, but I haven't finished thinking about it yet.

[alekdavisintel]

Is there a description of the codelens commands, like send all and send selected? Are they not supposed to be used? I must have wrongly assumed that send all would do the same thing as running the tests in the selected file from the Testing view. If they are not doing what is expected, is there a way to hide or remove them? What about the send and send repeat commands? Do they trigger tests?

I did not get the point about the parsing error. Is this something on my end? It did not happen before the last release. And I do not use output redirection. Do you need more info? Not sure about the empty header values, since I do not mess with headers. Should I check something specific?

[alekdavis]

Noticed one more thing after additional testing: if a test dependency fails to run for some reason the test that uses a dependency is always marked as successful in the tree (with the green checkbox) even if it does not run. E.g. test X depends on test A, and the OAuth settings are not configured correctly for test A, so test A is broken, even though test X does not run, it is marked as successful:

You can see this in the repo I created: https://github.com/alekdavis/httpyac-extras

[AnWeber]

That's right, I overlooked this case. I'll check the result now and add Skipped if necessary

[alekdavis]

I am still trying to figure out what makes the response window close and then not appear and it looks like this only happens when I run a file or folder test (which has more than one requests) from the Testing view. Here is a demo (from my https://github.com/alekdavis/httpyac-extras repo):

httpYac-ResponseTabIssue.mp4