Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests failing locally #598

Closed
Will-Sommers opened this issue Mar 9, 2022 · 15 comments · Fixed by #731
Closed

Tests failing locally #598

Will-Sommers opened this issue Mar 9, 2022 · 15 comments · Fixed by #731
Labels
code quality Improvements to code quality

Comments

@Will-Sommers
Copy link
Collaborator

While working on #597 I had erroneously reported failing tests locally. The suite passed on the PR build action but locally I am having failing tests. I also checked on the main branch and also have similar failures. The test run hasn't finished yet so I'm not sure if they are deterministically failing but I'll update the issue with more information. Here's an example locally failing test from the PR:

  22) recorded test cases
       /Users/.../recorded/languages/typescript/ifElseWrapThis:

      AssertionError [ERR_ASSERTION]: Unexpected final state
      + expected - actual

       {
      -  "documentContents": "if () {\n  const foo = \"hello\";\n} else {\n  \n}"
      +  "documentContents": "if () {\n    const foo = \"hello\";\n} else {\n    \n}"
         "selections": [
           {
             "active": {
               "character": 4

Also, noted in Slack. I currently have to quit VSCode to run the suite. Probably a separate issue but worth noting.

@pokey
Copy link
Member

pokey commented Mar 9, 2022

@wenkokke was having this issue too. Do you remember if we got to the bottom of it?

@wenkokke
Copy link

We never managed to fix it, I simply ignored it and you didn't have this issue locally.

@pokey
Copy link
Member

pokey commented Mar 11, 2022

We never managed to fix it, I simply ignored it and you didn't have this issue locally.

Right ok that was my recollection but was hoping my recollection was wrong 😅

I believe this one has to do with VSCode space insertion. When you define a snippet in VSCode, it reinterprets tab characters in the snippet body as spaces according to your configuration for the given file / language. So I believe it has something to do with your indentation settings. We somehow need to get VSCode to ignore your custom indentation settings I believe

@Will-Sommers
Copy link
Collaborator Author

Hrm, got it. I'll think about this a bit. I wonder if it would make sense to give local developers the ability to run tests akin to the GH actions, in a local container.

@pokey
Copy link
Member

pokey commented Mar 11, 2022

oh that's a cool idea. I wonder if that would fix the insiders' thing too

@Will-Sommers
Copy link
Collaborator Author

This looks pretty cool, although new, and might obviate the need for a separate and possibly diverging config. https://github.com/nektos/act

@pokey
Copy link
Member

pokey commented Mar 11, 2022

Funny I just ran across that today for the first time

Interesting idea, and might simplify things, tho seems a bit heavy handed, as I believe it installs VSCode in the container every time. But prob not a showstopper; might not be too slow and could potentially be cached

Also wonder how it handles matrix, as our CI uses a matrix to run tests on 3 platforms

@wenkokke
Copy link

That probably means that it would download three separate images, and run the test three times… not a huge problem overall I suppose, but it might be nice to be able to limit it to a single platform only.

@pokey
Copy link
Member

pokey commented Mar 12, 2022

Windows and macOS based platforms are currently unsupported and won't work (see issue nektos/act#97)
- https://github.com/nektos/act#runners

So presumably it will only run Linux? Either that or try to run Mac and Windows and fail? 😅

@Will-Sommers
Copy link
Collaborator Author

Let me try and just run it with a docker. I'll look at checkout@v2 and the yarn action and replicate the set up steps.

@pokey pokey added the code quality Improvements to code quality label Mar 13, 2022
@pokey
Copy link
Member

pokey commented Jun 3, 2022

Is this one still an issue?

@Will-Sommers
Copy link
Collaborator Author

Will-Sommers commented Jun 3, 2022

Yes, it is still an issue, even after init sandbox

On main:

  1534 passing (46s)
  56 pending
  26 failing

Failing tests:

src/test/suite/fixtures/recorded/actions/indentVest:
&&
all of the if/else/ifElse wrap calls

@pokey
Copy link
Member

pokey commented Jun 3, 2022

Ok let's see if #672 helps once it's merged. I don't think it will help the wrap tests, but should help some local failures

@Will-Sommers
Copy link
Collaborator Author

My "tab size" was 2, I moved it to 4 and they're all passing now. I guess we should add notes within the README.md and maybe something

  1560 passing (42s)
  56 pending

image

image

@Will-Sommers
Copy link
Collaborator Author

Green run for the first time locally!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code quality Improvements to code quality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants