Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate flaky unit tests failing intermittently after vscode source refresh #23509

Closed
kburtram opened this issue Jun 28, 2023 · 11 comments · Fixed by #23770 or #23913
Closed

Investigate flaky unit tests failing intermittently after vscode source refresh #23509

kburtram opened this issue Jun 28, 2023 · 11 comments · Fixed by #23770 or #23913

Comments

@kburtram
Copy link
Member

kburtram commented Jun 28, 2023

There are a handful of unit tests failing intermittently in the canary builds that need to be investigated. I'll use this issue to keep a list of these tests and provide status on fixing, disabling, etc.

Initial list of tests to investigate:

TerminalLinkManager
    getLinks and open recent link
      ✔ should return web links in order
      1) should return web links in order

DefaultConfiguration
    ✔ configuration default overrides are read from environment
    1) configuration default overrides are read from environment

Getting Started Markdown Renderer
    ✔ renders theme picker markdown with images
    1) renders theme picker markdown with images

A couple other ones...
1) Buffer Content Tracker
       should cap the size of the cached lines, removing old lines in favor of new lines:

 2) Buffer Content Tracker
       should cap the size of the cached lines, removing old lines in favor of new lines:
@kburtram
Copy link
Member Author

I've tried to disable the suites I'm seeing failing in the canary builds to understand if this xterm issue is isolated to a handful suites, and to try to stabilize the canary & PR builds. #23535. This doesn't seem like a good long-term fix so will continue to investigate after looking at some other issues.

@kburtram
Copy link
Member Author

This PR disables an additional suite for the same issue #23539.

@kburtram
Copy link
Member Author

This is one more suite impacted by this issue #23545.

@kburtram
Copy link
Member Author

One more test suite #23550.

@kburtram
Copy link
Member Author

One more #23557

@kburtram
Copy link
Member Author

The xterm glyph issue keeps popping up during other suites, so I'm going to stop disabling them. Though I will keep the above mentioned one disabled while investigating the builds are significantly more stable w/o them.

@kburtram
Copy link
Member Author

kburtram commented Jun 30, 2023

Additional suites that failed in canary yesterday #23589

@SakshiS-harma
Copy link
Contributor

SakshiS-harma commented Jul 9, 2023

A few more tests with similar error:

  • PreferencesService
    ✔ options are preserved when calling openEditor

    1. options are preserved when calling openEditor
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • CredentialsService - web
    ✔ Gets null for no service or a different service

    1. Gets null for no service or a different service
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • CredentialsService - web
    ✔ Delete removes the value

    1. Delete removes the value
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • CredentialsService - web
    ✔ Gets correct value for credential

    1. Gets correct value for credential
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ errors cases - parse errors

    1. errors cases - parse errors
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ remove a user keybinding

    1. remove a user keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ errors cases - did not find an array

    1. errors cases - did not find an array
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ edit a default keybinding to an empty array

    1. edit a default keybinding to an empty array
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ errors cases - dirty

    1. errors cases - dirty
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ add a new default keybinding

    1. add a new default keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ add a new keybinding to unassigned keybinding

    1. add a new keybinding to unassigned keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ edit an user keybinding

    1. edit an user keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
      KeybindingsEditing
      ✔ remove a default keybinding
    2. remove a default keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ add a new default keybinding using edit

    1. add a new default keybinding using edit
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ edit a default keybinding to an empty file

    1. edit a default keybinding to an empty file
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ edit a default keybinding in an existing array

    1. edit a default keybinding in an existing array
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ errors cases - dirty

    1. errors cases - dirty
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ add another keybinding

    1. add another keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ remove a user keybinding

    1. remove a user keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • KeybindingsEditing
    ✔ reset an edited keybinding

    1. reset an edited keybinding
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • Workbench - TerminalInstance
    TerminalLabelComputer
    ✔ should resolve to "" when the template variables are empty
    1) should resolve to "" when the template variables are empty
    Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')

  • FileDialogService
    ✔ Remote - open/save workspaces availableFilesystems

    1. Remote - open/save workspaces availableFilesystems
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • FileDialogService
    ✔ Virtual - open/save workspaces availableFilesystems

    1. Virtual - open/save workspaces availableFilesystems
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • FileDialogService
    ✔ Local - open/save workspaces availableFilesystems

    1. Local - open/save workspaces availableFilesystems
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • ExtensionStorageMigration
    ✔ migrate extension storage when does not exist

    1. migrate extension storage when does not exist
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')
  • ExtensionStorageMigration
    ✔ migrate extension storage

    1. migrate extension storage
      Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph')

@Charles-Gagnon
Copy link
Contributor

I chatted with Daniel from the VS Code team (owns the xterm stuff there) and his suggestion for a temporary fix would be to set up the tests to use the DOM renderer instead like they do for their smoke tests : https://github.com/microsoft/vscode/blob/0c3fb42a1e31f63ec7f17906a19bbcebdbe8bb3e/test/smoke/src/areas/terminal/terminal-helpers.ts#L15-L17

I'll try to send out for a PR for that this week, but if anyone else had time feel free to take this up 😄

@kisantia
Copy link
Contributor

There have still been test failures in the canary builds after #23770 was merged. @lewis-sanchez can you please take another look and disable these if needed?
Tests:

  • Buffer Content Tracker should cap the size of the cached lines, removing old lines in favor of new lines
    Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph') (/__w/1/s/node_modules/xterm-addon-canvas/lib/xterm-addon-canvas.js:1)
  • Buffer Content Tracker should refresh viewport with full scrollback
    Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph') (/__w/1/s/node_modules/xterm-addon-canvas/lib/xterm-addon-canvas.js:1)
  • DefaultConfiguration configuration default overrides are read from cache
    Error: Uncaught TypeError: Cannot read properties of undefined (reading 'getRasterizedGlyph') (/__w/1/s/node_modules/xterm-addon-canvas/lib/xterm-addon-canvas.js:1)

@Charles-Gagnon
Copy link
Contributor

Before disabling, please see if these tests are using the config that you added in the previous PR. If that test config service isn't actually used by every test by default then we may need to find another solution to ensure that every test picks up the config.

If it is being used then we should probably revert the change - since it doesn't seem to actually solve the problem - and investigate other options (such as fixing xterm itself).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment