[BUG] Serial + Retry : Failed test is marked as flaky even if it never passes

[javamaniac]

System info

• Playwright Version: [v1.38.0]
• Operating System: [All, Windows 11, Ubuntu 20, macOS 13.2, etc.]
• Browser: [All, Chromium, Firefox, WebKit]

Source code

import {expect, test} from '@playwright/test';

test.describe.serial("Tests", () => {
  // Flaky test
  test("First test", () => {
    expect(Math.random()).toBeGreaterThan(0.3); // some flaky step
  });

  test("Second test", () => {
    expect(true).toEqual(false);
  });
});

Description
The second test is marked as flaky if tests are in describe.serial block with "retries: 1" settled in tests config file.

Steps
Execute the test until:

• The second test is executed at least once and the first test failed on the last retry

Expected

  1 failed
    [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (Failed) ──────────
  1 passed (648ms)

Actual
If the second test is executed at least once and the first test failed on the last retry:

Running 2 tests using 1 worker

  ✓  1 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (21ms)
  ✘  2 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (11ms)
  ✘  3 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (retry #1) (23ms)
  -  4 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (retry #1)
...
  2 flaky
    [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test ────────────
    [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test ──────────

If the second test is executed on the last retry:

Running 2 tests using 1 worker

  ✓  1 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (21ms)
  ✘  2 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (11ms)
  ✓  3 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (retry #1) (21ms)
  ✘  4 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (retry #1) (11ms)
...
  1 failed
    [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test ──────────
  1 passed (648ms)

[javamaniac]

❤️