RangeError: Maximum call stack size exceeded when toMatchObject with array circular references

[hi-ogawa]

Describe the bug

Similar to #5533, but the cause of the issue looks slightly different, so I'm creating a separate issue.

The failing case is taken from Jest's issue:

• [Bug]: Stackoverflow in toMatchObject when matching w/ recursive object jestjs/jest#14734

import { test, expect } from 'vitest';

function gen() {
  const a: any = {
    v: 1,
  };
  const c1: any = {
    ref: [],
  };
  c1.ref.push(c1);
  a.ref = c1;
  return a;
}

// success
test('toEqual', () => {
  expect(gen()).toEqual(gen()); 
});

// RangeError: Maximum call stack size exceeded
test('toMatchObject', () => {
  expect(gen()).toMatchObject(gen()); 
});

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-ysdtzj?file=test%2Frepro.test.ts

System Info

[stackblitz]

  System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 18.18.0 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.2.3 - /usr/local/bin/npm
    pnpm: 8.15.3 - /usr/local/bin/pnpm
  npmPackages:
    @vitest/ui: latest => 1.5.0 
    vite: latest => 5.2.8 
    vitest: latest => 1.5.0

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[hi-ogawa]

In some cases, chai's utils.getMessage after jestEquals is also getting into loop:

vitest/packages/expect/src/jest-expect.ts

Lines 164 to 176 in 470f561

	const pass = jestEquals(actual, expected, [...customTesters, iterableEquality, subsetEquality])
	const isNot = utils.flag(this, 'negate') as boolean
	const { subset: actualSubset, stripped } = getObjectSubset(actual, expected)
	const msg = utils.getMessage(
	this,
	[
	pass,
	'expected #{this} to match object #{exp}',
	'expected #{this} to not match object #{exp}',
	expected,
	actualSubset,
	],
	)