Fix git core compatibility harness baseline handling

[svarlamov]

Motivation

• Ensure the Git core compatibility harness is runnable in CI and locally by building the upstream Git checkout before running tests.
• Avoid reporting normal test non-zero exits as harness errors and instead only flag true harness/parsing issues.
• Record the current observed failures as a baseline whitelist so CI only fails on new regressions.

Description

• Add ensure_git_build() to tests/git-compat/run-core-tests.py to run make in the cloned Git repo when GIT-BUILD-OPTIONS is missing.
• Set a default test hash via GIT_TEST_DEFAULT_HASH=sha1 in the harness environment so tests that expect SHA-1 run correctly.
• Parse and report only TAP/parse errors as harness issues (refined regex and parse_summary_issues()), while continuing to parse test failures for whitelist comparison.
• Update tests/git-compat/whitelist.csv with the current baseline of failing tests so the harness only fails for new, unexpected regressions.

Testing

• Built the project with cargo build --release --bin git-ai which completed successfully.
• Ran the harness with python3 tests/git-compat/run-core-tests.py, which cloned and built upstream Git, ran the selected prove test subset, and returned success because all remaining failures were applied to the updated whitelist.
• The harness now exits 0 when tests are either passing or explicitly whitelisted, and returns non-zero when parse/harness errors are detected.

---

Codex Task

---

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 6 additional findings.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 10 additional findings in Devin Review.

[devin-ai-integration]

🟡 Safety check not failures bypassed by empty-list entries from tests with Wstat but no individual failures

The fallback safety check at line 286 (if exit_code != 0 and not failures) is intended to catch cases where prove exits non-zero but no failure information was parsed. However, parse_failures creates entries with empty sets for any test matching the header_re pattern (Wstat line) even when no Failed tests: line follows—for example, a test script that crashes before running any assertions or exits non-zero without individual test failures.

Root Cause and Impact

In parse_failures at tests/git-compat/run-core-tests.py:127-129, any test matching the header regex gets a dict entry via failures.setdefault(current, set()), regardless of whether a Failed tests: line follows. When converted to the return value at line 153, this produces entries like {"t0000-basic.sh": []} (empty list).

An empty list is falsy, but a dict containing such entries is truthy:

failures = {"t0000-basic.sh": []}  # bool(failures) == True

At line 286, not failures evaluates to False, so the safety check is skipped. Meanwhile, apply_whitelist at line 269 produces an empty unexpected dict (no indices to filter). With no summary_issues either, the harness falls through to line 290 and returns 0 (success), even though prove exited non-zero for an unparsed reason.

Impact: The harness silently reports success when a test script crashes or has a non-zero Wstat without individual test failures, defeating the purpose of the safety check.

Suggested change

	if exit_code != 0 and not failures:
	print("\n[!] prove exited non-zero but no failures were parsed. Please investigate the output above.")
	return exit_code
	has_parsed_failures = any(indices for indices in failures.values())
	if exit_code != 0 and not has_parsed_failures:
	print("\n[!] prove exited non-zero but no failures were parsed. Please investigate the output above.")
	return exit_code

---

Was this helpful? React with 👍 or 👎 to provide feedback.