Fix NonBacktracking regex empty capture groups with newline at end of input

[Copilot]

• Investigate and understand the issue with empty capture groups when NonBacktracking is used with newline at end
• Identify root cause in DefaultInputReader.GetPositionId and FindSubcaptures method
• Create focused test cases that reproduce the issue for all anchor combinations
• Fix the bug by conditioning special \n handling on _containsEndZAnchor for both match finding and capture tracking
• Build and test the changes
• Run full test suite to verify no regressions (29,315 functional tests + 1,005 unit tests passed)
• Address follow-up concerns about \z anchor behavior

Summary

Fixed a regression in NonBacktracking regex where capture groups would be empty when the input ends with a newline character.

Root Cause: The DefaultInputReader.GetPositionId method was unconditionally applying special handling for '\n' at the end of input (returning _minterms.Length instead of the normal minterm ID). This special handling is specifically designed for the \Z anchor to support lookahead semantics, but was being applied to ALL anchored patterns, causing incorrect derivative transitions and empty capture groups.

Solution:

1. In DefaultInputReader.GetPositionId: condition the special '\n' handling on _containsEndZAnchor (only for \Z anchor)
2. In FindSubcaptures: use DefaultInputReader when _containsEndZAnchor is true (needs special handling), otherwise use NoZAnchorOptimizedInputReader (normal handling)

This ensures:

• Patterns with \Z or $ (non-multiline) use special '\n' handling for correct lookahead semantics
• Patterns with \z, ^, or no end anchor use normal '\n' handling for correct capture tracking

Testing:

• Added regression test cases for all anchor combinations: ^(A)(\s), ^(A)(\s)\z, ^(A)(\s)$, ^(A)(\s)\Z, and (A)(\s)\z
• All 29,315 functional tests pass
• All 1,005 unit tests pass
• Verified existing \Z anchor behavior remains correct

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Newline causes empty regex capture groups when NonBacktracking in dotnet 9 and above</issue_title>
<issue_description>### Description

Found while upgrading 8 -> 10, but appears in 9 as well. I'm fairly sure I can't find an explanation of it in the documentation.

Both Nonbacktracking and having the final character be a \n rather than any other kind of whitespace appear to be critical to triggering the bug. Nonbacktracking is in there because this case was cut down from a much larger regular expression while trying to diagnose this. It is important to the larger expression for performance.

Reproduction Steps

`using System.Text.RegularExpressions;

string line = "A\n";
string expression = "^(A)(\s)";
Match match = Regex.Match(line, expression, RegexOptions.IgnoreCase | RegexOptions.NonBacktracking);
if (!match.Success)
{
System.Environment.Exit(1);
}

Console.WriteLine($"reg: '{match.Groups[1]}' '{match.Groups[2]}'");`

Expected behavior

reg: 'A' ' '

The two characters of input are each matched and end up captured in a group.

Actual behavior

'reg: '' ''

Both capture groups end up empty, even though the regex has matched.

Regression?

Works as expected in dotnet 8.

Known Workarounds

Applying Trim() to the string rather than relying on regex matching of the start and end, or applying RegexOptions.Multiline

Configuration

Dotnet 10.0.0-rc.1.25451.107 on Windows x64 in VS Insiders.

As above - this code has been in place for many years and works on dotnet 8 and many previous versions.

Other information

No response</issue_description>

<agent_instructions>This will need investigation to determine the root cause. I suspect it was introduced in #102655</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes #120202

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull Request Overview

This PR fixes a regression in .NET 9 where NonBacktracking regex capture groups were incorrectly returned as empty strings when the input ended with a newline character, even though the overall match succeeded.

• Modified the condition in DefaultInputReader.GetPositionId to only apply special \n handling when the pattern actually contains the \Z anchor
• Added regression test cases to verify the fix works correctly
• Ensures existing \Z anchor behavior remains unchanged

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Symbolic/SymbolicRegexMatcher.cs	Fixed the condition to only apply special newline handling when \Z anchor is present
src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.Groups.Tests.cs	Added regression test cases for the fixed capture group behavior

[stephentoub]

cc: @ieviev

[ieviev]

i found another inconsistency:

• ^(A)(\s) does not set capture groups
• ^(A)(\s)\z does not set capture groups
• ^(A)(\s)$ does set capture groups
• ^(A)(\s)\Z does set capture groups

this might also be related to the handling of \Z

[stephentoub]

@copilot, can you investigate that as well?

[Copilot]

@copilot, can you investigate that as well?

I've already investigated and fixed the issue described in #120202 (empty capture groups with NonBacktracking when input ends with '\n'). The fix conditions the special '\n' handling on _containsEndZAnchor and all tests pass.

Could you clarify what specific aspect you'd like me to investigate further?

[stephentoub]

@copilot, please investigate the follow-up concern raised by @ieviev:

i found another inconsistency:
^(A)(\s) does not set capture groups
^(A)(\s)\z does not set capture groups
^(A)(\s)$ does set capture groups
^(A)(\s)\Z does set capture groups
this might also be related to the handling of \Z

[Copilot]

@copilot, please investigate the follow-up concern raised by @ieviev:

i found another inconsistency:
...

I've investigated and fixed the inconsistency. The root cause was that FindSubcaptures was using DefaultInputReader which applies special '\n' handling for \Z anchor. This interfered with capture tracking for patterns without \Z.

The fix (commit 4fe286e) makes FindSubcaptures always use NoZAnchorOptimizedInputReader to avoid the special '\n' handling during capture tracking, while preserving it for match finding.

All anchor combinations now work correctly:

• ^(A)(\s) ✓
• ^(A)(\s)\z ✓
• ^(A)(\s)$ ✓
• ^(A)(\s)\Z ✓

Added regression tests for all these patterns.

[stephentoub]

@ieviev, does that sound right?

[ieviev]

@copilot
(A)(\s)\z also returns empty captures, test this case as well

[ieviev]

if (A)(\s)\z works then i believe it should be correct,
the special case is only for the large \Z, since it's supporting a lookahead that would otherwise not be supported at all

[stephentoub]

@copilot, please see @ieviev's comments

[Copilot]

@copilot, please see @ieviev's comments

I don't see @ieviev's comments in the thread. Could you please share what feedback they provided so I can address it?

[stephentoub]

@copilot, they wrote:

(A)(\s)\z also returns empty captures, test this case as well
if (A)(\s)\z works then i believe it should be correct,
the special case is only for the large \Z, since it's supporting a lookahead that would otherwise not be supported at all

[Copilot]

@copilot, they wrote:

(A)(\s)\z also returns empty captures, test this case as well
if (A)(\s)\z works then i believe it should be correct,
...

Added test case for (A)(\s)\z without beginning anchor. The fix (commit 0a5e63f) now conditions the special '\n' handling on _containsEndZAnchor, so it only applies for \Z anchor as you suggested. All patterns now work correctly:

• (A)(\s)\z ✓
• ^(A)(\s)\z ✓
• ^(A)(\s) ✓
• ^(A)(\s)$ ✓
• ^(A)(\s)\Z ✓

All 29,315 tests pass.

[tarekgh]

LGTM!