Discussion: regex literals

[alrz]

Moved from dotnet/roslyn#5806

can be compiled into the resultant assembly, so an invalid pattern will be a compile-time error.

if(~"^(0[1-9]|1[012])[- \.](0[1-9]|[12][0-9]|3[01])[- \.](19|20)\d\d$".IsMatch( ... ))

It might be useful to make it implicitly verbatim to avoid double-escape in regular usage.

Also, we should be able to use it as a pattern against strings,

if (str is ~"\d+")

If language understand regex, we might be able to use string interpolations to produce parameterized regex patterns, e.g. ~$"a{{{count}}}" or if (str is ~$"{int number:\d+}").

[daniel-white]

• Is there a hard reason to have to look like a string? Could it look like an ECMAScript regex literal?

if (/^(0[1-9]|1[012])[- \/.](0[1-9]|[12][0-9]|3[01])[- \/.](19|20)\d\d$/.IsMatch( ... ))

• How would we specify any of the flags/options, especially case-insensitive? With the ECMAScript one, it would be nice to have the analog with i for case insensitive.
• Would these set the RegexOptions.Compiled implicitly?

[alrz]

@daniel-white

1. what are the advantages in doing so? string syntax is already being used for this and even R# provides syntax highlighting where it knows that a regex is intended.
2. we could use flag specifiers as part of the literal just like JS.
3. Yes.

[daniel-white]

@alrz i guess for me its a clarity issue because its not a "display" string a string that goes in logs, on the UI etc.. its code that executes.

[jnm2]

I like this just for the precompilation into the assembly.

[TonyValenti]

I like this alot. I would suggest using an 'R' prefix on the string to denote regexs though.

var MyPattern = R"[A-Za-Z0-9]+";

if(MyPattern.IsMatch(....){
    ....
}

Again, I'm a huge fan of making regexs be compile-time-compiled.

[TonyValenti]

Also, related to this, it would be awesome if named captures became properties or something like that so that I wouldn't have to use magic strings to find their values.

[CyrusNajmabadi]

Is there a reason an analyzer would not be sufficient here?

[CyrusNajmabadi]

For strings used directly in regex contexts, an analyzer could easily check it. For strings intended to be used for regexs later you could just have:

public static class RegexAnalyzer
{
     public static string R(string v) => v;
}

...

import static RegexAnalyzer;
...

R("myregexp");

And then have your analyzer check R("...") calls. That's effectively only two chars more than the R"..." approach.

[alrz]

@CyrusNajmabadi

The main advantage is the compile-time compilation, because even if you cache a Regex instance, it still needs to be compiled every time at runtime. I think Regex use a dynamic assembly to do that, this suggests to bring it out to the physical assembly that is going to be generated.

[CyrusNajmabadi]

The main advantage is the compile-time compilation

How would you propose that work?

[alrz]

@CyrusNajmabadi

AFAIK Regex already provides methods to compile to an assembly. Perhaps the compiler can pipe it to its own output. Though, the indirection might be not preferable, in that case we could take a page from RegexCompiler, in fact, it already directly emits opcodes to a dynamic assembly which can be ported to the compiler's code generator.

[yaakov-h]

Wouldn't that require that the compiler and the runtime both have the exact same implementation of a compiled regex?

[alrz]

@yaakov-h If we take RegexCompiler approach, they will have the exact same implementation. What should be different anyway?

[HaloFour]

It would mean that the C# spec would have to consume the entirety of the .NET-specific dialect of the regex spec. This is even assuming that the BCL code to dynamically generate assemblies from regex is something that could be reasonably consumed by Roslyn without some serious rewriting.

Having better analyzers to verify syntax would be a good thing. As would having an ahead-of-time compilation of regex, similar to sgen.exe. I don't know that it's worth completely embedding into the language, though.

[daniel-white]

Pardon my ignorance on the details on how the IL/CLR would work with something like this but I do see a benefit:

Would it be possible to make the references a compile-time constant like how strings can be? Is Regex even immutable or thread safe (if not would it be breaking to make that way)? That way we could use these literals in attributes. As an example

class Foo
{
    private const Regex myRegex = /[a-z]/i;

    [PatternValidator(myRegex)]
    public string Name { get; set; }
}

[HaloFour]

@CyrusNajmabadi

That would mean that the compiler/spec would have to track all of the variants of every possible paired piece of punctuation allowed? How far would that go?

1. “ and ”
2. « and »
3. „ and ”
4. 「 and 」

Would seem a bit overkill and tricky. Some cultures even invert their marks, e.g. French «Foo» v. Danish »Foo«. A quick poke at Wikipedia seems to indicate that such is not uncommon amongst the European nations.

This is probably why C++ includes the terminator within the string. It eliminates the need for the compiler to be concerned about any specific parsing of that character sequence. Could be an identifier, punctuation, whatever:

string raw_str=R"ABC«.*(This is the raw string content)ABC«.*";

No cute matching of brackets, though.

[ufcpp]

any punctuation?
⌈⌉⌊⌋❨❩❪❫❬❭❮❯❰❱❲❳❴❵⟅⟆⟦⟧⟨⟩⟪⟫⟬⟭⟮⟯⦃⦄⦅⦆⦇⦈⦉⦊⦋⦌⦍⦎⦏⦐⦑⦒⦓⦔⦕⦖⦗⦘⧘⧙⧚⧛⧼⧽⸢⸣⸤⸥⸦⸧⸨⸩〈〉《》「」『』【】〔〕〖〗〘〙〚〛〝〞

[HaloFour]

@ufcpp

My favorite would be Khmer: ៙ and ៚, cock's eye followed by cow's piss.

[CyrusNajmabadi]

That would mean that the compiler/spec would have to track all of the variants of every possible paired piece of punctuation allowed? How far would that go?

[CyrusNajmabadi]

This is probably why C++ ...

Pfshaw! We can do better than those guys!

No cute matching of brackets, though.

Pfshaw! We can do better than those guys!

Think big!

[CyrusNajmabadi]

FYI: I may still be carrying my new years 'exuberance' over to today... :D

[jnm2]

Apparently...

[daniel-white]

Can we close this thread and move the discussion on the new string literal syntax somewhere else? I’m only here for the regex support.

[CyrusNajmabadi]

@daniel-white Regex support is over at: dotnet/roslyn#23984
I don't think we need specific language support. THe language is already rich enough to give a way to write regexes. The only exception is that you need to double escape quotes "". If that's not good enough, then you need raw string literals, which is why that's being discussed now.

[orthoxerox]

Think big!

string veryRaw = @(╯°□°）╯︵ ┻━┻ ... ┬─┬ノ( º _ ºノ)?

[stepanbenes]

Please add some artistic value into the proposals.

string veryRaw = <*/((((><|

Content could go before the tail.

[madelson]

A while back I wrote a Fody build plugin that essentially implements this behavior: https://github.com/madelson/PrecompiledRegex.Fody. May be of interest to those who would find this language feature useful.

My worry with Regex literals is that there are just too many options I often want to specify (e. g. ExplicitCapture, IgnoreCase, Timeout, Multi/Single line modes) and it would be hard to encode those into the literal in a way that retained the readability of what we have today.

Since having this library available to me, I've also noticed that it has been less helpful in my own projects that I would have thought: too often I find I'm building my patterns dynamically rather that statically. While in theory a regex literal syntax could allow for dynamic insertions, unless those are always escaped this would turn off compile-time validation, strong-typing of capture groups, etc.

Another problem I've encountered is that compiled regex IL can get very bloated for complex patterns, in some cases causing noticeable startup stalls due to what I can only assume to be JIT compilation of the massive methods. Presumably this could be addressed with changes to the regex compilation logic.

[Pzixel]

Regex shouldn't be builtins. There are plenty of examples when regex evolved and with time one libraries were replaced with others. For example in C++ it was runtime regexes but then they came to compile-time regexes. Rust doesn't even have them at std, they are shipped as 3rd party library.

There is literally no reason to add this into the language. It's somewhat trashed already.

[GSPP]

What fraction of all C# lines of code deal with instantiating a regex? It is a very small fraction. Is it really worth adding this to the language? I cannot imagine it is.

In my opinion, managing regular expressions is not a significant pain point at the moment. Could be better but it's fine.

[IS4Code]

There is some advantage to this proposal – being able to cache the regex object constructed from the pattern. However, this is also possible with my static expressions proposal, see #3929:

if((static new Regex("...", RegexOptions.Compiled)).IsMatch( ... ))

The instance should be cached as a static field and so will not have to be created each time. This works well with other regex engines and similar singleton objects. Most code analyzers will highlight issues in the pattern anyway, and if that slips by, the type initializer should catch the rest.

[TonyValenti]

In .Net 7 we're getting a regex source generator which will precompile a regex into c#.