Proposal: Support "raw" string literals

[HaloFour]

Copied from dotnet/roslyn#2239

Problem:

Verbatim string literals are often used to copy in chunks of text such as SQL or XML. It's not uncommon for this text to contain embedded double-quotes, which then must be doubled-up in C# in order to be escaped. This makes it obnoxious if you have to copy and paste the text back and forth since it would have to be fixed each time.

string xml = @"<?xml version=""1.0""?>
<root>
    <item attr=""foo"" />
</root>";

Solution:

By supporting a custom notation for starting and ending a string literal it enables evaluating the double quotes as part of the string that do not have to be escaped. The programmer would use a pattern at the beginning of the string which then must be matched to denote the end of the string.

Proposed syntax would be based on verbatim string syntax:

raw-string-literal:
@(" raw-string-literal-characters ^opt " )
@( identifier " raw-string-literal-characters ^opt " identifier )

raw-string-literal-characters:
Any characters except " identifer ^opt )

The @ would be immediately followed by an open-parenthesis ((). Following that is an optional delimiter which is any legal C# identifier and then a double-quote. Next would be the raw string contents which would be interpreted exactly as they appear in the file. The terminator of the string literal would be a double quote immediately followed by the optional delimiter and a close-parenthesis ()).

The following example is a raw string without a custom delimiter. It is terminated simply by the sequence "). Double quotes that appear within the string are interpreted as a part of the string as long as they are not followed by a close-parenthesis character ()).

string xml = @("<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>");

The following example is a raw string with a custom delimiter. As the custom delimiter is specified as "foo" that sets the terminating sequence to be "foo). Any other characters are interpreted as a part of the string literal, including any portion of the terminating sequence. Note that the term "foo" appears within the string after a double-quote, but since it is not also followed by a close-parenthesis ()) it is not considered the terminating sequence of the literal.

In this example the custom delimiter is "foo". Note that the attribute value is not considered as the end of the string. However, if it was followed by a parenthesis it would be and a different custom delimiter might have to be chosen.

string xml = @(foo"<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>"foo);

Here are examples of raw strings containing the syntax for other raw strings:

string raw1 = @(foo"string s = @("Hello!");"foo);
string raw2 = @(bar"string raw1 = @("string s = @("Hello!");"foo);"bar);
string raw3 = @(baz"string raw2 = @(bar"string raw1 = @("string s = @("Hello!");"foo);"bar);"baz);

This syntax is loosely based on C++11 raw string literals, although I personally don't care for the parenthesis or delimiter appearing within the body of the string. I'm definitely not married to this syntax and variations are welcome.

Existing UserVoice suggestion, using C++11-like syntax: Allow to have custom delimiters in raw string literal

[scottdorman]

I like this, although I'm not sure about the syntax. It seems similar in nature to VB.NET's XML literals.

[HaloFour]

@scottdorman

The syntax is largely borrowed from C++ with some tweaks. C++ raw strings look like this:

const char* xml = R"foo(<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>)foo";

My personal opinion is that it's bizarre that the raw string termination identifier appears within the string.

While containing unescaped XML would certainly be a goal I wouldn't want to limit these strings to any specific form of content, such as HTML, SQL or JSON each of which has similar issues.

[scottdorman]

@HaloFour

While containing unescaped XML would certainly be a goal I wouldn't want to limit these strings to any specific form of content, such as HTML, SQL or JSON each of which has similar issues.

Absolutely. I was really just mentioning that this is similar in nature to VB's XML literals, but those literals don't require any special delimiters or structuring.

What about something like this:

string xml = @("foo(<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>")@;

So it's the pair of @( and )@ symbols that denote the start and end of the string literal? It's not quite as simple as the C++ syntax but I think using a pair of symbols keeps it from being ambiguous and keeps the delimiters consistent (which would better enable/allow code analysis).

[HaloFour]

@scottdorman

My proposal covers that syntax. But that would mean that the sequence ")@ becomes a special sequence that would require some form of escaping otherwise it would terminate the string. That's the purpose of the optional identifier, it establishes the termination sequence.

Note that the following two examples would produce the exact same string, the same string produced by the C++ example:

string xml1 = @("<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>");

string xml2 = @(foo"<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>"foo);

That foo within the parenthesis doesn't exist within the string.

[DavidArno]

With the example:

string xml = @(foo"<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>"foo);

You appear to be making the job of the compiler harder than it need be as, upon arriving at <item attr="foo, it has to check if the next token is ) to see if the end of the string has been reached or not. I'd therefore think it easier if the above were viewed as a compilation error (as there is a " following "foo ) and to make it necessary to use a custom delimiter that doesn't appear in the string:

string xml = @(bar"<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>"bar);

[orthoxerox]

Why not use here documents for that? Instead of a new delimiter you could use any custom delimiter that doesn't appear in the string. Something like:

string xml = @@bar
<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>
bar;

[HaloFour]

@DavidArno

I think having to ensure that the string can never contain the delimiter would make it harder on the developer than it should need to be. If you're copying/pasting a relatively large blob of formatted text you wouldn't want to have to scan the entire text to find out what would be a legal delimiter. That parsing requirement is exactly what has been imposed on C++ and it doesn't seem to have any issues with it. Also seems to be the common theme with heredocs.

@orthoxerox

Can't say I'm a fan of a string format that doesn't use double-quotes or is whitespace sensitive. I see that D has a syntax that is somewhat between other heredocs and C++ raw strings by using a q prefix, double-quotes and an identifier immediately followed by a newline:

string xml = q"foo
<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>
foo";

Of the heredoc syntax I'd probably prefer something along those lines the best:

string xml = @@"foo
<?xml version="1.0"?>
<root>
    <item attr="foo" />
</root>
foo";

[HaloFour]

@CyrusNajmabadi has suggested a different string literal syntax. It's not quite "raw" strings since there is still an escape sequence involved but the goal is to solve similar issues in avoiding having to use escape sequences:

@ <any-punc> contents <original-punc>

Within these strings, the only thing you'd have to escape would be the starting punc char

That would allow the following to be string literals:

var s1 = @`This is a string that contains a double quote:  "  `;

// with an escape sequence
var s2 = @`This is a string that contains a double quote: " and a backtick: `` `;

// change char to no longer need escape sequence
var s3 = @-This is a string that contains a double quote: "  and a backtick: ` -;

One advantage to this syntax is that it would allow interpolation:

var name = "Cyrus";
var json = $@`{{ "name": "{name}" }}`;

Which could potentially also be supported in my proposed syntax:

var name = "Halo";
var json = $@("{{ "name": "{name}" }}");

Although in both cases it is a tad awkward since you have to escape the curly brackets.

So between the two the major differences are:

1. Use of identifiers v. punctuation as terminators.
   1. What are the limits as to what punctation is supported?
2. Ability to use sequences of multiple characters as terminators.

[vladd]

I'd like to additionally propose Swift-like handling of the initial white space: if we have multiline raw literal, we want to allow ignored spaces at the beginning of each line. Swift resolves this in the following way:

To allow free formatting of the literal an indentation stripping operation is applied whereby any whitespace characters in front of the closing delimiter are removed from each of the lines in the literal.

[alrz]

var json = $@{{ "name": "{name}" }};

If you need to escape all braces, this is no longer a "raw string". probably it's better to escape interpolations instead.

[HaloFour]

@alrz

Agreed. I honestly don't care for trying to make interpolation and "raw" strings work. You still end up having to escape something.

The same is true in Swift. You still have to escape " if it appears in a triplet, and you still have to escape \( because the language supports interpolation.

[orthoxerox]

I think strings with custom delimiters would need custom interpolators, too. Something like: var json = $~@`{"name": "~name~"}`;. This way you would be able to use them for almost any raw data.

[alrz]

doesn't have to be a different character,

var json = $`{ "name": "`{name}`" }`;

so the interpolation is started via <punc>{

also I don't think @ is needed, this is a raw string so @ is implied.

[michael-hawker]

I feel like Python has solved this pretty eloquently with their triple quoted strings and support for both ' and " to define strings, as JavaScript provides too.

 var mystring = """Some text
 with a newline and "quotes" is here""";

I can't think of a time where I ran into a problem trying to store something with this approach.

[perbergland]

Is there any progress on this?

Being able to paste in raw json content instead of either quoting all quotes or putting the content in separate files would be an immense time/confusion saver.

Python's """ or some variation of heredoc would be fine with me.

[HaloFour]

@ZacharyPatten

But what if instead of trying all these tricks to force the string into a C# file, we just included a new keyword that would pull in a file as a raw string?

You can already use resources to accomplish something similar. But that moves the content into another file that you have to navigate to in order to view the string.

At this point most languages aside from C# seem to have added this feature, including C++ and Java. Its inclusion would be far from controversial.

[ZacharyPatten]

Why does the string need to be in the same file so bad if you can get the exact same functionality (or likely better considering you will have the extension of the file to color) from a file-to-string keyword?

You can already use resources to accomplish something similar. But that moves the content into another file that you have to navigate to in order to view the string.

As mentioned, IDEs could be made to treat this as if it were in-line while you are developing by simply having the file of the string open right below it. Example:

NOTE: I made this in an image editor and some of the values are off like the line numbers XML view.

At this point most languages aside from C# seem to have added this feature, including C++ and Java. Its inclusion would be far from controversial.

Just because they do doesn't mean it is proper for C#. If a file-to-string keyword existed I would personally never use the feature of this issue. There is no reason both couldn't exist though if people wanted choices though.

[HaloFour]

@ZacharyPatten

There's plenty of content in the 5-10 LOC range where splitting it into a separate file makes little sense and only makes it more difficult to track what is going on. Think SQL statements and the like. People do this already using C#'s verbatim strings, but you have to manually deal with escaping any quotes that might exist within the string. That's why this feature has been added to so many languages. C++ and Java have identical concerns to C# in this area.

Yes, tooling could improve that experience within the IDE, but it doesn't improve the situation in source control where a reviewer now has to manually bounce between completely separate parts of the PR to determine what is going on.

[ZacharyPatten]

I fail to see how having SQL in seperate files would make it harder to track. In my opinion it would make it easier. If you see a .sql file you know it is SQL; if you see .cs you don't immediately know if that has nested SQL inside it. Having seperate files for seperate languages is actually better organization in my opinion.

[HaloFour]

@ZacharyPatten

Well we'll have to agree to disagree. Having tons of little .sql or .json files strewn around the project to contain little bits of other languages to be used in C# is a nightmare to track and follow. Especially in integration tests where you might have to mock and compare multiple JSON chunks for a single test. In my current Java project we used to have dozens of separate JSON files for a single integration test suite and it was an awful experience. We've since moved them all to text blocks and it's an infinitely better experience. We've always followed this approach with our python tests. There's a reason so many languages have adopted this functionality, the alternative is objectively terrible.

[simonpbond]

@ZacharyPatten Personally I would prefer my SQL to remain inside my function (and typically the only function it relates to), otherwise it feels similar to breaking up functions into their own files. The SQL execution and the SQL query go hand in hand in my view. As far as feeling the actual frustration, try using Postgres as a database with .NET (or as a custom store for Identity Framework) you will feel the pain! : Here is an example from a Microsoft MVP on the issue. Not to say this should be the only reason for such a change, but I don't see the downside to it.

[simonpbond]

@ZacharyPatten Just to add, I see nothing wrong with your idea also, we are probably just seeing it from a different use case.

[HaloFour]

@ZacharyPatten

I'd also suggest that if you wanted to organize those files separately but not have to deal with loading them as files or as resources at runtime that a source generator would probably be a decent solution. You'd define a partial class annotated with an attribute with the directory and file pattern and the generator could embed the contents of each file as a string member or constant. That'd probably work better than the approach with project resource files today (where the tooling is not well suited to multiline content). But it still leaves the content as separate files.

To each their own. I know from my experience that people are already using verbatim strings to embed this content into their source. It works alright but requires escaping double quotes and doesn't round-trip particularly well.

[merarischroeder]

I think the tripple-quote method looks the cleanest. It's one way forward that works with both @ and $. This would solve the biggest issue, being double-quotes.

Once that's implemented, we could move onto more edge cases later.

(I absolutely disagree with the use of resource strings. Some may be happy with that, but this is about adding a simple feature that others would like. )

[alrz]

absolutely disagree with the use of resource

I think the biggest winner will be roslyn itself since it heavily use strings in tests and we still need to eacape quotes for string literals. and resources isn't really the answer.

[anthony-c-martin]

Out of interest, does anyone have any creative (hacky) solutions for dealing with this today? I'm working with a lot of multiline strings with embedded JSON in test libraries; it's super frustrating with having to escape " and { e.g.

var myJson = @"
{{
  ""prop"": ""val"",
}}";

I was specifically wondering about embedding the strings in multiline comments and using reflection to read them back, but couldn't find a way to do this cleanly...

[HaloFour]

I was specifically wondering about embedding the strings in multiline comments and using reflection to read them back, but couldn't find a way to do this cleanly...

Comments aren't compiled into the assembly so there wouldn't be a way to read them out at runtime via reflection.

You could use embedded resources, though. Add the JSON files to the project and set them to be embedded in the assembly as resources, then you can open them using Assembly.GetManifestResourceStream. Or you can add a Resources project item to your project in Visual Studio which will generate a helper class to retrieve the resources.

[ajdust]

I usually give up using string interpolation with raw strings in C#. Escaping " and { quickly becomes absurd. Generally I'll replace double quotes with single quotes and have a helper to finish the job. So something like this:

Generating:

{ "Foo": "thing", "Bar": "Baz", "Buz": 123 }

I'd do something like:

var body = @"{
  'Foo': $fooThing,
  'Bar': 'Baz',
  'Buz': $buzThing
}".Json(new { fooThing = "thing", buzzThing = 123 })

I find more readable than

var body = $@"{{
  ""Foo"": ""{fooThing}"",
  ""Bar"": ""Baz"",
  ""Buz"": {buzThing}
}}"

Replace the 3-line example with 100-lines of complex JSON slightly mangled to unit test some edge case that should error out, etc.

[worldbeater]

Typescript uses ` for this, so something as concise as the following would be really nice to have:

const example = @`{ "json": 42 }`;

[slang25]

F# adopts the Python style triple quotes, and I'm really happy with it. I would love for C# to adopt the same, I'm forever cursing under my breath everytime I try to write XML, JSON etc... in a C# string literal today.

[merarischroeder]

It's great that C# has got so many boundary-pushing features, but it would be great if the C# team could see the value in this feature. So much time is wasted worldwide on quote escaping for HTML/CSS/Javascript snippets.

[CyrusNajmabadi]

It's also a pain when the text in the string is C# syntax and you're writing tests for analyzers and source generators.

If it's c# syntax, then i'd want a flexible escape mechanism. The outer string always needs some way of escaping that won't collide with the inner string's fixed contents.

[HaloFour]

@CyrusNajmabadi

I think that the C++ raw string approach is really the only option there, since it'd be the only raw string syntax that can contain C# source with other raw strings (which can contain C# source with other raw strings, etc., etc.). That's why I suggested the syntax above, which is a slight play on C++ raw strings.

[ajdust]

Talk about SQL injection attacks reminds me of the SQL Template Strings, which uses JavaScript template strings to allow clear yet safe parameterized SQL as easy as sql`select * from t where t.id = ${id}`.

Combined with an IDE analyzer to detect common formats (SQL, JSON, etc) and pop some recommendation (and syntax highlighting inside the string), there could be a way to fight injection-vulnerable code directly with the language/IDE features. With the right language features, we could have libraries with security and convenience.

[merarischroeder]

... as easy as sql`select * from t where t.id = ${id}`.
.. to detect common formats (SQL, JSON, etc) ... With the right language features, we could have libraries with security and convenience.

That's a brilliant idea. I think. C# needs this and others so programmers have the tools to choose from.

i) triple quoted that work the same as python
ii) configurable schemes like r-"example"-r
iii) sub-type declaring as you suggest. In C# this might work better with string alias types and reverse inference. SQLString s = "select * from happyCoders";

iv) This does all point to the weakness of using lines of text though. Ideally the can be a fusion with the IDE, where such snippets are stored as resources but displayed/edited inline in a box. I do believe that in the end code is "data for the compiler". These issues with strings highlight that fact. We struggle with code-data encoding for strings.

[jnm2]

@ajdust Entity Framework already has this feature with FromSqlInterpolated (https://docs.microsoft.com/en-us/ef/core/querying/raw-sql#passing-parameters) but IIRC it is considered risky because simple-looking refactorings bring injection risk again:

// Okay
var blogs = db.Blogs.FromSqlInterpolated($"EXECUTE dbo.GetMostPopularBlogsForUser {user}").ToList();

// Not okay
var query = $"EXECUTE dbo.GetMostPopularBlogsForUser {user}";
var blogs = db.Blogs.FromSqlInterpolated(query).ToList();

[CyrusNajmabadi]

I'll champion this.

[CyrusNajmabadi]

Here's my proposal: #4304

[juwens]

I need to add perls qw into the race, it results in quite consistent, flexible and readable way, to use whatever string delimiter you like. (cause whichever predefined delimiter you pick, there is always this one string, where it will not fit.)

var foo = qw/This is a string/;
var foo = qw{This is a string};
var foo = qw[This is a string];
var foo = qw'This is a string';
var foo = qw"This is a string";
var foo = qw!This is a string!;
var foo = qw@This is a string@;

It's kinda similar to the authors:

string raw1 = @(foo"string s = @("Hello!");"foo);
string raw2 = @(bar"string raw1 = @("string s = @("Hello!");"foo);"bar);
string raw3 = @(baz"string raw2 = @(bar"string raw1 = @("string s = @("Hello!");"foo);"bar);"baz);

But IMO more readable:

string raw1 = qw'string s = @("Hello!");';
string raw2 = qw/string raw1 = @("string s = @("Hello!");"foo);/;
string raw3 = qw!string raw2 = @(bar"string raw1 = @("string s = @("Hello!");"foo);"bar);!;

PS: I'm no perl evangelist by any means, but i found this syntax always quite compelling.

[HaloFour]

Raw strings have already shipped as a part of C# 11 using the triple-double-quote syntax:

https://github.com/dotnet/csharplang/blob/main/proposals/csharp-11.0/raw-string-literal.md

var foo = """This is a string""";

var raw1 = """var s = "Hello!";""";
var raw2 = """"var raw1 = """var s = "Hello!";""";"""";
var raw3 = """""var raw2 = """"var raw1 = """var s = "Hello!";""";"""";""""";

[juwens]

I thought this is an ongoing process of improving the language.

Cause the current raw string literals (though an improvement) are really not ideal in different scenarios. Like when you want to express "\"\"\"" the raw string literal would be:

var s = """
    """
    """;

which is quite nasty.

[ufcpp]

You can use more than three quotes.

var s = """"
    """
    """";

[HaloFour]

I thought this is an ongoing process of improving the language.

Of course, although now that a feature has shipped for this space it's unlikely for another form to be considered without demonstrating an extraordinary need.

Cause the current raw string literals (though an improvement) are really not ideal in different scenarios. Like when you want to express "\"\"\"" ...

The design of raw string literals took the common cases into consideration, such as embedding markup and code, including C# code that contains raw string literals. They might not be ideal for all scenarios. A question would be whether those scenarios are considered critical enough to cover.

[AartBluestoke]

I thought this is an ongoing process of improving the language.

Cause the current raw string literals (though an improvement) are really not ideal in different scenarios. Like when you want to express "\"\"\"" the raw string literal would be:

var s = """
    """
    """;

which is quite nasty.

var s = """"""""""""""""""""""""""""""""""""""""
""" just be a literal thing as long as there is less than 40 quotes in a row """
"""""""""""""""""""""""""""""""""""""""";