Prototype: Non-null reference types

[jaredpar]

Feature Documentation

https://github.com/dotnet/roslyn/tree/features/NullableReferenceTypes/docs/features/NullableReferenceTypes

[jods4]

I guess that the rules for implicit types are nullable by default because they can't be "used" outside the scope where they are declared and in this scope flow analysis is enough.

The benefit is that even when they're initialized to a non-null value (locals and anonymous types have to) they might still be assigned null later on without warning.

I'd like to point out that this choice might hurt later, should C# ever allow implicitly typed members. I.e. if you ever allow replacing private Dictionary<string, int> cache = new Dictionay<string, int>() by private var cache = new Dictionary<string, int>().
If you allow that, then cache being nullable is going to be a pain point.

The same problem will arise if you ever allow to omit the return type of a lambda method or property: public var GetFrob() => new Frob();

Of course, a workaround would be to continue to explicitly declare the return type, but that kind of defeats the feature.

It would be nice if there was a "non-null" cast operator. I have no good syntax to propose, so I'll use (!) in my examples.

This is useful when a method returns a nullable type, but you know from the parameters that you passed that the result is actually not null. For example:

// Concat returns null if any argument is null
string? Concat(string? a, string? b)
{
  if (a == null || b == null) return null;
  return a + b;
}

var phrase = Concat("hello", "world");
phrase.Length;  // <-- Error because phrase might be null, although we know it's not

// With a "non-null" cast
var phrase = (!)Concat("hello", "world");
phrase.Length; // <-- OK

The only possible workaround is to perform a cast to the non-null type, which can be inconvenient if the type name is long or impossible if the type is anonymous and fails to be determined non-null by flow analysis.

It could also be used to help with the problem outlined above, to make implicit members non-null: private var cache = (!)new Dictionary<string, int>().

[HaloFour]

@jods4 IIRC there was also a proposal to add a ! postfix operator would which force a nullable to a non-nullable, throwing if the operand is null:

private var cache = new Dictionary<string, int>()!;

[jods4]

@HaloFour that's cool. I guess it works great when dotting into a ref, e.g. nullable!.toString(). and the symetry with ?. is nice.

It does the job but is slightly weird at the end of an expression: let array = [1, 2, 3, 4, 5]!; even more so if you split your expression several lines.

[alrz]

@AlekseyTs Just to mention, T? meaning two things is really awful. The fact that, even we have non-nullable reference types, we still require where T : class and where T : struct to use T? isn't really helping. Couldn't we unify nullables with reference and value types with Nullable<T>?

[yaakov-h]

I think this would be a lot more sensible if you could overload based on nullability. e.g.

string Concat(string a, string b) => ....;
string? Concat(string? a, string? b) => ....;

I'm not intimately familiar with the runtime and compiler platform, but from the little I do know, I don't think that the current approach would allow for this. Perhaps a modopt or modreq?

[jods4]

@yaakov-h Maybe we could get away without a new syntax?
T Concat<T>(T a, T b) where T: string
Or is that too weird that Concat is generic although it only supports a single type?

[yaakov-h]

@jods4 That sounds like a bastardisation of generics to me, and wouldn't necessarily let you do everything you'd want from nullability overloads.

[jods4]

@yaakov-h I agree that it feels a bit wrong.
What do you have in mind RE: "wouldn't necessarily let you do everything you'd want from nullability overloads"?

[yaakov-h]

@jods4: Assume the following:

string Concat(string a, string b) => string.Format("{0}{1}", a, b);
string? Concat(string? a, string? b) {
    if (a == null || b == null) { return null; }
    return Concat(a!, b!);
}

(Assuming the postfix-non-null-cast operator ! makes it in)

How would you do this with only T Concat<T>(T a, T b) where T : string?

[jods4]

@yaakov-h Ah I see what you mean. The problem is not the signature but implementing the function.
Varying nullability by generic parameters doesn't play nice.

On the other hand, overloads by nullability are not allowed either, are they? As they would compile to exactly the same IL method?

As long as the CLR doesn't support nullable proper, this is going to be hard APIs to model in C# 😞
The only way out that I see is introducing weird hacky annotations that instruct the C# compiler that the output is nullable iff some inputs are nullable...

[yaakov-h]

@jods4 I was under the impression that the CLR can overload by modopt or modreq (e.g. const in C++/CLI), but .NET (C#/VB) doesn't handle this properly.

[MadsTorgersen]

@yaakov-h @alrz These are indeed downsides to the design. The reason for it is to allow APIs to add nullability annotations while staying binary compatible. Using Nullable specifically would also cause a perf hit: That is a struct with two members, one of which (the bool tracking whether the content is null) would be redundant for reference types, yet expensive to carry around.

[jods4]

@yaakov-h You're right about modopt, it's made for that purpose and it does support overloads. We'll see what the C# team decides to use.

[alrz]

@MadsTorgersen For binary compatibility I believe an attribute would just work (as it does today if you are working with an analyzer) but for a type like T? falling back to that approach makes less sense, what if I want to use a nullable value no matter if it's a struct or class as it is part of my library contract? If non-nullability is going to be first-class and the default having a type for nullable types doesn't seem unreasonable. As for its perf hit, isn't this an optimization matter? I mean with special treatment of the type Nullable there wouldn't be an observable difference from client's point of view.

[AlekseyTs]

Using modopts will break backward compatibility story. It would mean that, once you annotate types in a library, old clients (previously built consumers of the library) won't be able to use now annotated APIs because modopts change signatures.

[jods4]

@AlekseyTs
Binary compat at runtime:
I'm not very knowledgeable in modopts implementation details. But since modopts can be ignored by languages that don't understand them, wouldn't that mean that if there is no better overload the runtime should still bind to functions with non-matching modopts, if there is no better overloads? Or is that resolution actually performed and encoded at compile-time?

Compile-time compat:
I don't see a problem here... As long as your file doesn't opt into the new non-nullable thing the compiler allows it to work just fine.

[alrz]

@AlekseyTs Couldn't we have both? I mean, If I want to take advantage of non-nullable reference types and don't break backward compat I'd use an attribute, and otherwise I'd use the type T?.

[AlekseyTs]

@jods4 Binary compat: Runtime doesn't do overload resolution, it looks for an exact signature match. Compiler can ignore modopts for the purpose of the analysis, but it has to spit them out in signatures anyway

@alrz Are you saying that anyone going after binary compatibility won't be able to use ? in code, but will have to resort to manually encoding this information in attributes, which becomes rather tricky with arrays and generics?

[alrz]

@AlekseyTs Yes, "which becomes rather tricky with arrays and generics" you mean something like array of nullables or nullable array of non-nullables? That seems quite rare (I usually want it all be nullable or non-nullable, so [Nullable] T[] would just work), however, we still can mix those two like [SomeAttr] T?[] to also be able to use ? and remain binary compatible, there are other ways to do this of course, whichever seems better but I definitely want to be able to use ? without worrying about the type being a struct or class. That seems part of the signature, doesn't it?

[jods4]

@AlekseyTs by runtime I meant JIT of course...
Thanks for the explanation, I learned something today: modopt is encoded by the compiler at call site and has to match the method signature exactly.
Which makes me wonder: how does a langage like C++ handle this? If a method had a non-const param, which is marked const later on, assemblies compiled against the old version are binary incompatible with the new one?
That's a bummer because if C# encodes (non-)nullability with modopt then binary incompatibility would certainly slow down adoption by library vendors (probably until the next major release?), who are good audience for using this feature (because it serves as documentation to the consumers).

@alrz

That seems quite rare (I usually want it all be nullable or non-nullable)

I wouldn't assume that too quickly... I could come up with examples without much trouble.
As a general design rule, having a type system that can't represent all its valid combinations seems broken.

One single example: say you implement List<T>. As an internal storage you have T[] buffer. Non-null obviously. Now a consumer instantiate List<Thing?>, which is surely a legal operation. Everything would (hopefully) work fine, but it seems weird that you actually are unable to encode the type of buffer (which would be Thing?[], a non-null list of nullable things).

[alrz]

@jods4 Ok, assuming that non-nullability is the default, when you want to opt out without a breaking change you use the [Nullable] attribute, it causes the compiler treat the type as before, so it's binary compatible. And when you use ? you are actually changing the type so it woudn't be.

For example, if you want nullable types for a function F, you just need to add that attribute:

[return: Nullable]
object F([Nullable] object obj) { ... }

whatever type it is. But when you change the types to object? you are changing them. There can be some interaction between [Nullable] attribute and nullable types (?) which I think can be handled behind the scene.

[HaloFour]

@alrz So, if they're required to use NullableAttribute they have to remember that:

public static List<Dictionary<string, string?>> Foo() { ... }

is really

[return: Nullable(new bool[] { false, false, false, true })]
public static List<Dictionary<string, string>> Foo() { ... }

[alrz]

@HaloFour No, when you use ? you are changing the type, as @AlekseyTs said, "anyone going after binary compatibility won't be able to use ? in code, but will have to resort to manually encoding this information in attributes" This is what I meant.

[HaloFour]

@alrz And my point is that that is the attribute notation someone would have to know in order to manually decorate a parameter. I recall modopt was researched before (probably more than once) and discarded due to the binary incompatibility problem. Why is it being brought up again?

[alrz]

@HaloFour I'm thinking that the NullableAttribute woudn't have any arguments, is that wrong?

[HaloFour]

@alrz I don't see how modopt would change that. You'd still have two different flavors of ? types, one Nullable<T> and the other modopt T. Such change would also represent the first time that a compiler could emit an assembly that would not be compatible with previous versions of the compiler regardless of the targeted framework, I believe.

[alrz]

@HaloFour Lots of proposed features already depend on the targeted framework, with separate branch of .NET framework itself (.NET Core 1.0) it makes more sense to rethink the design and don't repeat the same mistakes again, non-nullability seems most fundamental among others and it's really important how we start to support it. As for modopt I don't mind as long as it's not observable.

[HaloFour]

@alrz Yes, but they don't depend on the version of the compiler. I can compile an assembly using Roslyn targeting .NET 2.0 and I can consume it from a project using Visual Studio 2005 without issue.

I've not heard of any mention of Roslyn making hard breaks with the full .NET framework and I seriously doubt that a feature such as this would take a hard dependency on such a thing. The attribute-based implementation is the result of a lot of compromises spanning a lot of discussions and I think that everyone is in agreement that it's not a perfect solution but it is a good solution.

[alrz]

@HaloFour Generics also introduced to the .NET framework on top of non-generic interfaces e.g. IEnumerable you do have to write boilerplate but you don't lose backward compat, I was thinking that the same path can be taken for non-nullable references. That's it.

[HaloFour]

@alrz Yes, but you could still target .NET 1.1 using C# 2.0, compile an assembly, and consume it using C# 1.x without issue.

Anywho, I get what you're trying to do, but if the team is trying to get this feature into the next release (implied by the "Urgency-Soon" tag) I doubt that it will survive any serious consideration for redesign. And, as mentioned, modopt/modreq had already been thrown around.

[jods4]

@HaloFour

Yes, but they don't depend on the version of the compiler. I can compile an assembly using Roslyn targeting .NET 2.0 and I can consume it from a project using Visual Studio 2005 without issue.

I don't think that is the compatibility issue with modopt, is it? The ECMA standard says that a compiler is free to ignore modopt it doesn't understand, and nullable types fit into this model.

My understanding is that binary compatibility doesn't work: if Microsoft releases .NET next using nullability annotations everywhere (and they should do that), then existing assemblies built against earlier .net versions will not run against on top of the new framework (because method signatures have changed), which is a no-go.

The point about dynamic is a good one. It's exactly the same kind of annotation: targeted at compilers, which can be ignored by compilers that don't understand it and which can target any type anywhere (generics, etc.). The DynamicAttribute(bool[]) solution has been shipped and works, there is no reason to me NullableAttribute(bool[]) wouldn't work the same?

[yaakov-h]

@jods:

if Microsoft releases .NET next using nullability annotations everywhere (and they should do that), then existing assemblies built against earlier .net versions will not run against on top of the new framework (because method signatures have changed), which is a no-go.

Doesn't this only apply if .NET next is an in-place upgrade (e.g. 4.5 over 4.0), and not a separate framework (e.g. .NET 4.0 vs .NET 3.5)?

[jods4]

They allow more breakage, but generally MS keeps a high compatibility level even between side-by-side major releases. In fact, all of the projects I use or maintain run perfectly on 4.6, even those that were compiled for the 2.0 runtime.

Adding modopt to many BCL classes would basically mean that no existing program would run on vNext if nothing else changes.

[HaloFour]

@jods4 With extremely few exceptions you can expect the same going back to .NET 1.0 assemblies.

[alrz]

I think an Option<T> type with language support would be nice when we're dealing with generic code. Then we can use something like this to reduce interface invocations by half in foreach traslations.

using(var e = list.GetEnumerator()) {
  while(true) {
    let Some(var item) = e.Next() else break;
    // loop body
  }
}

using(var e = asyncSeq.GetAsyncEnumerator()) {
  while(true) {
    let Some(var item) = await e.NextAsync() else break;
    // loop body
  }
}

So T? would be Nullable<T>, Option<T> or nullable reference if T is a value type, generic type, or reference type, respectively.

[HaloFour]

@alrz Wouldn't that be T??? If you're using Option<T> to convey whether there is another element in the sequence you'd still need to be able to have that next element be null. Either way, we've gone from having T? meaning two different things being horrible to having T? meaning three different things being okay?

[alrz]

@HaloFour If you agree that T? meaning two different things is "okay" (horribly), you woudn't mind if T? would be an error if T is a generic type? That is pretty freaking far away from being okay I guess. I don't mind if Option<T> wouldn't have any syntactic sugar, though. F# almost forbids nulls (with help of AllowNullLiteralAttribute for backward compat) and has an option type which just works and even async sequences are implemented that way.

[jcouv]

Work items list (#22152) supersedes this for tracking work.
Championed issue (dotnet/csharplang#36) tracks language issues and decisions.
I'll close the present issue.