[Proposal]: Roles and extensions

[MadsTorgersen]

Roles and extensions in C#

• Proposed
• Prototype: Not Started
• Implementation: Not Started
• Specification: Not Started

Motivation

In C# today, one assembly can use extension methods to augment types of another assembly, effectively adding new instance methods to it. This proposal expands on that principle so that:

1. All kinds of members, including static ones, can be added from the outside, as long as they don’t add instance state
2. Augmentations can be “roles” which selectively augment individual values, as well as “extensions” which augment all values of a type throughout a scope (like today’s extension methods)
3. Roles and extensions present as first-class types, with names and conversions
4. Stretch goal: Roles and extensions can implement interfaces on behalf of the underlying types and values, acting as adapters

Common to these is that they don’t require involvement of the underlying type, and don’t intrude upon its inherent semantics, thus facilitating new degrees of decoupling and separation of concerns.

The main purpose of the proposal at this point is to lay out the motivating scenarios and a comprehensive set of ideas for addressing them. The terms, concepts and syntax are all up for discussion. From a language design perspective there is a lot of new ground here, so it's ok - even necessary - for us to spend some time and effort making sure we end up giving it the best embodiment that we can.

Syntax

Acknowledging the design latitude, here’s the style of syntax I’ll use in the following:

role Name<T> : UnderlyingType, Interface where T : Constraint
{
    // Member declarations
}
extension Name<T> : UnderlyingType, Interface where T : Constraint
{
    // Member declarations
}

Extensions are just roles that apply automatically to their underlying type when brought into scope with a using directive. Roles and extensions have a name, can be generic, can implement interfaces and can declare members that don’t introduce instance state. Unlike base types elsewhere, the underlying type can be pretty much anything, including value types, type parameters, pointer types etc.

A much more detailed description of the syntax and semantics follows under "Detailed design" later on.

Motivating examples

These examples are meant to present motivating "archetypical" uses of roles and extensions. There are some bigger examples near the end of the document.

Throughout these examples I will make use of a simple DataObject type meant to represent weakly typed, semi-structured data, e.g., wire data:

public class DataObject
{
    public DataObject this[string member] { get; set; }
    public string AsString() { }
    public IEnumerable<DataObject> AsEnumerable();
    public int ID { get; }
}

None of the examples modify the type itself, but instead enhance it in various ways with roles and extensions.

Layering with extensions

API layering often conflicts with discoverability. We want to create APIs with only a one-way dependency, but members relating to the depending layer would be most naturally found in the depended-on layer. With extensions the depending layer can add members to types in the depended-on layer:

public extension JsonDataObject : DataObject
{
    public string ToJson() { … this … }
    public static DataObject FromJson(string json) { … }
}

Here the DataObject type is extended by a Json conversion API to have a static and an instance method, converting to and from strings representing Json.

The instance method of the extension can be considered a new and better syntax for the extension methods we already have in C# today. Bodies of instance extension members may refer to the extended object with the this keyword.

The name of the extension can be used for disambiguation purposes, similar to the static class containing extension methods today. We might consider allowing extension declarations without names.

Extensions can be brought into scope the same way as extension methods today: With a using static directive on the extension type or a using on its namespace:

using JsonLibrary;

var data = DataObject.FromJson(args[0]); // Extension static method
WriteLine(data.ToJson());                // Extension instance method

Lightweight typing

Wire data is most naturally represented with weakly-typed dictionary-like types such as DataObject, but those don’t lend themselves well to helpful tooling and elegant program logic. However, conversion to strong types is expensive and lossy, and all-out wrappers still introduce a level of indirection. Roles can provide strongly typed augmentation with a lighter touch:

public role Order : DataObject
{
    public Customer Customer => this["Customer"];  
    public string Description => this["Description"].AsString();
}
public role Customer : DataObject
{
    public string Name => this["Name"].AsString();
    public string Address => this["Address"].AsString();
    public IEnumerable<Order> Orders => this["Orders"].AsEnumerable();
}

Roles are transparent wrappers. They are considered identical to their underlying types and have implicit conversions in both directions. The conversions are free at runtime, because the underlying representation of the object doesn’t change. The Order is the DataObject. As the Orders property shows, even constructed types over roles convert freely. So now strongly typed, IntelliSense-aided code can be written without clunky indirection or runtime penalty:

IEnumerable<Customer> customers = LoadCustomers();

foreach (var customer in customers)
{
    WriteLine($"{customer.Name}:");
    foreach (var order in customer.Orders)
    {
        WriteLine($"    {order.Description}");
    }
}

It is easy to imagine roles for wire data being automatically generated, e.g. by a source generator based on a schema or sample. This would address a similar scenario to F# type providers, but with named types and generated source code you can debug through.

While there would be nothing technically difficult about allowing identity conversion between two different roles with the same underlying type, I think it would be useful to disallow it – or at least warn about it – making people explicitly take the indirect route through the underlying type. That way the roles provide some protection against accidentally converting things across the role hierarchy.

Adaptation to interfaces

We can imagine using roles and extensions to implement additional interfaces on values of the underlying types.
Say there’s some framework that you plug into by implementing interfaces such as this:

public interface IPerson
{
    public int ID { get; }
    public string FullName { get; }
}

We would like some of our person-like roles over DataObject to plug in to this framework. We can do this by expanding our declaration of Customer to:

public role Customer : DataObject, IPerson
{
    string IPerson.FullName => Name;
    // … plus same members as before
}

The declaration of Customer explicitly maps FullName to Name, but does not need to contain an implementation of ID because it “inherits” a suitable one from the underlying DataObject type.

Now the Customer role can be used to treat DataObject instances as implementing IPerson! It will satisfy IPerson as a generic constraint, and there would be an implicit conversion – likely a boxing conversion – from Customer to IPerson.

Combining roles and extensions

In the previous example, the developer (or source generator!) who declared Customer may not be aware of the person framework or interested in integrating with it. Again, this can be solved with an extension. The Customer role itself can be extended to fulfill the interface!

public extension CustomerAsPerson : Customer, IPerson
{
    string IPerson.FullName => Name;
}

Whoever imports this extension can then use unsuspecting Customers – who are themselves unsuspecting DataObjects – as IPersons!

Generalized aliases

Simple roles without bodies can serve as a better form of aliases, essentially rendering using aliases obsolete:

public role DataSet<T> : Dictionary<string, T>;

Unlike using aliases, such a "role alias" can be public (or any other accessibility), can be generic, can participate in the file's declaration space of types, can be recursive, etc. They are real type declarations, but still work as if they are just synonyms for the underlying type.

Detailed design

Role declarations are a new form of type declaration:

role_declaration
    : attributes? struct_modifier* 'partial'? role_keyword identifier type_parameter_list?
      ':' type role_interfaces type_parameter_constraints_clause* struct_body ';'?
    ;
    
role_keyword
    : 'role'
    | 'extension'

role_interfaces
    : (',' interface_type)*
    ;

A role is in some ways like a struct and in some ways like a class. Like a struct it cannot be derived from, and therefore cannot be abstract or sealed. It also cannot have virtual or overriding members (except perhaps allowing it to override the members of object similar to structs). It can implement interfaces, using a combination of its own members and ones inherited from the underlying type. Just like a struct, the role has a boxing conversion to those interfaces (though not directly to object).

Somewhat like a class, however, a role specifies a kind of "base type" which we call the underlying type. Unlike a base type the underlying type is mandatory. On the other hand it can be any type, not just a class type. (We need to think about syntactic ambiguities in this position for tuples, pointer types etc.) At compile time the role inherits the members of the underlying type, just as a derived class inherits from its base class.

(Open design question: A role could be allowed to specify additional base roles. They would need to be applicable to the specified underlying type, and would provide a way to combine roles through a form of "multiple inheritance".)

The role transparently "wraps" the underlying type in such a way that values of the role and of the underlying type have identical runtime representation. For this reason a struct_member_declaration in a role body is prohibited from declaring additional instance state, either as a field, an auto-implemented property or a field-like event, on top of what's inherited from the underlying type.

The underlying type of a role can be any type, even an interface, a value type, an array type, a type parameter, the dynamic type, a pointer type or another role. Thus a role can be used to enhance types that cannot normally be extended with new functionality. Certain kinds of underlying types are subject to restrictions that limit how the role can be defined or used. For instance, a role on a ref struct cannot specify interfaces, and does not have a boxing conversion. A role on a pointer type or ref struct cannot be used as a type argument or an array element type.

In general, operations on a role are resolved exactly like those on a derived class: try to use the declarations in the role first, and if those aren't found, use the underlying type. The concrete mechanics of this for each operation needs to be specified in detail, but should generally be as similar as possible to how the corresponding operations are handled in a derived class.

The identical runtime representation of a role and its underlying type enables identity conversions to exist both ways between them. This in turn leads to the existence of identity conversions between constructed types where roles and their underlying types respectively are used as type arguments, as well as between array types where the element types are roles and their underlying types respectively.

Identity conversions in C# are generally transitive, which would allow them to apply not just to and from the roles and their underlying types but also between roles that have the same underlying type. However, it might nevertheless be reasonable to discourage direct conversion between unrelated roles on the same type, e.g. with warnings or even errors, so as to enable the enforcement of a "shadow type hierarchy" of roles. We have precedence for such diagnostics on identity conversions between tuples using different element names.

In addition, there is an implicit boxing conversion between a role and each of the interfaces it implements. This means that a new object is created whenever the role is converted to the interface, and the result will thus not be reference-equal to the underlying object, even if that is a reference type.

Any reference conversions, implicit and explicit, to and from the underlying type, are inherited by the role, although its own identity and boxing conversions take precedence, along with any user-defined conversions declared in the role.

The identity, reference and boxing conversions on a role can be used to satisfy generic constraints when the role is used as a type argument. This is the source of much of the feature's expressiveness, because roles can be used to "bridge" or "adapt" an existing class or struct (or other type) to an existing interface without modifying either, in such a way that values of the existing class or struct can be passed to generic methods using the existing interface as a constraint.

An extension declaration is simply a role declaration with a different keyword. In addition to all of the above, when an extension is in scope - either directly or via being brought in through a using or using static directive - it automatically applies as a "fallback" role for the underlying type. Similar to existing extension methods in C#, if a lookup fails on the underlying type, it is tried again against the extension. If multiple extensions apply, ambiguities between applicable members are resolved in the same manner as for extension methods.

In addition to member lookups, extensions can apply when the underlying type is used as a type argument, allowing interfaces implemented by the extension to help satisfy constraints on the corresponding type parameter.

Implementation strategies

It is a cornerstone of the proposal that there is an identity conversion between a role and its underlying type, and that just like other identity conversions in the language it is essentially free – there is no representation change at runtime.

One implementation strategy is to use “erasure”, just as we do with dynamic vs object, with tuple names and with nullability annotations. Here, in the IL we generate, we really do use the same type (the underlying type in this case) and then use other metadata (attributes etc.) to communicate the extra information about which language-level "embodiment" of the type is meant here. In this way, at runtime, the value doesn’t change its type at all and the identity conversion is trivially “free”. Even for constructed types, e.g. arrays, the conversion is free because the underlying type is the same:

DataObject[] dos = new Customer[10]; // Erasure: Customer[] is really DataObject[]

Of course the added members still need to be encoded somewhere; it is likely that we would generate some form of type declaration under the hood to hold them.

Unfortunately, and erasure strategy will come up short for the scenario of roles and extensions implementing interfaces. Consider this generic data structure:

class Rolodex<T> : IEnumerable<T> where T: IPerson
{
    Dictionary<string, T> people = new ();
    void Add(T person) => people.Add(person.FullName, person);
    T Find(string name) => people[name];

    public IEnumerator<T> GetEnumerator() => people.Values.GetEnumerator();
    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

Assuming Customer implements IPerson, we should be able to create a Rolodex<Customer>:

var rolodex = new Rolodex<Customer>();

We can’t just “really” pass DataObject as the type argument because it doesn’t satisfy the constraint. There needs to be a real Customer type at runtime so that the constraint can be satisfied and the call to person.FullName in the body of the class can be resolved. Values of the Customer type must have identical runtime representation to their underlying DataObject, and the runtime must be able to understand that it can freely convert between them as a no-op, using the same bits. Essentially it needs to be like a wrapper struct: The bits are the same, the type is different.

Furthermore, the returned Rolodex<Customer> implements IEnumerable<Customer> and DataObject is identity convertible to Customer. So Rolodex<Customer> must be implicitly reference-convertible to IEnumerable<Customer>! The runtime needs to understand and allow this conversion:

IEnumerable<DataObject> dataObjects = new Rolodex<Customer>();

In short, there are things the runtime will need to understand about roles if we embrace their ability to implement interfaces.

More examples

Implementing IEnumerable<T> with an extension

Say we want to view a ulong as an IEnumerable<byte>. We can do that with an extension implementing that interface:

public extension ULongEnumerable : ulong, IEnumerable<byte>
{
    public IEnumerator<byte> GetEnumerator()
    {
        for (int i = sizeof(ulong); i > 0; i--)
        {
            yield return unchecked((byte)(this >> (i-1)*8));
        }
    }
    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

Now, wherever the extension is brought in you can iterate over the bytes in a ulong:

foreach (byte b in 0x_3A_9E_F1_C5_DA_F7_30_16ul)
{
    WriteLine($"{e.Current:X}");
}

You can also use its boxing conversion to IEnumerable<byte> to access LINQ functionality:

const ulong ul = 0x_3A_9E_F1_C5_DA_F7_30_16ul;
var q1 = Enumerable.Where(ul, b => b >= 128);  // method argument
var q2 = ul.Where(b => b >= 128);              // extension method receiver
var q3 = from b in ul where b >= 128 select b; // query expression
// etc.

Implementing IComparable<T>

IComparable<T> is most commonly used as a constraint on generic types or methods that need to compare several values of the same type.

With extension interfaces we can make types IComparable<T> that wouldn't normally be. For instance, if we have an enum

public enum Level { Low, Middle, High }

We can make it comparable with the extension declaration:

public extension LevelCompare of Level : IComparable<Level>
{
    public int CompareTo(Level other) => (int)this - (int)other;
}

We can also extend only certain instantiations of generic types with an interface implementation. For instance, let's make all IEnumerable<T>s comparable with lexical ordering, as long as their elements are comparable:

public extension EnumerableCompare<T> : IEnumerable<T>, IComparable<IEnumerable<T>> where T : IComparable<T>
{
    public int CompareTo(IEnumerable<T> other)
    {
        using var le = this.GetEnumerator();
        using var re = other.GetEnumerator();
        while (true)
        {
            switch (le.MoveNext(), re.MoveNext())
            {
                case (false, false): return 0;
                case (false, true): return -1;
                case (true, false): return 1;
            }
            var c = (le.Current, re.Current) switch
            {
                (null, null) => 0,
                (null, _) => -1,
                (_, null) => 1,
                var (l, r) => l.CompareTo(r)
            };
            if (c != 0) return c;
        }
    }

Now we can use an IEnumerable<T> as an IComparable<IEnumerable<T>> wherever this extension is in force, but only as long as the given T is an IComparable<T> itself. For instance, an IEnumerable<string> would be comparable to other IEnumerable<string>s because string is comparable, whereas an IEnumerable<object> would not, because object is not.

So with these two extensions in force, we can now compare two arrays of Levels:

using static Level;
Level[] a1 = { High, Low, High };
Level[] a2 = { High, Medium };
WriteLine(a1.CompareTo(a2));

Because of the LevelCompare extension on Level it satisfies the constraint on the EnumerableCompare extension on IEnumerable<Level>, which therefore in turns makes the Level[]s comparable!

Conclusion

Roles and extensions support proper separation of concerns, lightweight typing and adaptation, reducing dependencies and the costs of wrapping and conversion. There's a good amount of design work left to do, and in their full generality they are probably a fairly big feature to implement, requiring the participation of the runtime.

[HaloFour]

It's awesome to see some movement in this space.

I have some concerns of the loose typing with roles. I understand that it's intentional but it feels like it could very easily lead to situations where you're treating an instance as the incorrect role with undefined results. To me it also begs for a way to pattern match the underlying type to a given role, perhaps through a conditional member declared on that role. Otherwise how could you ever tell that a DataObject is a Customer vs an Order?

I'm also curious as to where roles/extensions fit into the conversation around shapes and structural typing. Assuming that interfaces remain the mechanism through which you define the constraint and this feature does hit the stretch goal of supporting extension implementation does it require manually writing a bridge/witness from an arbitrary type to that interface or can the compiler provide one given the appropriate extensions are in scope? I recall there was also a conversation about runtime changes necessary in order to avoid boxing the witness?

[MgSam]

I'm glad to see this thread (structural typing) being picked up again.

A few initial impressions:

• role and extension feel extremely similar to the point that it seems like there should be a way to marry the two concepts
• I think finding a way to "extend" interfaces onto existing types is where 99% of the value is and should be a core, not a stretch goal. I say this with an extensive background in TypeScript enjoying the benefits of structural typing. The ability to just use data as it is without copying it all over the place is really what this proposal is trying to achieve.
• I still feel that "no new instance state" for extensions/roles is too harsh of a restriction. The language and framework have the tools to do this (ConditionalWeakTable), choosing to make them hard to use because it might have performance implications if used incorrectly is too harsh of a decision. You guys did add dynamic into the language, after all. It doesn't come with a warning telling you not to use it because its expensive; it's just something you learn to use appropriately as a C# developer. As with any sharp tool, you need to know how to use it to not cut yourself. Instance state in extensions should be the same way. Give us the tools, let us make the decisions as to when to use them.
• I'm curious as to why the shapes proposal was apparently jettisoned wholesale. I don't recall it being discussed heavily in LDM but maybe I'm misremembering

[HaloFour]

@MgSam

I'm curious as to why the shapes proposal was apparently jettisoned wholesale. I don't recall it being discussed heavily in LDM but maybe I'm misremembering

I believe that was covered in the previous roles discussion that detailed the "Comparison to previous proposals".

The two major remaining downsides of the shapes proposal are: a) it still introduces a whole new abstraction mechanism, that in many ways competes with interfaces for expressing contracts. And b) for this reason, there is no way of adapting types to existing generic code that uses interfaces, not shapes, as constraints.

I chalk it up to being spaghetti thrown at the wall.

[YairHalberstadt]

The following two design aims seem difficult to reconcile, both semantics wise and implementation wise:

1. If R is a role/extension of T, there should be an identity conversion from a constructed type of T to a constructed type of R.
2. If R/T implements an interface, it should be able to be used as a type parameter which is constrained to that interface.

Here's a couple of examples where this doesn't make sense:

Example 1

Since there is an identity conversion between List<DataObject> and List<Customer>, this can only work by erasure - if you have to wrap or copy the instance of List<DataObject> to get that to work that's no longer an identity conversion.

This leads to the following unexpected behavior:

var dataObjects = new List<DataObject>{do1, do2};
var customers = (List<Customer>)new List<DataObject>();

Console.WriteLine(customers.GetType()); // prints "System.Collections.Generic.List`1[DataObject]"
Console.WriteLine(((object)customers) is List<Customer>); // prints "False"

Meanwhile if I use a role to fulfill a constraint this can't work through erasure. So if I have the following:

public class PersonList<T> : List<T> where T : IPerson

var customers = new PersonList<Customer>{d0, d1};

Console.WriteLine(customers.GetType()); // prints "PersonList`1[Customer]"
Console.WriteLine(((object)customers) is List<Customer>); // prints "True"

So there's this weird situation where the runtime type of a constructed type of a role/interface sometimes reflects that it's the role/interface and sometimes doesn't which I expect to be highly confusing.

Example 2

Consider the following:

class A {}

class NamedWrapper<T> where T : INamed
{
     public void PrintName() => Console.WriteLine(T.Name);
}

interface INamed
{
    public static string Name { get; }
}

namespace N1
{
    extension E1 : A, INamed
    {
        public static string Name => "Monty";
    } 
}

namespace N2
{
    extension E2 : A, INamed
    {
        public static string Name => "Python";
    } 
}

namespace N3
{
    using N1;
    public NamedWrapper<A> GetNamedWrapper()
    {
        var namedWrapper = new NamedWrapper<A>();
        namedWrapper.PrintName() // prints "Monty"
        return namedWrapper
    }
}

namespace N4
{
    using N2;
    using N3;

    public static class Program
    {
        public static void Main()
        {
             var namedWrapper = GetNamedWrapper();
             namedWrapper.PrintName(); // prints "Monty"
             var namedWrapper2 = (NamedWrapper<E2>)namedWrapper; // identity conversion
             namedWrapper2.PrintName(); // Should print "Python", but will in fact print "Monty" because it's an identity conversion
        }
    }
}

I think that the idea of having identity conversions between constructed types to constructed types of roles and interfaces just isn't really doable.

[En3Tho]

I agree with @MgSam about the ability to implement interfaces through type extension. I believe this should be the main goal. Because then it will allow decorating any type (and this js like type too) with static typing anyway (just declare an interface and extend that DataObject or dictionary or anything). But not vice versa.

[TahirAhmadov]

Wire data is most naturally represented with weakly-typed dictionary-like types such as DataObject, but those don’t lend themselves well to helpful tooling and elegant program logic. However, conversion to strong types is expensive and lossy, and all-out wrappers still introduce a level of indirection.

I disagree with this statement. The hypothetical DataObject has a dictionary of some sort behind the scenes. In the typical deserialization scenario, be it XML, JSON or binary, properties will be added to the dictionary. This incurs the standard dictionary costs: key lookup and behind the scenes allocations, as needed; plus the overhead during get operations. On the other hand, a properly implemented deserializer which creates an optimized type-specific method (using something like LINQ compilation or IL emit) can output code which sets target properties directly - no dictionary overhead; and the result is a strongly typed DTO, whose properties can be accessed natively with no overhead. I think it's a bad motivational example for the roles proposal.

[TahirAhmadov]

Looking at this more, it looks like the extension part of this is great, especially including the interface "adapters".
On the other hand, the role part is counter intuitive, introduces runtime difficulties, and is motivated by anti-patterns, IMO.
Even if we forget the DataObject scenario for a moment and look at strong types, role Customer for class Person has a code smell. If you have people, some of whom are customers and others are staff/vendors/etc., this needs to be stored somehow, both long term and in memory. These "roles" will have their specific attributes; for a customer, it may be shipping address, for an employee, it may be salary, etc. Therefore, these "roles" should really be their own types; trying to re-purpose Person as Customer just isn't a good solution.
And this is before you go full "enterprise" where a customer can be a company or a person, etc. - completely incompatible types.
In short, thumbs up for extensions and interfaces; thumbs down for roles.

[iam3yal]

I really like it but I do feel like it should be a single concept instead of having roles and extensions an extension should either extend an existing type or wrap an existing type what if in the example above I want to use JsonDataObject on Customer?

Here is an example of what I mean.

// Customer.cs
namespace Data;

public extension Customer : DataObject // Wrapper type
{
    public string Name => this["Name"].AsString();
    public string Address => this["Address"].AsString();
    public IEnumerable<Order> Orders => this["Orders"].AsEnumerable();
}

// JsonDataObject.cs
namespace Data;

using JsonLibrary;

public extension JsonDataObject : DataObject // Extension type
{
    public string ToJson() { … this … }
    public static T FromJson<T>(string json) { … }
}

// Program.cs / Main method
using Data;
using Data.JsonDataObject; // Importing a specific extension type
using Data.*; // Importing all extensions types in the namespace Data

var customer = customer.FromJson<Customer>(args[0]);
WriteLine(customer.ToJson());

[CyrusNajmabadi]

I want to use JsonDataObject on Customer?

I don't see why you wouldn't be able to. a Customer is a DataObject (both in terms of type hierarchy, and in terms of actual raw erased value). So all of JsonDataObject would apply to it.

[iam3yal]

@CyrusNajmabadi Okay thank you but still I think and feel like extensions and roles can be a single concept but applied differently but then that's just my opinion.

[CyrusNajmabadi]

@eyalalonn We just had an LDM meeting on htis. Trust me that that sentiment is understood and we're definitely going to be thinking heavily about these concepts and hte best way to expose them :)