[Proposal][Roslyn] Add structural typing support

[glennblock]

This proposal is based on discussions with @MadsTorgersen and @davidfowl. The design incorporates feedback from @crmckenzie and @edandersen.

In golang interfaces are structural and implicit. Once an interface is declared, all go types that have the members defined in the interface present, automatically become implementers of that interface. This provides benefits for testability and extensibility.

C# interfaces today require the implementer to explicitly implement the interface. C# does have the dynamic keyword however when you use dynamic, you lose all compile time checking / type safety. Interfaces in golang however allow retaining type safety while reducing type coupling (because the implemented does not have to explicitly implement the interface).

The proposal is to add structurally typed interface support to C#.

Overview

Here is an example of how interfaces work in go.

//Based on https://tour.golang.org/methods/5, but further simplified
type Writer interface {
    Write(b []byte) (n int, err error)
}

func output(w Writer, msg string) {
  ... 
}

func main() {
    var w Writer

    // os.Stdout implements Writer
    w = os.Stdout
    output(w, "Hello World");
}

The user's code defines a Writer interface with a Write method. Next an output function is defined which accepts a Writer. Finally in the main function the code then creates a new Writer instance and sets the built in go os.Stdout object to that instance. The setting of the value succeeds though the Stdout object had no prior knowledge of the Writer interface.

From a testability perspective this is extremely nice. If the user wants to test the output function, he/she can easily provide a mock Writer. It is not required to use any special mocking tools, nor do custom adapters and wrapper classes have to be created to wrap the native framework objects to make them testable. This will be beneficial for testing sealed classes, or non-virtual members.

From an extensibility perspective this is also nice as new Writers can easily be plugged in simply by defining new types with the required members.

Proposal

Allow add structural typing support for interfaces in .NET. Several variant approaches are proposed.

Benefits

• Allows existing .NET classes to be more easily tested.
• Implementers of libraries have to jump through less hoops to make their libraries testable.
• Simplifies authoring of 3rd party extensions, removes the need for a shared library reference.
• Enables new scenarios for anonymous types

Variant 1 - Structural keyword

One way to do this would be to introduce a new structural keyword which defines a structural interface. A structural is similar to a standard interface behavior-wise with one big exception, it is implicitly implemented by any class that has the members defined in the structural interface. The members of the interface can be any valid reference or value type.

Example

Below is a sample C# snippet illustrating a scenario where this could be used. The example is more complicated than the go example for illustration of a real world pain point. It uses System.Web.HttpRequest, a class that today is often difficult to test against because it is sealed. Today testing is improved through helper/wrapper classes like System.Web.HttpRequestBase and System.Web.HttpRequestWrapper. These put a burden on frameworks to support these helpers and they add complexity.

Let's assume a library has a method which performs validation against the current request.

public structural IHttpRequest() {
    string UserAgent {get;}
}

public class RequestValidator {
    private IHttpRequest _req;

    public RequestValidator(IHttpRequest req) {
        _req = req;    
    }

    public void ValidateUserAgent() {
        var ua = _req.UserAgent;
        if (ua == null) {
          throw new ArgumentException("UserAgent", "User Agent must be provided");
        }
    }
}

//in prod code somenwhere
public class Handler {
    public void HandleRequest(HttpContext context) {
        var req = context.Request;
        var validator = new Validator(req);
        validator.ValidateUserAgent();
    }
}

//unit test, checks to see the QueryString is accessed.
using Xunit;
using Should;

public class ValidatorTests {
    [Fact]
    public void ValidateUserAgentThrowsExceptionWhenUserAgentIsNull() {
        var mock = new Mock<IHttpRequest>();
        mock.SetupGet(r=>r.UserAgent).Returns(null);
        var validator = new RequestValidator(mock.Object);
        var ex = Assert.Throws<ArgumentException>(()=>validator.ValidateUserAgent());
        ex.Message.ShouldEqual("User Agent must be provided");  
    }

    [Fact]
    public void ValidateUserAgentShouldSucceedWhenUserAgentIsNotNull() {
        var mock = new Mock<IHttpRequest>();
        mock.SetupGet(r=>r.UserAgent).Returns("foo");
        var validator = new RequestValidator(mock.Object);
        validator.ValidateUserAgent();
    }
}

A few details about the code

• IHttpRequest defines a structural interface. This was created by the library author and and is not part of the .NET framework.
• RequestValidator is the system under test. It validates the UserAgent on the request and accepts an IHttpRequest instance.
• Handler is a class deployed to production which passes in the current HttpContext.Request. HttpRequest becomes an implicit implementer of IHttpRequest as it matches on members.
• The unit tests mocks the request using the structural interface in order to test the Handler class.
• No wrapper or utility types are needed i.e. HttpRequestBase, HttpRequestWrapper.

The high level benefit illustrated here is being able to easily test the RequestValidator class though the types that it relies on themselves were not designed in a testable manner.

Variant 2 - Structural attribute at the callsite (suggested by @crmckenzie)

In this variant the implementer of a method declares in the definition that one or more members are structurally matched. This is done via the useage of a new [Structural] attribute which is applied to a member that has an interface-type. Similar to Variant 1, all objects which have the members defined in the interface present, will match.

Note: Once the instance has been received within the method, it can be set to a variable of the same interface type or passed as a parameter to a method that accepts a parameter of that interface whether it is marked structural or not. This is in order to remove the need to declare structural in a viral manner throughout the code.

Example:

public IHttpRequest() {
    string UserAgent {get;}
}

public class RequestValidator {
    //not necessary to be marked as Structural
    private IHttpRequest _req;

    public RequestValidator([Structural] IHttpRequest req) {
        _req = req;    
    }

    public void ValidateUserAgent() {
        var ua = _req.UserAgent;
        if (ua == null) {
          throw new ArgumentException("UserAgent", "User Agent must be provided");
        }
    }
}

//in prod code somenwhere
public class Handler {
    public void HandleRequest(HttpContext context) {
        var req = context.Request;
        var validator = new Validator(req);
        validator.ValidateUserAgent();
    }
}

//unit test, checks to see the QueryString is accessed.
using Xunit;
using Should;

public class ValidatorTests {
    [Fact]
    public void ValidateUserAgentThrowsExceptionWhenUserAgentIsNull() {
        var mock = new Mock<IHttpRequest>();
        mock.SetupGet(r=>r.UserAgent).Returns(null);
        var validator = new RequestValidator(mock.Object);
        var ex = Assert.Throws<ArgumentException>(()=>validator.ValidateUserAgent());
        ex.Message.ShouldEqual("User Agent must be provided");  
    }

    [Fact]
    public void ValidateUserAgentShouldSucceedWhenUserAgentIsNotNull() {
        var mock = new Mock<IHttpRequest>();
        mock.SetupGet(r=>r.UserAgent).Returns("foo");
        var validator = new RequestValidator(mock.Object);
        validator.ValidateUserAgent();
    }
}

Differences between this code and Variant 1.

• IHttpRequest defines a standard interface rather than a Structural.
• RequestValidator defines the req parameter as [Structural]

Variant 3 - All interfaces are implicitly matched on structure, does not require any specific annotation or types. (suggested by @clipperhouse)

With this variant, the suggested functionality will just work without any special type or annotation required.

Any object that matches an interface on structure will automatically implement that interface. This aligns closest to how go does it.

Questions

• Q: Will general casting work i.e. I can cast Foo to IFoo simply because Foo has the members?
• Q: Should the structural match be recursive? Meaning StructuralA can have params of type StructuralB and StructuralC?
• Q: How does this work with the type system. What will Type.GetInterfaces return?

Notes

• A completely diff approach is to allow defining the expected interface inline within the method using a declarative syntax. Scala supports both approaches.

[clipperhouse]

A question, perhaps naive: why is a keyword necessary? Couldn’t the interface satisfaction simply be inferred, as in Go? For the developer, the feature essentially means the “implements” declaration (after the colon) becomes optional.

[glennblock]

That's a really good question. I guess my main reason for the keyword /
attribute was to make it more opt/in and explicit and being cautious about
any side effects in existing code. However you make a good point that in go
it applies to all interfaces so why not the same behavior. I am fine either
way.
On Thu, Jan 7, 2016 at 6:29 PM Matt Sherman notifications@github.com
wrote:

A question, perhaps naive: why is a keyword necessary? Couldn’t the
interface satisfaction simply be inferred, as in Go? For the developer, the
feature essentially means the “implements” declaration (after the colon)
becomes optional.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/roslyn/issues/7844#issuecomment-169871649.

[HaloFour]

Making classes automatically start to implement interfaces just by happening to match the required members could lead to a lot of problems with existing code cases.

The typing is controlled by the CLR which provides no form of structural equivalence like this. How would C# itself overcome this? The only thing I can think of would be for C# to recognize those situation in which you're trying to assign a reference of a Duck to something that requires an IDuck in which case the compiler would automatically emit a wrapper class which implements the interface and forwards the calls to the matching members:

Given the following interface and class:

public interface IDuck {
    void Quack();
}

public class Duck {
    void Quack() {
        Console.WriteLine("quack!");
    }
}

The following:

Duck a = new Duck();
IDuck b = a;

Would be converted into the following:

// compiler generated
private class DuckWrapper : IDuck {
    private readonly Duck parent;

    public DuckWrapper(Duck parent) { this.parent = parent; }

    public void Quack() {
        this.parent.Quack();
    }
}

Duck a = new Duck();
IDuck b = new DuckWrapper(a);

This has several obvious problems. The first of which is the performance impact from two virtual calls. This would be a lot lighter than dynamic dispatch, that's for sure, but it's still a hit. The bigger problem is that quite unintuitively b could not be cast back into a Duck despite the fact that it appears that it should be one.

[MatthieuMEZIL]

@HaloFour: It is something that I also would like to see in C#, particularly with generic constraint.

[glennblock]

@HaloFour you make some valid points. My suggestion here was not to
automatically wrap, it would need a new first class mechanism. It could not
lose information as that would be self-defeating.
On Thu, Jan 7, 2016 at 9:37 PM Matthieu MEZIL (MSFT) <
notifications@github.com> wrote:

@HaloFour https://github.com/HaloFour: It is something that I also
would like to see in C#, particularly with generic constraint.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/roslyn/issues/7844#issuecomment-169901094.

[paulomorgado]

@glennblock, since you're talking about compile-time checking, wouldn't something like dynamic(SomeType) do the job?

The emitted code would be the same as now, but it would be statically typed as SomeType. With all the analysis and tooling benefits.

[glennblock]

@paulomorgado improved testability and extensibility is the main goal with compiler safety being a benefit of the outlined approached.

I am not familiar with using dynamic in that style, is that a c# 6 lang feature?

[paulomorgado]

@glennblock, no one is familiar with that style because it doesn't exist. Yet! 😄

[glennblock]

Oh you are proposing this as an alternative.
On Fri, Jan 8, 2016 at 1:19 AM Paulo Morgado notifications@github.com
wrote:

@glennblock https://github.com/glennblock, no one is familiar with that
style because it doesn't exist. Yet! [image: 😄]

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/roslyn/issues/7844#issuecomment-169943960.

[paulomorgado]

No! I'm just asking you. I've proposed that so many times that I won't bother again.

[svick]

For reference, I believe @paulomorgado is talking about #3012.

[HaloFour]

@glennblock

I would agree, which would mean that the CLR itself would have to support some notion of transparent duck typing.

@MatthieuMEZIL

As in permitting a type to be used as a generic type argument even if it does not meet an interface generic type constraint? Again, fairly certain that would require CLR support since it is the CLR that will enforce that constraint at runtime.

I also think that it's important to separate "being" an IDuck from "looking like" an IDuck so that nominal typing remains intact. Otherwise marker interfaces would completely lose their utility. Not to mention I think a lot of existing code would be severely impacted in very subtle ways.

[glennblock]

Yes it would probably need CLR support to do it right. It raisss some interesting questions. For example is the interface actually appended to the interface collection of the type or dynamically calculated? If I access Type.GetInterfaces will it be returned? Can I do simple casts? Like cast Bar as an IFoo as long as it matched on members?

[HaloFour]

@glennblock

If I access Type.GetInterfaces will it be returned?

I seriously doubt it. The CLR would have to scan all interfaces in every assembly of the current AppDomain to figure that out. Seems wasteful. Not to mention that would break the semantics of existing code. I'd imagine you'd need a new reflection method which would compare a type against a specified interface for compatibility.

I could only guess as to how this could be implemented in the CLR. Maybe you'd want to open a proposal on CoreCLR and see where that goes.

[clipperhouse]

The implemented interfaces could be determined entirely at compile time, no? There is an expense but it would not need to be at runtime. (Pls correct me.) Dynamically loaded .dll’s would be challenging, not sure of what those guarantees should be…

As far as reflection, it’s true that Type.GetInterfaces could return different results. Maybe an overload to indicate whether to return implicits, defaulting to false.

[Thaina]

I think the way it should work is apply it to generic constraint interface

I mean Baz(new Bar())
should not work when void Baz(IFoo foo) { foo.DoSomething(); }
But it should, at compile time, work with void Baz<I>(I foo) where I : IFoo { foo.DoSomething(); }

When we do duck type it should not use as. But for it to work without breaking backward compatibility. I think to limit it at constraint function might work

[davidfowl]

The generic constraints are very Scala like but I'm not sure it would scale with c# currently very limited type inference. You'll eventually hit a wall where you'll need to specify the T explicitly:

TResult GetRequestThing<THttpRequest, TResult>(THttpRequest request) where THttpRequest : IHttpRequest { }

The call site now has to specify TResult and THttpRequest at every callsite.

GetRequestThing<HttpRequest, string>(request)

Not to mention that generics are viral. What happens when I want to take a duck typed thing as a ctor argument? Do I now need s generic class so I can put the constraint somewhere?

[glennblock]

Agreed. My hope was if the language addressed this it would not introduce
another viral construct. I am talking to you IDisposable.

On Wed, Feb 17, 2016 at 12:56 AM David Fowler notifications@github.com
wrote:

The generic constraints are very Scala like but I'm not sure it would
scale with c# currently very limited type inference. You'll eventually hit
a wall where you'll need to specify the T explicitly:

TResult GetRequestThing<THttpRequest, TResult>(THttpRequest request) where THttpRequest : IHttpRequest { }

The call site now has to specify TResult and THttpRequest at every
callsite.

GetRequestThing<HttpRequest, string>(request)

Not to mention that generics are viral. What happens when I want to take a
duck typed thing as a ctor argument? Do I now need s generic class so I can
put the constraint somewhere?

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/roslyn/issues/7844#issuecomment-185103207.

[rsmckinney]

The discussion here is very similar to one I struggled with a while back, mostly in my own head, and finally decided to confront it. I wrote up a post here you might find helpful: http://gosu-lang.github.io/2014/04/22/structural-types-in-gosu.html

I implemented structural typing for the Gosu language pretty much as the paper describes. Although we have since added more support for it, mostly in areas such as generic variance inference and so forth, which I've also written a bit about here: http://gosu-lang.github.io/2015/11/22/threading-the-needle.html

Cheers.

[glennblock]

" // With new features that make java fun again"

👍

[masaeedu]

@HaloFour Fair points regarding the runtime impact, but wouldn't there be a way to implement this so that the existence of the interface is erased using information from the assignment location? It would require some changes to the CLR so that an appropriate implementation is JIT-ed out using the structurally compatible concrete type, but this is very similar to how generics work in the CLR already.

Imagine for example a more limited variant of this proposal where instead of the ability to declare and use structural interfaces everywhere, you only gained the ability to specify structural type constraints for generic type parameters. Then this becomes mostly an exercise in type checking, and you wouldn't need to do much work outside Roslyn.

[iam3yal]

@glennblock

Maybe, just as yet another alternative, we can have something like this:

public interface IHttpRequest {
}

public type THttpRequest as IHttpRequest;

The benefit of this is that you can gradually update the implementation without breaking anything and opt-in/out based on the convention of the name and you wouldn't need to change existing interfaces, sometimes you might not even own them and you may still want to introduce a structural version, maybe through extension methods or something.

Finally, the method would look like this:

public RequestValidator(THttpRequest req) {
    _req = req;    
}

This method would accept both instances that implements IHttpRequest explicitly and these who have the shape of IHttpRequest.

I'm not sure about overload resolution but because the name of the type is different it seems like it would be possible to add an overloaded version that support structural typing.

I think that because structural typing isn't the default in C# it should always be explicit but this doesn't mean it needs to be verbose.

[Opiumtm]

Structural typing essentially is a case of extension of already existing class or interface with another (defined elsewhere) interface. It more like extension methods, but this time it's an extension interface.

Structural typing (do not confuse it with duck typing) is a way to provide polymorphism when it isn't provided by already existing objects. It's a way of bunching together under some interface umbrella classes which doesn't know about your interface as they're defined elsewhere.

When you have full control over class hierarchy, you can define interface and implement it by all your objects and it will do what you want without any need of structural typing.

So, desire for structural typing is essentially have roots in limitations of interfaces.
As for now, interface is implemented "nominally" - you have to explicitly declare that you support interface. It's OK when your interface is defined alongside your classes or when interface is considered "standard" and a part of some common code foundation.

So, to overcome this interface limitations, there would be some means to extend existing types with interfaces defined later and by other people in other code, probably with the support of some additional adapter logic. If some class provide exactly required functionality, but have different member names - adapter is required.

Structural typing as it defined have one very important limitation - it require nominal identity of class/interface members. Your members have to be named exactly as required by structural interface consumer. It's a big "structural typing" limitation.

These problems of "later polymorphism" (introducing polymorphism when you reference older code on which you have no control and can not bind interface nominally on already existing in binary form class) can be solved (and indeed is solved by any developer) by creating of custom wrappers or adapters in code. Such a wrappers often require lots of boilerplate code.

Requirement to write lots of boilerplate code for custom wrappers is why people so desperately want structural typing.

Compiler at compile time can generate those wrappers and developer would be saved from lots of boilerplate code. It could be implemented as a part of "extension everything" feature maybe. If "extensions" feature would allow to bind interface to already existing class or interface, probably with some customization and with zero boilerplate code when class members already nominally compatible (have exact same types and names) with bound interface - it would solve "structural typing" proposal and would introduce much powerful features than plain "structural typing".

So, it should look like this:

1. If type is already nominally compatible with extension interface (all required by extension interface members have same names and same types), developer should write zero boilerplate code. Default extension logic should work.
2. Default extension logic should be customizable to any degree. But wrapper complexity in code should be dependent on member list or internal logic difference between types, not on the count of target type members.
3. This also would solve entire class of "object-to-object mapping" common task. When system is divided into isolated layers (Data access layer, business logic layer and so on), object hierarchies often very similar on different layers but not always can implement same interface - their public API should be different, but not completely different. So, "object-to-object mapping" is required and it typically have 10-20% of actual meaningful code and 80-90% of boilerplate code.
4. This should solve "aspect programming" problem as same interface extensions (extension of interface into self adding some intermediate custom code provided by extension with the same "you implement only differences, not all members burden") can be layered.

To sum up said above

Core problem is a requirement to write lots of boilerplate code when you implement custom wrappers/proxies by bare hands in code. It's a problem that should be fought. "Structural typing" proposal is a special case of this.
As of today when you develop wrapper/adapter around already existing type, you should proxy all its members even if it's utterly trivial - just a plain call of same-named underlying member.
As this is quite common task, "structural typing" is often proposed as an universal salvation from this wrapper boilerplate code hell.
But "structural typing" have its own drawbacks and limitations. Requirement to nominal member compatibility, for example. When you have class that almost same as required structural interface, but have some small differences (one member having different return type of one member named differently), structural typing would give you exactly nothing and you would anyway develop custom wrapper/adapter and would experience same boilerplate code hell.

So, correct solution is to develop language features that would help you extend types, provide adapters/wrappers resolving only differences between extended type and extension. Extending existing type (so, generating wrapper class and implicit conversion operator) with interface when both types are structurally compatible should be one-liner as compiler already know all information to provide trivial wrapping. When there are differences, developer should resolve them in code. Only differences, not all trivial members.

[masaeedu]

@Opiumtm

When you have class that almost same as required structural interface, but have some small differences (one member having different return type of one member named differently), structural typing would give you exactly nothing

Structural typing is usually accompanied by the ability to destructure and restructure types, so this is a moot point.

[panost]

If the interface has one method, you can achieve that by using a delegate.
If the interface has more than one method, you can't
So what you need is a multi-method delegate (or a delegated interface) implemented in CLR

The structure of this will be similar to the structure of a Delegate
One member will be the object it's self (see Target property in Delegate)
And another member will be (instead of a method pointer) a pointer to the interface's VMT (virtual method table).Probably this will not be directly exposed with a property (no Methods property)

You can declare those types by using some generic declaration similar to the nullable types or you can use a special character, ! for example.

interface IDuck {
  void Quack(int times); // 1st slot in the VMT
  void Swim(int miles); // 2nd slot
}
void doDucks(IDuck! duck) {
  duck.Swim(15); // will invoke the 2 slot of the VMT using the same methology as if it was a delegate
}
class Duck {
  void Quack(int times) {...}
  void Swim(int miles) {...}
}
var d = new Duck();
doDucks(d);

How can an object create a delegated interface:

1. If the object already implements the interface, the VMT already exists, so the operation is fast, a simple assignment
2. If the object does't implement the interface, we have to create a dynamic VMT. We may need to cache those VMTs for speed up

Perhaps we may need a static method similar to Delegate.CreateDelegate that will be invoked transparently from the C# compiler. Let's say it's named Interface.CreateInterface with the following signature

T! Interface.CreateInterface<T>(object obj)

Since we have this, nothing prevent us from creating a VMT entirely on the fly, using an Interface.CreateInterface overload that accepts an array of delegates

var writer = File.CreateText( @"C:\log.txt" );
IDuck! d = Interface.CreateInterface<IDuck >(writer , // or null which means we have no object instance similar to delegates on static methods
(int times)=> writer.WriteLine("Quack:"+times),
(int miles)=> writer.WriteLine("Swim:"+miles) );
doDucks(d);

Benefits:

1. You don't need any special interface declaration. All interfaces can use it.
2. Most of the implementation code is there, already used by delegates
3. You don't lose the information of the original object. A property named Target (same as in Delegate) will be there. The compiler can transparently use it in cast expressions
4. I expect that using delegated interfaces will be much faster than normal interfaces, since you don't have to find the VMT slot each time you need to make a call

[HaloFour]

I think that this is conceptually covered by shapes/extensions:

#164

[MatheusLFreitas]

Would be possible to use as a anonymous generic type specified as a new generic constraint?

class Foo {

  void Bar<T>(T whatever)
      where T : class, //or struct
                         IVerifiable, //optionally
                        {
                              string Name;
                              bool Bar(decimal x);
                         }
   { 
        var barred = whatever.Bar(10m);
        Console.WriteLine(whatever.Name);
   }
}

Interface IVerifiable {
 bool Valid { get; set; }
}

class Usage : IVerifiable {
     public string Name { get; set; }
     public bool Valid { get; set; }
     public bool Bar(decimal d) => true;
     public bool Bar(double d) => false;
}

var f = new Foo();
f.Bar( new Usage());

The ideia is to be complementary with another constraints if present eg, interfaces and inheritance.

[emperador-ming]

Here it is a real use case not related to testing:

public IEnumerable<IParameter> GetParameters(Guid proposald) =>
        _context.Parameters
            .ByProposalId(propuestaId)
            .Select(p => new ParameterDto(p.ProposalId, p.User, p.Value));

private record ParameterDto(Guid ProposalId, User User, ParameterValue Value) : IParameter;

With structural typing:

public IEnumerable<IParameter> GetParameters(Guid proposald) =>
        _context.Parameters
            .ByProposalId(propuestaId)
            .Select(p => new 
            { 
                  p.ProposalId, 
                  p.User, 
                  p.Value
            });

[hez2010]

Superseded by #5497
Note that the Go style structural typing (ducking type) is extremely dangerous and anti-pattern because it can cause interfaces being implemented by types unexpectedly, even if those types do not want to implement an interface.

[emperador-ming]

I can see a problem with Marker Types. However, structural typing is enforced during static analysis by the compiler, whereas duck typing is a runtime non-safe phenomenon.

[johnknoop]

Variant 2 sounds a lot like this suggestion I posted two years ago. I wasn't aware of this discussion. I truly hope it can becomes reality one day.