Concepts and type-checking generics

[Araq]

Concept redesign

Goals:

• Empower concepts to make generic code type-checkable at declaration time,
  not only at instantiation time.
• Make concepts easier to understand and easier to implement.
• Provide an escape hatch in generic code so that e.g. adding a
  debug echo or log statement does not require a far reaching type
  constraint addition.
• Old code with unconstrainted generic parameters keeps to work.
• Do not base concept's implementation on system.compiles which is
  under-specified in Nim, very tied to the current implementation, and
  finally, slow.

Non goals:

• Support every little detail that the old concept design supported. For this
  we have the escape hatch plus the fact that you can leave your T
  underspecified.
• Support accidentical features. ("Look! With this hack I can specify that the
  proc needs a cdecl calling convention!")
• Support "cross parameter" constraints like "sizeof(a) == sizeof(b)". I have
  yet to see convincing examples of these cases. In the worst case, we could
  add enableif for this without bloating the concept's design.
• Turning concepts into "interfaces". This can already be accomplished with
  macros. Having said that, since these concepts are declarative, they are
  easier to process with tooling and arguably easier to process with macros
  as well.

Atoms and containers

Concepts come in two forms: Atoms and containers. A container is a generic
concept like Iterable[T], an atom always lacks any kind of generic
parameter (as in Comparable).

Syntactically a concept consists of a list of proc and iterator
declarations. There are 3 syntatic additions:

• Self is a builtin type within the concept's body stands for the
  current concept.
• each is used to introduce a generic parameter T within the
  concept's body that is not listed within the concept's generic
  parameter list.
• either orelse is used to provide basic support for optional
  procs within a concept.

We will see how these are used in the examples.

Atoms

  type
    Comparable = concept # no T, an atom
      proc cmp(a, b: Self): int

    ToStringable = concept
      proc `$`(a: Self): string

    Hashable = concept
      proc hash(x: Self): int
      proc `==`(x, y: Self): bool

    Swapable = concept
      proc swap(x, y: var Self)

Self stands for the currently defined concept itself. It is used to avoid
a recursion, proc cmp(a, b: Comparable): int is invalid.

Containers

A container has at least one generic parameter (most often called T). The
first syntactic usage of the generic parameter specifies how to infer and bind T.
Other usages of T are then checked to match what it was bound to.

  type
    Indexable[T] = concept # has a T, a collection
      proc `[]`(a: Self; index: int): T # we need to describe how to infer 'T'
      # and then we can use the 'T' and it must match:
      proc `[]=`(a: var Self; index: int; value: T)
      proc len(a: Self): int

Nothing interesting happens when we use multiple generic parameters:

  type
    Dictionary[K, V] = concept
      proc `[]`(a: Self; key: K): V
      proc `[]=`(a: var Self; key: K; value: V)

The usual ": Constraint" syntax can be used to add generic constraints to
the involved generic parameters:

  type
    Dictionary[K: Hashable; V] = concept
      proc `[]`(a: Self; key: K): V
      proc `[]=`(a: var Self; key: K; value: V)

each T

Note: each T is currently not implemented.

each T allows to introduce generic parameters that are not part of a
concept's generic parameter list. It is furthermore a special case to
allow for the common "every field has to fulfill property P" scenario:

  type
    Serializable = concept
      iterator fieldPairs(x: Self): (string, each T)
      proc write(x: T)


  proc writeStuff[T: Serializable](x: T) =
    for name, field in fieldPairs(x):
      write name
      write field

either orelse

Note: either orelse is currently not implemented.

In generic code it's often desirable to specialize the code in an ad-hoc manner.
system.addQuoted is an example of this:

  proc addQuoted[T](dest: var string; x: T) =
    when compiles(dest.add(x)):
      dest.add(x)
    else:
      dest.add($x)

If we want to describe T with a concept we need some way to describe optional
aspects. either orelse can be used:

  type
    Quotable = concept
      either:
        proc `$`(x: Self): string
      orelse:
        proc add(s: var string; elem: self)

  proc addQuoted[T: Quotable](s: var string; x: T) =
    when compiles(s.add(x)):
      s.add(x)
    else:
      s.add($x)

More examples

system.find

It's straight-forward:

  type
    Findable[T] = concept
      iterator items(x: Self): T
      proc `==`(a, b: T): bool

  proc find(x: Findable[T]; elem: T): int =
    var i = 0
    for a in x:
      if a == elem: return i
      inc i
    return -1

Sortable

Note that a declaration like

  type
    Sortable[T] = Indexable[T] and T is Comparable and T is Swapable

is possible but unwise. The reason is that Indexable either contains
too many procs we don't need or accessors that are slightly off as they don't
offer the right kind of mutability access.

Here is the proper definition:

  type
    Sortable[T] = concept
      proc `[]`(a: var Self; b: int): var T
      proc len(a: Self): int
      proc swap(x, y: var T)
      proc cmp(a, b: T): int

Concept matching

A type T matches a concept C if every proc and iterator header
H of C matches an entity E in the current scope.

The matching process is forgiving:

• If H is a proc, E can be a proc, a func, a method, a template,
  a converter or a macro. E can have more parameters than H as long
  as these parameters have default values. The parameter names do not have
  to match.

• If H has the form proc p(x: Self): T then E can be a public
  object field of name p and of type T.

• If H is an iterator, E must be an iterator too, but E's parameter
  names do not have to match and it can have additional default parameters.

Escape hatch

Generic routines that have at least one concept parameter are type-checked at declaration time. To disable type-checking in certain code sections an untyped block can be used:

proc sort(x: var Sortable) = 
   ... 
   # damn this sort doesn't work, let's find out why:
   untyped:
      # no need to change 'Sortable' so that it mentions '$' for the involved
      # element type!
      echo x[i], " ", x[j]

EDITED 2021/03/09 self was renamed to Self and is what the experimental implementation uses.

[andreaferretti]

I would like to understand how to write the concepts in Emmy under this proposal. Currently they look like

type
  AdditiveMonoid* = concept x, y, type T
    x + y is T
    zero(T) is T
  AdditiveGroup* = concept x, y, type T
    T is AdditiveMonoid
    -x is T
    x - y is T
  MultiplicativeMonoid* = concept x, y, type T
    x * y is T
    id(T) is T
  MultiplicativeGroup* = concept x, y, type T
    T is MultiplicativeMonoid
    x / y is T
  Ring* = concept type T
    T is AdditiveGroup
    T is MultiplicativeMonoid
  EuclideanRing* = concept x, y, type T
    T is Ring
    x div y is T
    x mod y is T
  Field* = concept type T
    T is Ring
    T is MultiplicativeGroup

I guess they would become like this?

type
  AdditiveMonoid* = concept
    proc `+`(x, y: self): self
    proc zero(T: typedesc[self]): self
  AdditiveGroup* = concept
    self is AdditiveMonoid # will concept refinement be allowed?
    proc `-`(x: self): self
    proc `-`(x, y: self): self
  MultiplicativeMonoid* = concept
    proc `*`(x, y: self): self
    proc id(T: typedesc[self]): self
  MultiplicativeGroup* = concept
    self is MultiplicativeMonoid
    proc `/`(x, y: self): self
  Ring* = AdditiveGroup and MultiplicativeMonoid
  EuclideanRing* = concept
    self is Ring
    proc `div`(x, y: self): self
    proc `mod`(x, y: self): self
  Field* = Ring and MultiplicativeGroup

[Araq]

I guess they would become like this?

Yes.

self is AdditiveMonoid # will concept refinement be allowed?

I thought about it and we could support it via type AdditiveGroup = concept of AdditiveMonoid. I don't know if it is important enough that we really need it.

[mratsim]

I like the proposal. Can it deal with recursivity? That's important for my needs (mratsim/Arraymancer#299)

In Arraymancer I define a deep learning trainable layer like this but I don't use them at the moment as it's too brittle:

https://github.com/mratsim/Arraymancer/blob/bde79d2f73b71ece719526a7b39f03bb100784b0/src/nn_dsl/dsl_types.nim#L63-L81

type
  Variable[T] = object
    value: Tensor[T]
    gradient: Tensor[T]

  Tensor[T] = object
    data: seq[T]

  TrainableLayer*[TT] = concept layer
    block:
      var trainable: false
      for field in fields(layer):
        trainable = trainable or (field is Variable[TT])
      trainable

  Conv2DLayer*[TT] = object
    weight*: Variable[TT]
    bias*: Variable[TT]
  LinearLayer*[TT] = object
    weight*: Variable[TT]
    bias*: Variable[TT]
  GRULayer*[TT] = object
    W3s0*, W3sN*: Variable[TT]
    U3s*: Variable[TT]
    bW3s*, bU3s*: Variable[TT]
  EmbeddingLayer*[TT] = object
    weight*: Variable[TT]

#########
# sanity check

var convolution: Conv2DLayer[float32]
echo convolution is TrainableLayer[float32] # Why false?

#########
# user-defined

type
  MyNonLayer = object
    foo: int
    bar: float32
  MyLayer[TT] = object
    metadata: MyNonLayer
    weight: Variable[TT]

var x: MyNonLayer
var y: MyLayer[float32]

echo x is TrainableLayer[float32]
echo y is TrainableLayer[float32] # Why false

[Araq]

What does that even mean though? "Somewhere in the object type there must be a field of type Variable[T]"? How do you know which one? What if you have more than one?

[zah]

A) How come I must repeat in every discussion about concepts that any proposal must cover the requirements of associated types and constants? I guess it would have to be:

type Foo = concept
  proc ElemType(_: typedesc[self]): typedesc
  proc compressedSize(_: typedesc[self]): static[int]

I find it a bit funny how these result in syntax that is outlawed in regular Nim. And hey, why was proc chosen here? Isn't using func the recommended way for new Nim code?

---

B) A "container concept" must be able to infer static parameter values as well. For example

type FixedSizeMatrix[M, N: static[int]; T] = concept
  # How do I infer M and N here from a non-generic type?
  # In the current design it's done with:
  x.Rows == M
  x.Cols == N 

Please not that Rows and Cols can be templates defined over a non-generic type in order to adapt it to the concept requirements:

template Rows(T: type Matrix4x4): int = 4

Having the FixedSizeMatrix concept above, how do I define a different concept called SquareMatrix (where the rows and columns must be the same)? There are some algorithms that work only on square matrices.

---

C) The VTable types are a very important feature in the current concepts design, but I would agree that they can be delivered as a library.

---

D) I'd really like to have the converter type classes though and these require custom code in semcall/sigmatch.

---

E) Regarding the main idea here, mainly the type checking of generic functions, I would approach this with a lot of caution. The internal compiler structures can be defined and derived from the current concept syntax. I think it makes a lot of sense to try to experiment with the approach before making any major decisions about removing the existing concepts-related code in the compiler.

In particular, a lot of attention has to be put into code like the following:

https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz/types.nim#L85-L101

func isFixedSize*(T0: type): bool {.compileTime.} =
  mixin toSszType, enumAllSerializedFields

  when T0 is openarray|Option|ref|ptr:
    return false
  else:
    type T = type toSszType(default T0)

    when T is BasicType:
      return true
    elif T is array:
      return isFixedSize(ElemType(T))
    elif T is object|tuple:
      enumAllSerializedFields(T):
        when not isFixedSize(FieldType):
          return false
      return true

Notice how the function accepts an underspecified type, but it discovers a more detailed type using the is operator. Then the function is able to use operations defined over this more detailed type. This is a very convenient construct enabled by the current lax rules of Nim and we should not lose it.

[mratsim]

What does that even mean though? "Somewhere in the object type there must be a field of type Variable[T]"? How do you know which one? What if you have more than one?

I don't care which field, but as long as there is a "Variable[T]" field, the user-defined type constitutes a TrainableLayer indeed. All deep-learning libraries work like that, except that they use dynamic dispatch and inheritance from a Layer class.

[Araq]

How come I must repeat in every discussion about concepts that any proposal must cover the requirements of associated types and constants?

Because it's covered by the proposal as you noticed yourself.

Isn't using func the recommended way for new Nim code?

func could be allowed but then require func'ness in the matching process. Though I think it doesn't matter. ;-)

Static T could be done like so:

type FixedSizeMatrix[M, N: static[int]; T] = concept
  proc rows(x: self): M
  proc cols(x: self): N

I'd really like to have the converter type classes though and these require custom code in semcall/sigmatch.

Well the proposal implies that we have dedicated, custom code for concept matching, moreso than currently.

I think it makes a lot of sense to try to experiment with the approach before making any major decisions about removing the existing concepts-related code in the compiler.

Of course.

In particular, a lot of attention has to be put into code like the following:

That code is not affected, under-constrained generics continue to exist and your isFixedSize does not use a concept. I should stress that only real concepts that use the concept keyword trigger the type-checking for generics. Of course, we could also come up with a different rule when exactly this checking is triggered, we need to tinker with it -- a lot.

[zah]

  proc rows(x: self): M
  proc cols(x: self): N

Hrm, you are using a static value as a return type. Seems like an irregularity to me.

---

My underspecified code shouldn't be ruled out as "non using concepts". I could have easily constrained the input type with a concept such as SszSeriazable (or any other type class for that matter). It's just a lucky coincidence that this particular example is not doing this. My general remark is that the pattern of using when X is Y inside generics is a very nice alternative to overloading in many situations and the type checking engine should accommodate it.

[Araq]

Well the eternal question is "is static[int] a type or a value?". ;-) My RFC assumes it's actually a type.

[Araq]

My underspecified code shouldn't be ruled out as "non using concepts". I could have easily required a concept such as SszSeriazable in place of the auto type there (or any other type class for that matter).

Well no, under this RFC it couldn't "easily require" such a thing. If you cannot be precise about the generic type constraints, don't fake it with an imprecise concept, use an unconstrained generic instead. If you don't like this, fine, but that it is what this RFC says.

[zah]

Well the eternal question is "is static[int] a type or a value?". ;-) My RFC assumes it's actually a type.

I've never understood why you insist that values bound to static parameters are "types". Can they be passed to functions accepting regular values - yes. When you initialize a constant from a static[int] parameter, what's the type of the constant - it's int. Where should be the compilation error in the following code:

proc foo(N: static int) =
  const N2 = N
  proc bar(a: N) = discard
  proc baz(a: N2) = discard

For me, the "eternal" question has a very definitive answer. The static parameters are values having tyStatic as a modifier (just like the var parameters are values having tyVar as a modifier).

[zah]

Here is some more fun. If we continue with the "static parameters are types" thinking, we reach the following:

type 3DTransform = concept
  self is FixedSizeMatrix 
  proc cols(x: self): 4
  proc rows(x: self): 4

[Araq]

Here is some more fun...

I'm not sure you're asking the right question. You are thinking about whether these alternative concepts can support what the current concepts can. But the real question is why does it matter to infer the value 4 in order to type-check a generic? And how often does it matter to be able to do this?

[Araq]

I don't care which field, but as long as there is a "Variable[T]" field, the user-defined type constitutes a TrainableLayer indeed. All deep-learning libraries work like that, except that they use dynamic dispatch and inheritance from a Layer class.

Why not use this:

type
  TrainableLayer[T] = concept
    proc variableT(x: self): Variable[T]

# implementation

type
  MyLayer = object
     f: Variable[int]

template variableT(x: MyLayer): Variable[int] = x.f

[andreaferretti]

Because there are many variables, not just one. One actually wants to express a (fixed shape) tree whose leaves are variables

[Araq]

Ok, how about:

type
  TrainableLayer = concept
    iterator variables(x: self): Variable[each T]

# implementation

type
  MyLayer = object
     f: Variable[int]

iterator variables(x: MyLayer): Variable[int] = yield x.f

[andreaferretti]

I guess that could work, but I leave the final word to @mratsim , who has actually tried to put the idea into practice

[zah]

But the real question is why does it matter to infer the value 4 in order to type-check a generic? And how often does it matter to be able to do this?

This example was not inferring 4, it was requiring it. A 3D transform is an affine transformation matrix with dimensions 4 by 4. There are various algorithms that work only on such matrices, so I would say it's a perfect example for a requirement that a generic 3G graphics library might have.

[mratsim]

It needs to support a variadic number of layers like

type
  Conv2DLayer*[TT] = object
    weight*: Variable[TT]
    bias*: Variable[TT]
  LinearLayer*[TT] = object
    weight*: Variable[TT]
    bias*: Variable[TT]
  GRULayer*[TT] = object
    W3s0*, W3sN*: Variable[TT]
    U3s*: Variable[TT]
    bW3s*, bU3s*: Variable[TT]
  EmbeddingLayer*[TT] = object
    weight*: Variable[TT]

User-defined

Residual NN layers: https://www.quora.com/How-does-deep-residual-learning-work

type
  ResidualBlock*[TT] = object
    conv1*: Conv2DLayer[TT]
    conv2*: Conv2DLayer[TT]

The concept should be able to match with TrainableLayer[TT] provided with this type definition.
Anything that requires creating a proc or an additional level of indirection at the type level is a lot of friction compared to Python.

At most we can have this:

type
  ResidualBlock*[TT] = object
    conv1*{.trainable.}: Conv2DLayer[TT]
    conv2*{.trainable.}: Conv2DLayer[TT]

but I'm not sure how to make a concept look for a pragma.

[Araq]

It doesn't sound like we're gonna be able to type-check "somewhere in there Variable[TT] fields will be processed somehow" anytime soon. That doesn't mean the idea of type-checking generics with concepts is pointless... ;-)

[dom96]

Nice to see that this is basically what I proposed in #167 + each, self, either orelse :). Of course, you've obviously thought far more about the semantics.

Personally I would still do away with the self, mainly because it's yet another built-in type and reusing the concept name makes more sense to me:

  type
    Comparable = concept # no T, an atom
      proc cmp(a, b: Comparable): int

    ToStringable = concept
      proc `$`(a: ToStringable): string

    Hashable = concept
      proc hash(x: Hashable): int
      proc `==`(x, y: Hashable): bool

    Swapable = concept
      proc swap(x, y: var Swapable)

To be honest I'm really after simplicity here, to me introducing any new custom syntax is going too far. Because of this I would simply omit each as well (it's a very specific case and I wonder how many other languages actually support it, Rust at least doesn't), why don't we start with the basics first?

For the elseor case I can see this being possible and quite intuitive too:

  type
    Appendable = concept
      proc add(s: var Appendable, elem: Appendable)

    ToStringable = concept
      proc `$`(x: ToStringable): string
    Quotable = Appendable or ToStringable

But all in all, I still prefer this over the current concept syntax even if it does include these additional elements.

[krux02]

First of all, nice to see this work being done. At first sight it looks clean. Regarding the either/orelse, I don't think it should be supported, at least not because of the gives use case. Generally it is much better to just require one operation, in this case it would be add. Then, when a concept doesn't match because of a missing add, it is trivial to add this overload to the type. It is just for historic reasons that this is not done in the standard library.

type
  Quotable = concept
      proc add(s: var string; elem: self)
proc addQuoted[T: Quotable](s: var string; x: T) =
  s.add(x)

type
  MyType = object
var str: string
var mt: MyType
proc add(s: var string; elem: MyType) =
  s.add $elem
str.addQuoted(mt)  # without the previous declaration: error

[zah]

If we zoom out a bit, a concept can represents two distinct set of requirements:

1. Supported syntax requirements
   This is a set of required expressions that the compiler must be able to resolve in the context of a generic function after an overload has been selected. If we know this set, we can type check the definition of the generic function before it's instantiated.

2. Type predicates
   These are arbitrary facts that must be true for the input types before we consider them eligible for overload selection.

This proposal focuses entirely on the first type of requirements while ignoring the second type. By doing this, you only lose expressivity and nothing is gained in return. Gathering the set of syntax requirements is all you need to type check generics and it's a false assumption to believe that this is somehow hindered by the current approach to concepts (the set of requirements is well defined - it's exactly what would be needed in order to compile a generic function with the same body as the concept).

If one really insists on using more declarative syntax, I'd suggest at least the following additions:

• Support of as a mechanism for refining concepts
• Support == as a mechanism for inferring static parameter values
• Support assert-like statements as a way to specify arbitrary predicates for the types
• Support return-like statements as a way to define converter type classes (e.g. StringConvertible)

Some optional points to consider:

• Consider using a different keyword such as callable to reduce the proc/template confusion
• Think about how custom error messages can be delivered
• Think about conditional compilation (when defined(foo): ...)

[Araq]

By doing this, you only lose expressivity and nothing is gained in return.

Er, it allows us to avoid the guesswork we do in the "generic prepass" with its ideas of "bind" vs "mixin" symbols. That is a gain. Also, proper type-checking for generics is done by C#, Scala, Rust, etc -- the C++'s template solution so far only has been copied by Nim and D, both languages which lacked an expert on type theory for their initial designs.

These are arbitrary facts that must be true for the input types before we consider them eligible for overload selection.

Yeah, well, "arbitrary facts" are dangerous. And as I said, if the concept cannot describe the type precisely, there is always the under-specified generic T that can be used.

If one really insists on using more declarative syntax, I'd suggest at least the following additions: ...

Yeah, agreed, especially inferring static values needs to get special support.

[zah]

By "facts", I meant the predicates that the existing concepts support which doesn't necessarily concern the availability of certain overloads. Requiring a square matrix is an example for this. I meant that by not supporting such predicates you only lose expressivity.

Er, it allows us to avoid the guesswork we do in the "generic prepass" with its ideas of "bind" vs "mixin" symbols. That is a gain. Also, proper type-checking for generics is done by C#, Scala, Rust, etc -- the C++'s template solution so far only has been copied by Nim and D, both languages which lacked an expert on type theory for their initial designs.

I also tried to explain that the desired type checking is possible with or without such predicates. It's also possible with the current concepts, so this message is a bit of a straw man argument.

[andreaferretti]

Requiring a square matrix is an example for this. I meant that by not supporting such predicates you only lose expressivity.

My standard techniques to do this is to create a distinct type with private fields, and ensure that the only way to construct it is via some function that checks the predicate. Some sugar for this would go a long way

[sighoya]

As someone who isn't familiar with Nim, what does each T mean?
Why is each T marked in one signature and in the other not?

Further, is there any chance of nominal concepts ala type implements concept? I could imagine that it is easier for the user to know when a type implements some concept and also for the compiler to know that the type is compatible to the concept after typechecking the implementation once instead to do it every time at the call site.

[gemath]

I like the proposal except for these issues:

• either orelse is yet another conditional syntax beside if and when.
• It seems to limit itself to defining API requirements and looks like an interface, but doesn't want to be mistaken for one.
• Not differentiating between a publicly accessible object field and a unary proc with the same name is not always what's intended.

These could be solved with some modifications:

• New rule: a concept is a list of expressions which must all evaluate to true at compile time for a type to match.
• Introduction of an implemented macro which takes the block of proc and iterator declarations which is the concept body in the original proposal as an argument. It returns true if a type matches the argument as specified in the OP, with the following exception:
• A public object field no longer matches a unary proc declaration of the same name.

Code example:

  type
    Dictionary[K, V] = concept
      implemented:
        proc `[]`(a: self; key: K): V
        proc `[]=`(a: var self; key: K; value: V)

      # either orelse is no longer needed
      implemented( proc capacity(d: self): int ) or
        implemented( proc size(d: self): int ) or self.hasField(size, int)

      # any other compile-time test, if necessary even
      compiles(someExpression)

      # ... or some other awesome test that doesn't even exist now.

Less compact than the OP, but with less non-obvious rules and much more flexible.

[Araq]

Is there more to concept refinement than duplicating the concepts constraints (aka the concept's body) in a different concept?

[ornamentist]

I don't understand Nim well enough yet to answer that question unfortunately.

If refinement is not available though I'd like to avoid the situation where a "chain" of refined concepts has the same block of constraints copy-pasted in at every member of the chain.

For example, in a library like Emmy I could have concepts for Magma, Semigroup, Monoid, Group, AbelianGroup, Ring, CommutativeRing and Field, each of which is refinement of the previous concept. The Magma constraints would be repeated seven times, the Semigroup constraints six times etc across the chain, with a corresponding increase in maintenance and readability issues.

I'm not sure if that's how the new concepts proposal would work in practice though?

[Araq]

Copy and paste can always be avoided via Nim's template mechanism anyway.

[sighoya]

Why do copy & pasting at all, doesn't it suffice to point just to the constraints of the concepts that need to be satisfied/implemented too?

Question:

1. Is a concept a compiled time only abstraction or could we use them for boxing too, i.e. we know there is a type satisfying a concept, but we don't know what it is, and we may want to change this type at runtime. Also, concept refinements would imply a subtype relationship.

2. Does Self refer to the implementing type of concept or to the concept type itself. I find the former better in builder patterns and operations requiring the same underlying type as it is the case for type safe equality.

[andreaferretti]

I agree that refinement is not strictly needed, and copy/paste can be avoided via templates. I guess one could also implement refinement via a type macro, that just substitutes the body of the refined concept in the right spot. Still, conceptually is not bad to have it, if it can be supported with low cost.

[konsumlamm]

This is just a collection of my thoughts about concepts so far:

I find the argument that refinement is not needed, because it can be done with templates a bit weird. Yes, you could copy/paste all the declarations by defining them inside a template, but that would look something like

template common() = ...

type
  A = concept
    common()
  B = concept
    common()
    ...

which isn't very ergonomic imo and it should be really easy to implement in the compiler (just leave the copy/paste to the compiler, or point to the parent concept).

Another thing that I find a bit weird is that proc/func means "any routine" in the context of a concept. I'd prefer some special syntax (routine test(a: int)? test(a: int)?) or that any routine declaration (i.e. func, proc, template, macro etc.) is allowed and they each mean "this or something less powerful", so that func only accepts funcs, proc accepts procs and funcs, template accepts templates, procs and funcs, and so on.

Regarding associated types and constants (as they exist in Rust, for example), I'd love to see them, but I'm not sure how that's supposed to work, since items aren't bound to types in Nim (refs #380). I tried to simulate them using a proc that takes a typedesc[Self] parameter, but that doesn't seem to work currently:

type
  Associated = concept
    proc T(self: typedesc[Self]): typedesc
    proc A(self: typedesc[Self]): static[int]

proc T(self: typedesc[int]): typedesc = bool
proc A(self: typedesc[int]): static[int] = 42

static: assert int is Associated # Error

Finally, I think when should be supported in a concept body, like it is possible in other type definitions (objects, enums, ...).

[DanielSherlock]

@dom96 on 10 Sep 2019:

For the elseor case I can see this being possible and quite intuitive too:

  type
    Appendable = concept
      proc add(s: var Appendable, elem: Appendable)

    ToStringable = concept
      proc `$`(x: ToStringable): string
    Quotable = Appendable or ToStringable

I like this option, where the syntax is kept as unexceptional as possible, a lot. On refining concepts, would a similar approach do the trick?

(modifying the example given by @andreaferretti)

  type
    AdditiveMonoid* = concept
      proc `+`(x, y: Self): Self
      proc zero(T: typedesc[Self]): Self
    AdditiveGroup* = AdditiveMonoid and concept
      proc `-`(x: Self): Self
      proc `-`(x, y: Self): Self

[konsumlamm]

I like this option, where the syntax is kept as unexceptional as possible, a lot. On refining concepts, would a similar approach do the trick?

  type
    AdditiveMonoid* = concept
      proc `+`(x, y: Self): Self
      proc zero(T: typedesc[Self]): Self
    AdditiveGroup* = AdditiveMonoid and concept
      proc `-`(x: Self): Self
      proc `-`(x, y: Self): Self

What's wrong with just using of? It is already used for inheritance (which is similar), so it would be even less exceptional:

type
  AdditiveMonoid* = concept
    proc `+`(x, y: Self): Self
    proc zero(T: typedesc[Self]): Self
  AdditiveGroup* = concept of AdditiveMonoid
    proc `-`(x: Self): Self
    proc `-`(x, y: Self): Self

[DanielSherlock]

What's wrong with just using of? It is already used for inheritance (which is similar), so it would be even less exceptional:

I find nothing wrong with of at all; as you say it does echo inheritance for objects, which is nice and consistent. It just seemed like not everyone above liked the idea of adding an inheritance-like feature for concepts "just" to save some repeated declarations (which Nim's templates would be happy to help with).

My thinking with and was that, like or, I'd expect it to be useful in its own right. Also, as soon as you have the ability to check that an instantiated type matches both concepts, we might want to do something like:

  type
    AdditiveMonoid* = concept
      proc `+`(x, y: Self): Self
      proc zero(T: typedesc[Self]): Self
    AdditiveGroupMonoidDiff = concept
      proc `-`(x: Self): Self
      proc `-`(x, y: Self): Self
    AdditiveGroup* = AdditiveMonoid and AdditiveGroupMonoidDiff

...but then why not skip separately declaring the awkwardly-named ...Diff concept, and just write it directly in the and expression? It feels very natural (to me!) to be able to just substitute the name for the definition like that.

It's perhaps also worth mentioning refining more than one concept at once. This is natural (and inevitable) with and. In fact, using again part of @andreaferretti's example above (this time quoting directly), we have:

    Field* = Ring and MultiplicativeGroup

Here, Field refines MultiplicativeMonoid twice: once via Ring and once via MultiplicativeGroup. Of course, the concepts are just checking that the suggested instantiated type satisfies certain conditions - we don't even have default implementations of the given procs - so if it satisfies MultiplicativeGroup once it'll satisfy it again! Perhaps the ease with which we can refine multiple concepts is enough to choose a different syntax than of... though I don't expect that will always be fully apparent immediately?

[konsumlamm]

Something that came up on the forum a while ago: https://forum.nim-lang.org/t/8642#56264 (I think it's worth reposting here, for the sake of documentation).

I really like the idea of having a requires construct that basically falls back to "old style" concepts for everything the new style concepts can't express (at least until we have a better solution). I'm not entirely sure on the syntax though, I'd prefer to not make it too DSLy, so perhaps make it a pragma ({.requires: c.}: ...)?

[beef331]

Presently this RFC does not address having fields or properties of a specific type my sugestion is that an ident def could be used there so the following would be valid. Though this doesnt allow knowing if you can assign a field so maybe var x: type could be used:

type 
  Vector2f = concept
    x, y: float32
  Vector3f = concept
    x, y, z: float32

type MyVec = array[3, float32]
template x(myVec: MyVec): float32 = myVec[0]
template y(myVec: MyVec): float32 = myVec[1]
template z(myVec: MyVec): float32 = myVec[2]
assert MyVec is Vector3f
assert tuple[x: float32, y: float32] is Vector2f

[Araq]

A declaration like proc x(s: Self): var T should match if the concrete type has a public field x of type T.

[ZoomRmc]

A declaration like proc x(s: Self): var T should match if the concrete type has a public field x of type T.

"Should" as in "should already", or as in "should if implemented"?
This doesn't work:

type
  Fieldy = concept
    proc field(x: Self): var int
  Foo = object
    field*: int

let bar = Foo()
doAssert bar is Fieldy

[Araq]

"should be implemented"

[omentic]

Generally, I like the proposal a lot. Although I don't terribly like Rust's traits implementation, I really do like that it is universally used. I think it would be fantastic if an interface like system was stabilized and integrated into the standard library and if this proposal gets them closer to that then I'm all for it. As far as I understand it (thanks to beef) the problem with old concepts are that function / general routine calls and field access (is that all?) are not accessible to the compiler: and so you cannot know that those operations are defined ex. inside a function taking parameters of a concept type?

That said: I have some minor syntax/semantics squabbles and a few major issues with the current proposal.

Major issues

I strongly think these should be called interfaces, and used complementarily with existing concepts. What is an interface if not a series of method signatures and fields valid for a particular type? The overwhelming majority of usages of old concepts are as interfaces, true, and with all the problems that come with a lack of concrete function signatures there (invalid code can compile). Yet there are some concepts (heh) that concepts are useful for that fundamentally cannot be represented by interfaces, and so I think old concepts should also stay.

IMO there should be a distinction between all routines: not just iterator and everything else. The current method as proposed is a bit unintuitive and limiting otherwise. I also think there should be a field keyword to note fields rather than special-casing them as procs. (a field keyword is all you need to structurally represent other types though: types other than object (and enum, I suppose) can be distinguished by their routines)

I don't think the field keyword needs to distinguish between writable and non-writable fields if I'm thinking about this correctly. The use case of interfaces is as function parameters: and so there is already a func foo(param: Parameter) and func foo(param: var Parameter) distinction.

This could look like the following:

type Stack[T] = interface
  func push(s: var Self, val: T)
  func pop(s: var Self): T
  iterator items(s: Self): T 
  func `[]=`(s: var Self, a: uint, v: T)
  field stack: array[256, T]
  field stackptr: uint

Minor issues

I don't like the syntax of each T. Why not stick with existing generic/iterator syntax?

type Serializable = interface
  iterator fieldPairs[T](x: Serializable): (string, RootObj)
  proc write[T](x: T)

I don't like either orelse at all. It seems like a nightmare for monomorphization and IMO it goes against the point of a proc-first concept redesign: you're describing a type by its features, but some of them only sometimes exist? The existing when compiles() works and neatly sidesteps compilation time blowups that I think would happen as a result.

[Araq]

Why not stick with existing generic/iterator syntax?

Because it does something fundamentally different. There is not a single T to be inferred but multiple -- one for every object field. You don't get simplicity but cramming in lots of features into the same syntax, you get simplicity when you drop the barbaric notion of keyword bean counting.

[metagn]

Ideally either orelse could be lowered from standard and/or types with concepts

[Araq]

I'm happy to take either orelse back but I cannot see how we can do without each. I mean, you can play some syntax games to avoid it but the feature needs to exist in some form.

[omentic]

Why not stick with existing generic/iterator syntax?

Because it does something fundamentally different. There is not a single T to be inferred but multiple -- one for every object field.

I don't mind each T terribly much, I just don't understand how it's different from using RootObj and generics.

I am wondering though: what exactly is the distinction between concepts v2 and interfaces? Is it the lack of a particular impl? As far as I can tell, these are functionally equivalent to Go's interfaces, though without the disgusting parts ("empty interface" ugh).

(IMO an explicit impl is far more complexity than it's worth in Rust and I am quite glad to see it's usage-based here)

[omentic]

This has been bumping around my head again recently. What do you think about removing the proc and iterator keywords in concepts v2 altogether? Is distinguishing between iterators and everything else terribly necessary?

type Stack[T] = interface
  push(s: var Self, val: T)
  pop(s: var Self): T
  items(s: Self): T 
  `[]=`(s: var Self, a: uint, v: T)
  stack: array[256, T]
  stackptr: uint

I think something like this could be nice.

[Araq]

That doesn't solve much as the matching needs to be "fuzzy" in other aspects too: Type modifiers like var or sink can be left out, procs usually can take additional parameters as long as they use default values, etc etc.

[konsumlamm]

What do you think about removing the proc and iterator keywords in concepts v2 altogether? Is distinguishing between iterators and everything else terribly necessary?

This doesn't make sense to me. Iterators behave very differently from funcs/procs/templates/macros (returning the same type), you can use iterators in a for loop and they yield multiple items, whereas funcs/procs/templates/macros just return a single value.