every symbol becomes 1st class; defines 0-cost lambda and aliases; generics/iterators/templates/etc can be passed to any routine

[timotheecour]

This PR makes every symbol 1st class, which enables these:

• passing any symbol in scope (generic/proc/macro/template/iterator/module/etc) to any routine (proc/iterator/macro/etc). This works even if the symbol is overloaded, or if it's a template/macro with all optional params
• aliases (const myecho = alias echo) that work with any symbol; can also return a tuple, eg (fun1: alias echo, fun2: alias doAssert)
• 0-cost lambdas (eg: x~>x mod 2 == 0, (x,y) ~> x*y, lambdaIt: it*2 etc), which are defined in library code on top of alias
• composable iterators, which allows define lazy functional programming primitives (eager toSeq and lazy map/filter/join etc; which can have speed benefit compared to sequtils); the resulting code is quite nice and compact, see tlambda_iterators.nim which uses library lambdaIter that builds on top of lambda; it allows defining primitives that work regardless we're passing a value (@[1,2,3]) or a lazy iterate (eg iota(3))

Extensive unittests are included, see tlambda.nim, tlambda_iterators.nim

fixed issues

• symbols of no argument templates can't be passed to macros #11870 symbols of no argument templates can't be passed to macros
  see tests: every symbol can be passed to every routine, eg foo(alias mysym)
• Cannot pass inline iterator to inline iterator #4516 Cannot pass inline iterator to inline iterator
  see tests
• type inference with lambdas doesn't work (requires explicit types) #7435 type inference with lambdas doesn't work (requires explicit types)
  see tests, eg: myfun(10, a~>a*2)
• fix iterator: wrong + called - lib/system.nim(1102, 12): Cannot instantiate T #5702 (see unittests; can pass builtin + just like any other symbol)
• supersedes [TODO] Nim now supports true lambdas; eg allows map(a~>a*localVar) (wo limitations of sugar.nim =>) #8679 (lambdas); current PR has none of the drawbacks of [TODO] Nim now supports true lambdas; eg allows map(a~>a*localVar) (wo limitations of sugar.nim =>) #8679:
  • in particular lambdas are well typed instead of having to be passed as untyped
  • usage is simpler since there's no need for an extra makeLambda inside each user of the lambda
  • the lambda can be passed as a generic param in proc callFun[Fun](fun: Fun): callFun(a~>a*10) or callFun(myTemplate) etc, requiring 0 change in existing code (since the lambda can be passed as a generic)
• RFC: Lambda expression (with inferred placeholders) #8675 RFC: Lambda expression (with inferred placeholders)
• can be used to fix [TODO] fix issue #8303 map overly restrictive #8325 (see tests that implement show map, filter, join, toSeq, etc)
• implements [WIP] [RFC] first class inline iterators: iter it = myiter() to allow perfect forwarding of iterators, with delayed evaluation #9374 ([RFC] first class inline iterators: iter it = myiter() to allow perfect forwarding of iterators, with delayed evaluation): see lambdaIter in unittests
• can be used to fix Exporting a toSeq overload messes up iterator visibility RFCs#512 because toSeq could be rewritten to be without untyped as argument to accept iterators, see tlambda_iterators.nim ; likewise, overload resolution (with typed and untyped) doesn't work with iterators #9219 could be closed since we can now pass the iterator as done in unittests
• closes Internal compiler error for proc alias #8935 since we can do const testForMe=alias assert, ditto with aliasing generic/template/macro/overloads
• closes Unable to alias echo from the system module #11731 since const my_echo=alias system.echo works
• using alias instead of static solves this Generic types as static parameters: type mismatch #9679 (Generic types as static parameters: type mismatch), so might be closable too
• would fix map shouldn't care whether proc arg is cdecl #8303 (see map defined in tlambda_iterators.nim)
• would fix Error using toSeq in map #16439
• fixes alias declarations #7090 (alias declarations) ; this was wrongly closed: the workarounds didn't work in many cases and each case required a different syntax); this feature request keeps appearing, eg:
  • https://forum.nim-lang.org/t/5015
  • https://forum.nim-lang.org/t/1515
  • Syntax for aliasing btbytes/nim-cookbook#21
  • https://www.reddit.com/r/nim/comments/5v2rgw/alias_echo_to_print/
  • https://forum.nim-lang.org/t/1810

speed benefit

• can be used to improve speed and ease of use of procs that have callback (functor) parameters: the callback can now be a zero-cost lambda instead of a closure (which incurs overhead, eg due to lack of inlining once we have a closure);

for example the following code gives a 3X to 4X speedup (with both debug build and -d:danger) over algorithms.isSorted which uses a callback closure via
func isSorted*[T](a: openArray[T], cmp: proc(x, y: T): int {.closure.}, order = SortOrder.Ascending): bool =

func isSorted2*[T, Fun](a: openArray[T], cmp: Fun): bool =
  result = true
  for i in 0..<len(a)-1:
    if not cmp(a[i],a[i+1]):
      return false

# benchmark code:
  let n = 1_000
  let m = 100000
  var s = newSeq[int](n)
  for i in 0..<n: s[i] = i*2
  benchmarkDisp("isSorted2", m): doAssert isSorted2(s, (a,b)~>a<b)
  benchmarkDisp("isSorted", m): doAssert isSorted(s, cmp)

other benefits

• with this PR, patterns like macro mydefine(name: untyped, val: typed) can now become: const myname = mydefineAux(val) ; see tests
• supersedes [superseded] alias: myecho=echo to alias any symbol #11822 (alias: myecho=echo)
• partially addresses [RFC] Better iterators syntax RFCs#1, see tlambda_iterators.nim
• reduces need for passing a symbol as untyped followed by calling bindSym since we can now pass the symbol directly (either via an explicit generic param foo[T](fun: T) or implicit generic foo(fun: aliassym))
• helps with things like https://nimble.directory/pkg/zerofunctional (the resulting iterator can now be returned with this PR)
• this actually generalizes typedesc and static without their restrictions (see Generic types as static parameters: type mismatch #9679) ; using alias you can also pass a type or a const (via passing a template symbol that returns that const, eg myConstReturningTemplate or ()~>myConstVal); maybe static/typedesc could now be implemented in terms of alias which would fix bugs inherent with these
• provides a cleaner alternative to using mixin, for example this helps with issues like Generics "sandwiched" between two modules don't mixin their scope symbols properly #11225 (Generics "sandwiched" between two modules don't mixin their scope symbols properly)

notes

• [parser] a => counter+=1 parsed as (a => counter) += 1, violating the spec, and causing issues with => #8759 is a minor bug that affects both sugar.=> and this PR's ~> in some cases; should be easy to fix

• alias2 and aliassym are placeholder names; I'll change these once we agree on final names

• I'd prefer the syntax: alias x = foo.bar instead of const x = alias foo.bar; this requires a minor parser change; it would simplify usage in general. The code in this PR wouldn't change much with this change, but user code would become simpler. If there's agreement, I'll switch to alias x = foo.bar and proc bar(fun: alias)

• compared to sugar.=>, the library code alias defined in this PR on top of alias are easier to use (doesn't suffer from type inference with lambdas doesn't work (requires explicit types) #7435 as shown in unittests), and more efficient (no function call overheard that would prevent inlining)

• will hide behind a feature flag (eg: {.experimental: aliassym.}) once we agree on name

[timotheecour]

just rebased (it was previously green but had started to bitrot) /cc @Araq

[timotheecour]

rebased and workaround after new gensym handling
friendly ping /cc @Araq

[timotheecour]

rebased again against latest devel /cc @Araq ; this PR (which garnered a lot of +1's) would fill an important gap in nim IMO;
I've been benefitting from it in my own code but it's unpleasant to have to keep rebasing a patched nim against latest devel :)

[kristianmandrup]

Looks fantastic!! Please merge this ASAP, perhaps release it for 1.1.0?

[Araq]

This really needs a RFC process. Negative sides:

• I need to constantly think about this feature when adding new code to the compiler, comparable to skipTypes which is burdensome.
• It probably impacts the compiler's speed.

[timotheecour]

For now it doesn't even subsume typedesc[T]. How to infer the T?

here:

when true:
  import std/lambdas
  template typedesc2[T](a): bool = a is type and a is T

  # this currently works:
  proc bar1(T: aliassym): auto = T.default
  doAssert bar1(alias2 float) == 0.0
  proc bar2(T: aliassym): auto {.enableif: typedesc2[SomeInteger](T).} = T.default
  doAssert bar2(alias2 uint8) == 0'u8

  when false:
    # this hasn't yet been implemented but will work:
    proc bar3(T: aliassym): seq[T] = discard
    # pending future enableif syntax sugar:
    proc bar2(T: typedesc2[SomeInteger]): seq[T] = discard

I see concepts as our ticket to get typechecked generics and I don't see how "enableif" can get us there.

{.enableif.} is much simpler and more powerful, it can express constraints that can't be expressed with concepts (isIterable, constraints involving more than 1 arguments, constraints involving untyped, etc) without all the bugs.

just look at how much easier it is to write a correct+efficient toSeq with this PR.
As the author of #8711, I can tell you how refreshing it is to be able to use proc instead of template and not have to worry about evalOnceAs (or all the gymnastics in toSeq, toSeq1, toSeq2).

Also note that sequtils.toSeq is still incorrect for openArray (multiple evaluation bug, and it's really hard if not impossible to fix once you understand the problem), whereas with this PR, it just works:

import std/lambdas
template isIterable(a): bool = typeof(block: (for ai in a: ai)) is type
proc toSeqImpl(a: auto): seq {.enableif: isIterable(a).} =
  type T = typeof(block: (for ai in a: ai))
  when compiles(a.len):
    result = newSeq[T](a.len)
    var i=0
    for ai in a: (result[i] = ai; i.inc)
  else:
    result = newSeq[T]()
    for ai in a: result.add ai

template toSeq(a: untyped): untyped = toSeqImpl(lambdaIter a) # sugar to avoid calling code having to write `lambdaIter`

doAssert toSeq(10..13) == @[10,11,12,13]
iterator iota(n: int): int = (for i in 0..<n: yield i)
doAssert toSeq(iota(3)) == @[0,1,2]
doAssert @[10,11].toSeq == @[10,11]
doAssert [10,11].toSeq == @[10,11]
doAssert not compiles toSeq 10

# works with openArray, safe wrt multiple evaluation bug, unlike sequtils.toSeq!
from sequtils import nil
proc fn3(a: openArray[int]) =
  template fn(x: int): untyped = (echo "in fn"; x)
  echo "this PR toSeq: no bug"
  doAssert toSeq(toOpenArray(a,0,fn(1))) == @[10,11]
  echo "sequtils.toSeq: multiple evaluation bug and hard/impossible to fix"
  doAssert sequtils.toSeq(toOpenArray(a,0,fn(1))) == @[10,11]
fn3 @[10,11]

prints:

this PR toSeq: no bug
in fn
sequtils.toSeq: multiple evaluation bug and hard/impossible to fix
in fn
in fn
in fn
in fn
in fn
in fn

[Araq]

As long as your return type remains auto you don't have a solution that is satisfying. Also, how can you "enableif" constraints for the return type? Remember that the documentation generator cares about proc headers, not about proc bodies.

[SixteNim]

aliassym simulates a #define name (....) well known from C, but locally scoped instead of C's rather undefined (global) scopes. If the (...) stands for a name itself, alpha-conversion will happen - the most basic (trivial) substitution. Therefore, a proc that annotates a a : aliassym b can be read as a substitution a for b , where the proc body uses an "a" that stands for a "b" and the "b" itself stands for a witness, for both a prototypic implementation and a pattern for other substitutions as well. Logically, aliassym is sound - as long as a witness is presented. But what about substitutions? Substitutions should be made possible with an attribute, e.g. {. witness 'name' .} where "name" stands for the name of the witness. The attribute is sufficient enough to get a sound aliassym , noenableif etc. needed. As long as scope borders are respected, the implementer of a particular module will have full control about the module's intended behaviour. This is important for safety reasons. If scope borders are not respected, the compiler should give a warning (at least).
aliassym is crucial at every place where structural substitution is needed (iterators) ore desired (lambdas to be inlined). It would improve Nim's expressiveness, safety and speed at the same time. Why don't other languages (except D) have this feature? Well, they are not flexible enough at the AST level.

[mwerezak]

Notwithstanding the technical discussion in the PR, having first class symbols and composable iterators would be a huge boost to expressiveness of the language.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.

[github-actions]

This pull request is stale because it has been open for 1 year with no activity. Contribute more commits on the pull request and rebase it on the latest devel, or it will be closed in 30 days. Thank you for your contributions.

[github-actions]

This pull request has been marked as stale and closed due to inactivity after 395 days.

[mwerezak]

rip

[ringabout]

Although this PR is incredible. It has been a struggle to maintain the existing features of the Nim language. What really is needed for now is to refactor existing features and make it accessible among backends. Anyway the stale label means eveyone could pick it up and complete it.