An Alternative to Implicits

[odersky]

[This PR was called "Explore Witnesses as an Alternative to Implicits"]. The current iteration replaces witness with instance. I have therefore changed the wording in the title and description of the PR]

Excerpted from the motivation.md file in this PR:

Scala's implicits are its most distinguished feature. They are the fundamental way to abstract over context. They represent a single concept with an extremely varied number of use cases, among them: implementing type classes, establishing context, dependency injection, expressing capabilities, computing new types and proving relationships between them.

At the same time, implicits are also a controversial feature. I believe there are several reasons for this.

First, being very powerful, implicits are easily over-used and mis-used. This observation holds in almost all cases when we talk about implicit conversions, which, even though conceptually different, share the same syntax with other implicit definitions. For instance,
regarding the two definitions

implicit def i1(implicit x: T): C[T] = ...
implicit def i2(x: T): C[T] = ...

the first of these is a conditional implicit value, the second an implicit conversion. Conditional implicit values are a cornerstone for expressing type classes, whereas most applications of implicit conversions have turned out to be of dubious value. The problem is that many newcomers to the language start with defining implicit conversions since they are easy to understand and seem powerful and convenient. Scala 3 will put under a language flag both definitions and applications of "undisciplined" implicit conversions between types defined elsewhere. This is a useful step to push back against overuse of implicit conversions. But the problem remains that syntactically, conversions and values just look too similar for comfort.

Second, implicits pose challenges for tooling. The set of available implicits depends on context, so command completion has to take context into account. This is feasible in an IDE but docs like ScalaDoc that are based static web pages can only provide an approximation. Another problem is that failed implicit searches often give very unspecific error messages, in particular if some deeply recursive implicit search has failed. The dotty compiler implements some improvements in this case, but further progress would be desirable.

Third, the syntax of implicit definitions might be a bit too minimal. It consists of a single modifier, implicit, that can be attached to a large number of language constructs. A problem with this for newcomers is that it often conveys mechanism better than intent. For instance, a typeclass instance is an implicit object or val if unconditional and an implicit def with implicit parameters if conditional. This describes precisely what the implicit definitions translate to -- just drop the implicit modifier, and that's it! But the cues that define intent are rather indirect and can be easily misread, as demonstrated by the definitions of i1 and i2 above.

Fourth, the syntax of implicit parameters has also some shortcomings. It starts with the position of implicit as a pseudo-modifier that applies to a whole parameter section instead of a single parameter. This represents an irregular case wrt to the rest of Scala's syntax. Furthermore, while implicit parameters are designated specifically, arguments are not. Passing an argument to an implicit parameter looks like a regular application f(arg). This is problematic because it means there can be confusion regarding what parameter gets instantiated in a call. For instance, in

def currentMap(implicit ctx: Context): Map[String, Int]

one cannot write currentMap("abc") since the string "abc" is taken as explicit argument to the implicit ctx parameter. One has to write currentMap.apply("abc") instead, which is awkward and irregular. For the same reason, a method definition can only have one implicit parameter section and it must always come last. This restriction not only reduces orthogonality, but also prevents some useful program constructs, such as a method with a regular parameter whose type depends on an implicit value. Finally, it's also a bit annoying that implicit parameters must have a name, even though in many cases that name is never referenced.

None of the shortcomings is fatal, after all implicits are very widely used, and many libraries and applications rely on them. But together, they make code using implicits more cumbersome and less clear than it could be.

Can implicit function types help? Implicit function types allow to abstract over implicit parameterization. They are a key part of the program to make as many aspects of methods as possible first class. Implicit function types can avoid much of the repetition in programs that use implicits widely. But they do not directly address the issues mentioned here.

Alternative Design

implicit is a modifier that gets attached to various constructs. I.e. we talk about implicit vals, defs, objects, parameters, or arguments. This conveys mechanism rather than intent. What is the intent that we want to convey? Ultimately it's "trade types for terms". The programmer specifies a type and the compiler fills in the term matching that type automatically. So the concept we are after would serve to express definitions that provide the canonical instances for certain types.

This PR works out this idea in detail. It consists of

• 5 doc files in docs/docs/reference/instances
• an implementation of what is described in the docs
• updated test files

The PR is based on #5114

Direct link to docs:

1. Motivation

2. Instance Definitions

3. Context Parameters and Arguments

4. Replacing Implicits

5. Discussion

[dottybot]

Hello, and thank you for opening this PR! 🎉

All contributors have signed the CLA, thank you! ❤️

Commit Messages

We want to keep history, but for that to actually be useful we have
some rules on how to format our commit messages (relevant xkcd).

Please stick to these guidelines for commit messages:

1. Separate subject from body with a blank line
2. When fixing an issue, start your commit message with Fix #<ISSUE-NBR>:
3. Limit the subject line to 72 characters
4. Capitalize the subject line
5. Do not end the subject line with a period
6. Use the imperative mood in the subject line ("Add" instead of "Added")
7. Wrap the body at 80 characters
8. Use the body to explain what and why vs. how

adapted from https://chris.beams.io/posts/git-commit

Have an awesome day! ☀️

[lihaoyi]

Reading through, at a first glance this seems like a mostly sideways change that isn't clearly better or worse than the status quo, and for something with such a huge & clear migration cost it should have correspondingly huge & clear benefits.

It's not clear to me that the proposed syntax/semantics significantly helps solve the problems stated up front:

1. Is defining too many implicit conversions really a problem right now?

   • As a library user I'd argue that the most problematic usage patterns are people defining too many extension methods, which this proposal seems to encourage.
   • While this proposal makes defining extension methods somewhat easier, AFAICT it does not really change the confusion at the usage-site: to me it seems it's mostly the people using extension methods which bear the brunt of the confusion, not the people defining them.
   • Non-extension method implicit conversions (implicit constructors in the companion object) can be used very heavily with no usage-site confusion at all (e.g. see most of the com.lihaoyi libraries: requests-scala, os-lib, fastparse, sourcecode, etc., nobody has complained in years about those implicit conversions!).

2. How does changing implicit val or implicit def for witness help tooling? How can tools give a precise list of options, when witnesses are similarly context-dependent as implicits? How does it help make error messages of deeply recursive failures more specific? It's obvious that more progress is good; it's less obvious how this proposal constitutes progress.

3. How is witness foo less minimal than implicit def foo or implicit val foo? Certainly the former has fewer tokens and fewer characters. Would letting people define things as simply implicit foo solve the problem?

4. In terms of syntactic bikeshedding, it seems to me like a sideways change.

   • The current weirdness of "implicit only occurs once per-parameter list" is a non-issue for me

   • The "definition site doesn't align with callsite" issue can be solved trivially by making people put an implicit keyword at the callsite argument list too.

   • The overloading of with and for to mean something completely unrelated to what they currently mean is a negative to me.

   • I can't understand what the dependency injection example is trying to do even after reading it and the surrounding text several times.

   • I would like to offer New implicit parameter & argument syntax #1260 (comment) as an alternative syntax that I believe would be more intuitive, and could be fleshed out more completely if necessary.

There's some interesting bits and pieces in here (e.g. anonymous witnesses/implicits) but overall I think the proposal doesn't make a sufficiently good case about why exactly this, and not something else, is the correct-enough solution for a big-enough problem that is worthy of the huge syntactic change over the status quo.

For example, while the Convertible_List_List_witness example looks like a big improvement in clarity for the new system, anonymous implicits can be implemented on their own as their own thing, and so can something like C# 8's Target-Typed New Expressions, and together would give you all the benefits of clarity and conciseness with only tiny changes:

implicit def [From, To](implicit c: Convertible[From, To]): Convertible[List[From], List[To]] = new {
  def convert (this x: List[From]): List[To] = x.map(c.convert)
}

Or for the Ord example:

implicit val IntOrd: Ord[Int] = new {
  def compareTo(this x: Int)(y: Int) =
    if (x < y) -1 else if (x > y) +1 else 0
}
implicit def ListOrd[T: Ord]: Ord[List[T]] = new {
  def compareTo(this xs: List[T])(ys: List[T]): Int = (xs, ys) match {
    case (Nil, Nil) => 0
    case (Nil, _) => -1
    case (_, Nil) => +1
    case (x :: xs1, y :: ys1) =>
      val fst = x.compareTo(y)
      if (fst != 0) fst else xs1.compareTo(ys1)
  }
}

If we could find similar ways to get most of the other benefits of this proposal with small, standalone changes that are themselves valuable on their own, to me that would be preferable over a big-bang rewrite-everything migration. I suspect we can.

Some ideas to make the proposal better:

• List out explicitly how the initial problems laid out get solved by this proposal: how does this affect tooling? How does changing implicit def for witness make the syntax less minimal?

• Can we separate the syntactic changes from the semantic changes? e.g. can we get the sweet tooling-side improvements to deep recursive implicit failures without overhauling a bunch of irrelevant syntax?

• Perhaps define the proposal in a series of step-by-step delta change from the status quo, rather than a "here is a whole new world and how it works". That would make it a lot easier to properly analyze each step without getting overwhelmed in a sea of unrelated changes

[fanf]

I understand @lihaoyi feedbacks, and I believe his stated 3 points are a good trail to help the proposal keep going forward.
I would still let know that from my corner of social network (https://twitter.com/fanf42/status/1063573841014923265), the general feedbacks are goods, some are even almost hyporbolic about the goodness of the proposal. The main concerns are the same than the one pointed by @lihaoyi: the DI part is more a burden than an asset, the value proposition past the syntax change is not sufficiently apparent, the benefits to help with the tools not made clear - and of course, the migration path seems daunting. All in all extremelly positive :)

[Jasper-M]

@fanf A lot of the positive feedback seems to be directed at #5114 on which this PR is based. I don't really see the benefits of the extra witness stuff over #5114 with implicit.

[odersky]

I dropped the dependency injection section. It is now mostly confusing and does not convey a new point. There was a point once to it, in an earlier design (before b86e91d) where the two roles of an implicit parameters of (1) taking implicit arguments, and (2) being itself a witness for further implicits were separated. Then the example was much nicer. It was left in, to show that it's in principle still possible to do this even if the two parameter roles are not separated. But without that earlier design
it misses context.

[jeremyrsmith]

Just wanted to say, as someone who "gets" implicits, this is much harder for me to understand than implicits. The only problem with implicits (problem being that they are difficult for newcomers to grasp) is that the term is applied to two different things – implicit conversions (I call these "non-explicit conversions") and actual implicits (facts, derivations, and bounds).

Newcomers conflate these things and end up thinking all implicits are bad, because implicit conversions are bad (I've heard so many people call typeclasses "implicit conversions"). But it is the actual (good) implicits that are getting hosed here. It seems backwards to me that the good parts have to change rather than the bad parts.

Wouldn't it be simpler to just change the syntax for implicit conversions? Like maybe you have to say implicit conversion fooToBar (foo: Foo): Bar instead of implicit def, to make it clearer? I don't know what the syntax would be, but I don't look forward to having to re-learn implicits as witnesses instead (not to mention the confusion with shapeless' Witness, though I suppose shapeless will have a very different form when it gets to dotty in any case).

Another thought: Just because something takes a little bit of understanding doesn't make it bad. I think a lot of the hysteria (for lack of a better word) around implicits in Scala is misplaced and is better solved with education than with trying to radically redesign it. Once people learn about it they generally come to understand that it's a really elegant mechanism for abstracting over many elements of the type system in terms of "theorem proving" (in a lightweight sense). But without removing that power, or removing the elegant abstraction, you can't change the fact that it requires a little bit of a paradigm shift for a career Java programmer to understand. And they have to want to do that – some people won't, and that ought to be OK.

[jeremyrsmith]

And one more comment, sorry: Solving some of the ergonomics around implicits would get a lot further than changing the name to escape its baggage. A better story around user-friendly error reporting for failed proofs would solve 80% of the complaints. A story around coherence and orphans would solve probably a further 15% (if such a story is possible). The remaining tail of complaints typically come from people who complain about "magic" (but then will happily use runtime-reflection driven dependency injection stuff like Guice...)

[mdedetrich]

I am going to have to go through this in a bit more detail but I am having a hard time getting convinced about what the improvement witness's actually bring, kind of echoing @lihaoyi points but I am have more of my own which I am going to list.

With your example of the apply being mixed up with the implicit parameter invocation in

def currentMap(implicit ctx: Context): Map[String, Int]

This can easily be solved by simply stating that all apply's must use the non implicit parameters. If you want to then implicitly supply a parameter, you can simply do something like (in the above example)

currentMap.implicitly(someContext)
// or this version which can also solve the currying issues that Scala has with implicits
currentMap("someKey").implicitly(someContext)

This would allow you to do

implicit val ctx: Context = ???
currentMap("someKey")

As desired.

The proposed syntax also has a lot of consistency issues, I am actually for more making the current implicit syntax more consistent, i.e. something like #1260 (comment)

Regarding dependency injection, this can already be done naturally with implicits and traits currently.
i.e. you can define your dependencies like this (for your typical standard backend app)

class UserClient(implicit connectionDetails: ConnectionDetails, scheduler: Scheduler) {
   ...
}
class AddressClient(implicit connectionDetails: ConnectionDetails, scheduler: Scheduler) {
  ...
}

trait ServicesConfig {
   lazy val userClient = ???
   lazy val addressClient = ???
}

trait DatabaseConfig {
   implicit lazy val connectionDetails = ConnectionDetails("localhost",8080)
}

trait SchedulerConfig {
   implicit lazy val scheduler = Scheduler(executionContext)
}

object Main extends App with ServicesConfig with DatabaseConfig with SchedulerConfig {
   logger.info("Initialized")
}

And you can do the thing that you actually want to do with dependency injection, which is replacing specific "nodes" of your dependency graph, i.e.

lazy val testApp = new ServicesConfig with DatabaseConfig with SchedulerConfig {
   override lazy val userClient = MockUserClient(....)
}

EDIT: I realized that the DI section was dropped but I will still leave this here to demonstrate that the current implicit's solve the DI problem fine.

Also in regards to syntax, we really shouldn't overload the meanings of keywords. Although claiming that Scala has fewer keywords than other languages is a notable goal it shouldn't be at the cost of giving keywords overloaded meanings.

For example, for is typically only ever used with for comprehensions. Using the syntax this way for witness's is confusing at best.

Note I am not really saying that the witness is pointless, but I would rather not throw away the baby with the bathwater and fix the current issues we have with implicit's and then evaluate if we need witness and how much value it adds. For example claiming that current implicits are problematic because the compiler gives bad information when implicits are missing, honestly this to me sounds like a scapegoat. There is nothing stopping the compiler from giving detailed warnings proviso performance issues and if thats the case I don't see how witness's would be any better here.

Also I would consider removing implicit conversions (which iiirc are currently deprecated) or severely limit them, this is the only concrete issue I have seen with implicit abuse in the Scala ecosystem

[odersky]

Just wanted to say, as someone who "gets" implicits, this is much harder for me to understand than implicits.

That's a good point. Witnesses are at best a "sideways" move for someone who gets implicits, which is everyone in this discussion. But my focus is, what about somebody who does not get implicits because they are new to Scala? Programs should express meaning and intent. Implicits express mechanism instead. An eye opener for me was when I worked on extension methods (which are by themselves a huge improvement for expressing typeclasses) and was left with

implicit object Foo {
  def extMethod(this x: T): U
}

Really? That seemed like a sore thumb sticking out. implicit object is such a weird incantation to make an extension method available! This was an eye-opener: Scala uses the modifier "implicit" in many situations which are positively misleading to a newcomer. An implicit object or an implicit def is just as explicit as a regular object or method. The fact that implicit means it's available as an argument to an implicit parameter is mystifying to someone who is not already familiar with the concept. I like to think of Scala generally as a clear and readable language, but the use of implicit goes counter to that.

Doing some language archeology, the word implicit was originally used in Scala to define an implicit conversion. There it makes sense. You define a method that can be inserted implicitly. A second step (around 2006) was that implicit resolution for conversions could also be used for implicit parameters which could model type classes. I had a talk "poor man's type classes" at a WG 2.8 meeting in that year. Since it seemed so close, we re-used the implicit modifier for parameters and type class instances.

Fast forward 12 years. By now, most uses of implicit conversions are frowned upon and the language will make it ever harder to define and use them. But the word implicit sticks around, and is getting attached to more and more concepts (i.e. implicit classes, implicit function types). For someone who just discovers Scala, this must be very confusing. I argue we should stop it, and 3.0 is a good occasion to do so.

It's really easy to rewrite automatically, so I am not worried about migration at all. In terms of breaking code I am not worried either, since old implicits will be kept around for a while, then be deprecated for a while, before they would be finally removed in 3.2 or 3.3. It's true that a lot of books and other material will have to be rewritten. But let's face it: Much of the material in these books dealing with implicits would benefit from a rewrite. Many treatments that I have read (or written!) start with implicit conversions, which by now is quite cringe-worthy!

[odersky]

@lihaoyi

Is defining too many implicit conversions really a problem right now?

The feedback I get from others indicates this. It seems to be the #1 problem with implicits.

As a library user I'd argue that the most problematic usage patterns are people defining too many extension methods, which this proposal seems to encourage.

This is interesting. I did not see it that way. There are languages (i.e. Rust, and, to a lesser degree Swift) built around extension methods to do everything! Why do you think they are a problem? Is it that a missing import causes obscure errors? We could try to come up with solutions that address this problem separately.

Non-extension method implicit conversions (implicit constructors in the companion object) can be used very heavily with no usage-site confusion at all (e.g. see most of the com.lihaoyi libraries: requests-scala, os-lib, fastparse, sourcecode, etc., nobody has complained in years about those implicit conversions!).

True. That seems to be the one usage of implicit conversions that carries its weight: Define them with that target type to which they convert.

How does changing implicit val or implicit def for witness help tooling?

It's completely orthogonal. Witnesses are a fairly simple syntactic and type transformation. The most important thing is that they force alignment of parameter definitions and arguments.

How is witness foo less minimal than implicit def foo or implicit val foo?

• it's shorter
• it's just one word for the concept rather than two which have to be combined

This adds up! Meaning instead of mechanism...

Certainly the former has fewer tokens and fewer characters. Would letting people define things as simply implicit foo solve the problem?

No, since it is not implicit. Implicit is a misnomer.

I have to jump on a plane, so I have to leave it at that.

[arturopala]

Maybe implicit is a misnomer, but IMHO it sounds correct for implicit conversions and implicit parameters. I agree it sounds weird when declaring implicit object or class, so maybe only for the later use case, we could find the right keyword, e.g. particular, evident, explicit or singular?

As for with keyword to supply specify implicit arguments, does it express the intent? It sounds somewhat confusing to me in this context, whereas explicitly would be clear and unambiguous.

[lihaoyi]

This is interesting. I did not see it that way. There are languages (i.e. Rust, and, to a lesser degree Swift) built around extension methods to do everything! Why do you think they are a problem? Is it that a missing import causes obscure errors? We could try to come up with solutions that address this problem separately.

I suppose Go also can be thought of as using extension methods for everything, and they seem to not have a problem.

The only difference i can think of is that in Go/Rust/Swift, extension methods aren't "first class" IIRC: you can only define them top-level and import them from top-level packages, and can't do clever tricks like defining them in a trait, inheriting them in a val, pass the val around as and them importing them from the val.

Scala is one of the only language which allows this, because most languages don't let you import from arbirary values. In fact, Scala explicitly prohibits defining extension methods top-level in a file like most other languages! Perhaps that is what causes confusion in Scala?

it's shorter
it's just one word for the concept rather than two which have to be combined

This seems like using witness foo is more minimal than implicit val foo. However, you stated up top that the problem with implicit val foo was that it was too minimal already! Making something that is too minimal even more minimal doesn't seem like solving the problem you stated.

No, since it is not implicit. Implicit is a misnomer.

I'd argue that the idea of implicits in programming are actually very well known, are present in several very popular languages by that exact name:

• C++ has implicit constructors
• C# has implicit constructors
• Most languages have threadlocals which are used in a similar way to Scala implicit parameters, just less safely, which can also be described as "implicit"

Having parameters or conversions be implicits is not that foreign a concept.

On the other hand, the idea of a witness in programming is almost unheard of: perhaps discussed in academic or advanced theorem proving scenarios, but nowhere else. People generally do not talk about passing witness or evidence values around, even people using Scala!

Consider Googling programming implicit vs programming witness:

programming implicit

• "An explicit conversion is one which has to be stated explicitly; an implicit version is one that can be used implicitly, i.e. without the code having to state it" - dictionary - explicit and implicit c# - Stack Overflow
• What is an implicit variable?
• What is an implicit function in C++?
• What is an implicit casting?
• What is an implicit declaration in VB?

programming witness

• Semantic Compression (blog post about the video game The Witness)
• A note about programming design - The Witness
• Witness wednesday on How to Program - The Witness
• Testing the Jai Compiler - The Witness
• Jonathan Blow "C++ is a weird mess": The Witness developer on his new programming language

By replacing "implicit" with "witness", to the vast majority of programmers you are replacing a probably-seen-before-in-similar-concept key word with the name of a video game. You can't get much more misnomer than that!

---

It's worth bringing up the question: what is people's real problem with implicits in Scala? People get by just fine with extension methods in Rust, Go or Swift, and people get by just fine with implicit conversions/constructors in C++ or C#: maybe you don't like them or use them all the time, but they don't cause the average person too much pain using the language. You can very much use Scala like C#: with implicit constructors to save on boilerplate and extension methods to tweak the syntax, once in a while. Why then do implicit parameters and implicit conversions get a bad rap in Scala?

I'd argue the problem isn't so much implicits themselves, but the usage of implicits in the community. This starts, first and foremost, in the standard library: collections and CanBuildFrom are code that everyone uses, everyone jumps-to-definition, and either learns from or recoils away (both of which are bad!). The core/early libraries also help set the culture: with Databinder Dispatch and it's operators, Scalatest and Specs2 with their zoo of inherited extension methods, SBT (which also contributed scala.sys.process), Slick and it's path-dependent inherited implicit-parameter-taking implicit-conversions, scala-pickling's error-prone top-level implicit imports, and others all setting the standard for "this is how Scala is and this is how Scala should be written". Even if you ban implicits in user-code entirely, e.g. with compiler flags, they still end up suffering from confusing implicits in the standard and third-party libraries.

To solve this, the solution would be to build a core ecosystem that uses implicits in a "good way", get it out there into the community, popularize it, and get people to build upon that both for their own happiness and as example code to learn from:

• Can we write a HTTP library that uses implicits in a "good" way?
• A subprocess library?
• A filesystem library?
• A build tool?
• A unit testing framework?
• A serialization library?
• A http client? Server framework?

If we can, then at least we'll have figured out what the "good" ways of using implicits are, and therefore we would like to encourage via language tweaks. And if we don't even know what it means to build "good" implicit-using library code ourselves, there's no way we'd be able to make the right decision on how to tweak the language to make other people improve their usage of implicits!

[arnolddevos]

It's worth bringing up the question: what is people's real problem with implicits in Scala?

The problem is implicit scope. I am surprised that is not mentioned more prominently in these comments.

Implicit scope is hard to visualise. For the user of an implicit there always seems to be one too many places to look for the definition.

For the author there is always a niggling concern that the implicit might not be available when it should or that an implicit might escape and affect something it shouldn't.

Different libraries have different conventions that the user should follow for bringing necessary implicits into scope.

I don't think users of rust traits, for example, have these mental speed bumps.

[katy-p]

I believe it should be Ord[List[T]] instead of List[Ord[T]].

[katy-p]

I believe it should be Ord[List[T]] instead of List[Ord[T]]. (save as above)

[LPTK]

I agree that the word "witness" seems inadequate, outside of academia.

Why not "capability"? After all, that's what implicit parameters really are.

It works well with extension methods:

capability Foo {
  def foo(this x: T): U
}

I'd say that "capability" is also a better name than "type class" (which is quite terrible, because "class" already means something else in Scala). The following makes perfect sense to me:

def foo(x: T)(capability s: Show[T]) = ...

So first-class capabilities are the distinguishing feature of Scala, not implicits, even though capabilities are usually (but not always!) passed around implicitly. Other languages have various forms of implicits, but no mainstream language has first-class capabilities.

Also, as @arnolddevos says, it does seem like implicit scope (first-class capability scope) seems to be the most problematic thing that people complain about.

[jenshalm]

@odersky I feel like this is going in the right direction, I completely agree with the problem analysis of what is inconsistent and confusing about current implicit syntax and think the new proposal provides a neater syntax that describes the intent more clearly.

The sole concern for me, which seems to be shared by some commenters here, is the choice of name (witness), which I find mildly confusing. I might be missing something from common cs terminology, but for me a witness is something rather passive, where the action had been somewhere else and the witness merely testifies that, e.g. in an implicit witness parameter that just testifies that A is indeed a subtype of B. But in this case a witness does not just testify that an instance for type class F[A] exists, it actually does provide the implementation for it (apart from the alias examples). Maybe this is semantic hair-splitting, I was just getting a bit worried about the blank stare I get from newbies in the future when I tell them: "you just need a witness in scope". :-)

My mind is in a lazy Sunday haze so I sadly do not have a counter-proposal for a name and instead would just like to ask: could you describe why you think this is a good choice for a name? Have you considered any alternative names that you later discarded? Are there examples from other languages where this term is used in a similar way?

The only other question is: from the four reasons you give in the motivation section, three are addressed by the proposal in a pretty obvious way. The only exception seems to be tooling. If there is an obvious way this proposal makes smarter tooling easier, it might be nice to explicitly describe the advantage.

[odersky]

@jenshalm I arrived at "witness" after a some iterations. I had "extension"and "instance" before, but both looked too generic. I don't know. Judging from Twitter, people doing type classes like "witness" and it seems quite clear to them.

Other languages are not much better: impl, instance, extension are all fairly generic and ad-hoc names. What I like is that witnesses enforce the mindset that types are properties and term instances, being witnesses of types, show that the property is implementable.

[jducoeur]

(Still need to read properly into the proposal, but based mainly on the conversation --)

What I like is that witnesses enforce the mindset that types are properties and term instances, being witnesses of types, show that the property is implementable.

I can sort of see that, but honestly -- I'm not at all sure how to teach that to novice Scala programmers in a way that they will actually understand.

I kind of like "implicit" because I've figured out over the years how to teach it: I describe a "cloud" of implicit definitions around the code. The implicit keyword either adds stuff to that cloud or summons stuff from. Folks generally seem to get the idea from that description, rough and ready though it is.

This comment kind of worries me in that respect. The concepts of "type" vs. "term" rarely come up in introductory Scala courses -- I don't think we even get into that in our advanced ones. And the notion of program-as-proof rarely comes up. On-the-ground programmers generally don't care about this stuff, and I avoid it rigorously while teaching -- it's the sort of topic that drives many folks to think of Scala as "too hard"...

[senia-psm]

Can one create a class with context parameters?
Like this:

class Context
object Context extends Context

class MyClass(name: String) with (ctx: Context)

In this case it's quite hard to tell what new A with B means:

new MyClass("name") with Context // looks like inheritance

[odersky]

new MyClass("name") with Context // looks like inheritance

I would be in favor of dropping new A with B. If you need a class with more than one parent, name it! This would us allow to retirewith for inheritance altogether. Since with no longer means intersection type, using it for inheritance is kind of pointless. A comma would do just as fine, would be shorter and more readable.

[odersky]

I can sort of see that, but honestly -- I'm not at all sure how to teach that to novice Scala programmers in a way that they will actually understand.

I believe implicits or witnesses is not a subject you want to teach to a novice programmer anyway!

[jducoeur]

I believe implicits or witnesses is not a subject you want to teach to a novice programmer anyway!

Honestly, I don't think there's much choice. Even a fairly straightforward concept like Ordering requires basic typeclasses, and doing anything with Future requires explaining implicit ExecutionContext, so having the slightest clue what's going on there requires explaining implicit parameters.

No, it's not something I teach in the first few days. But in practice, as soon as an engineer hits a non-trivial codebase they will hit implicits, and the more curious ones (especially the experienced programmers -- I'm not talking about new programmers, I'm talking about new Scala programmers) get really annoyed if you try to brush them off as something they shouldn't worry their little heads about.

I don't go deeply into implicits at that point, mind. But being able to provide some intuition about what is going on, in terms that they can grok, is pretty important fairly early on, in my experience...

[acjay]

I'm a big supporter of rethinking implicit. As someone who has had to mentor new Scala practitioners, implicit is one of the biggest hurdles. The semantic mismatch between the actual definition of the word and how it works in Scala is a big part of this. Even now, there are advanced uses I can't wrap my head around because of the divergence between the literal code and its intended semantics. If accessibility to newcomers is a top priority, even a lateral change could make a big k improvement.

I think the parameter list representation of implicits is indeed one of the hardest parts to understand, and it's also one of the hardest usages of the keyword to avoid in basic real-world Scala programming. Consider the pervasive need to pass ExecutionContext if you're using standard library Future or ActorSystem if you're using Akka. It scares the hell of of people on Day 1 and confirms their suspicion that the language is complex. Any reform should be measured against these use cases just as much as against the more advanced uses of implicit.

I disagree with the idea that implicit is familiar to users of other languages. My experience elsewhere is almost totally limited to implicit conversions, which is often not what people do in idiomatic Scala. I'd argue that this association actually makes Scala's system more difficult to grasp, because the analogy doesn't go very far.

capability seems like the best suggestion so far, but I can also imagine context being a good descriptor. The latter is reminiscent of React, which has a concept called context, which is rather like a really simple version of the implicit scope mechanism.

Related to that question, I do think it could be very helpful to separate the conveyance of implicits down the call stack from their summoning. For conveyance, complete anonymity is preferable. For summoning, you don't necessarily need to convert the implicit any further. It could be very helpful to separate these concerns in a way that could be combined, but only if necessary. The context bound sugar does a great job of this in that particular use case of implicit.

As an aside, React's context is actually more implicit, in the literal sense, than Scala implicits, because you don't need to declare it anywhere between the provision and consumption, unlike Scala's use of implicit parameter lists. Would this be possible to achieve? In this case, conveyance could literally be implicit.

[odersky]

@jducoeur @acjay @LPTK I agree implicit parameters come up immediately when teaching some libraries. One of the first times I mention implicits in my courses is ExecutionContext. And it is confusing to make them look like normal parameters, which is another reason to change the syntax to with.

Implicit objects and classes? I believe that's comparatively rarer, and it's a harder concept to grasp, for the following reason:

Type classes are a second order concept - they are types that describe properties of other types, which may or may not hold depending on context. I.e.

• Ord[T]: T has an ordering
• Monoid[T]: T has "combine" and "unit" operations
• CanBuildFrom[From, Elem, To]: One can construct a collection of type To with elements of type Elem from a collection of type From.

That idea takes some getting used to but I believe it is essential to be clear about this if one wants to teach type classes. My earlier, abandoned attempt at extensions was in part motivated by the wish to delay the introduction of type classes by modelling context dependent extensions instead. That approach introduced other complexities; in particular it required additional machinery for talking about non-extension methods such as unit in Monoid. The idea of, first, the current version of extension methods and, subsequently, witnesses came when realizing that ultimately the language becomes simpler if we embrace type classes rather than hiding them.

But once one introduces type classes as properties of types, I believe witness is a descriptive term for type class instances. Is T ordered? It depends on the context. One needs a witness to establish that fact.

capability is an interesting alternative, which has a lot going for it. We often use implicit parameters to represent capabilities. That's already handled with with but capability also works well for something like the global ExecutionContext - that's a system wide capability to execute something. It's less obvious for Ord[Int] - yes, you can say, here's a capability to compare Ints, but I prefer to start with the property "Int is ordered" and say here's a "manifestation" or "witness" for that fact.

The other problem with capability is that defining a "global" capability like we do with a type class instance is an anti-pattern. The capability security model explicitly disallows this.

The last (maybe decisive?) problem is that it is longer than "witness" and has 5 syllables instead of two. So there's that...

[odersky]

The other thing I like about witness is that it is an easy to grasp description for Curry-Howard: "Types are properties and terms are proofs". Sure, that's academic, but I believe it will become a lot more industrially important in the future.

[drdozer]

I think what this expands to is partly a question of how good the compiler is at inlining things enough to get rid of allocations. When it defaults to minting new instances all over the place, there's an argument that we need the manual control to make things objects or vals to reduce allocations. Manually control the flyweighting. If the compiler (with appropriate hints) can inline enough to get rid of most allocations then I'm very relaxed about everything being defs. I tend to weigh in on the side of simpler language, more complex compiler --- but then I don't take responsibility for the compiler.

I am coming around to

instance f of T  =  E

with the expansion

final implicit def f: T = E

[LPTK]

By the way, if with is deprecated as both the mixin composition operator and the intersection type operator, then why not write implicit function types with A => B instead of A |=> B? Like @b-studios I find the latter really inscrutable, and it reminds me of the dark days of symbolic operator abuse in the Scala community, and other obscure operator mishaps like <% view bounds.

It would be much easier for beginners to see the correspondence between implicit parameter sections and implicit function types:

def foo(x: Int) with (y: Foo): Bar = ...

val foo2: (x: Int) => with (y: Foo) => Bar = foo

// equivalent to:
val foo: Int => with Foo => Bar

[milessabin]

I'm going to merge this now, with the proviso that it being merged doesn't represent endorsement by, or preemption of, the SIP committee. It's a large and controversial proposal which will be discussed in detail by the committee and there's absolutely no guarantee that it will appear in Scala 3.

Speaking only for myself, I share some of the concerns raised above: whilst the concision gained for type class declarations and instances is welcome, the cost in terms of syntactic churn is very high. It's not clear to me yet that the price is worth paying.

My attitude to implicits has evolved significantly over the last few years, and will most likely evolve even more over the next few. What I've come to realize is that implicit resolution is to terms as type inference is to types. Or to put it another way implicit resolution is term inference. This is very explicit in languages like Coq, Agda, Idris and Lean.

Inference though, whether of types or of terms, is orthogonal to typing, and I'm finding it increasingly difficult to see why there should be a difference in typing between any of the following definitions,

def foo(i: Int): Int = i+i
def bar(implicit i: Int): Int = i+i
implicit def baz(i: Int): Int = i+i

All three are methods which take an Int argument to an Int result. Adding the implicit keyword doesn't change the meaning of the definitions in any way (I'm ignoring the implicitness of i on the inside of bar ... it's a detail that doesn't affect the substance of this comment). All that use of the implicit keyword affects is the elaboration (ie. the filling out of type and term arguments and the insertion of conversion calls) of terms at call sites.

I think it's something like this observation which motivates the comment made several times above that "implicit arguments are just arguments". I think those comments are almost spot on. Why should a marker that relates to elaboration change an argument from being an ordinary argument to being something else, when the semantics of the definition in question, and the semantics of the elaborated term are otherwise unchanged?

The direction this takes me is that I'd now love to see elaboration behaviour separated out from method/function signatures and other definitions altogether (so, arguments are just ordinary arguments) and I think we should try and come up with some alternative mechanism for specifying how elaboration should happen and how programmers can influence it.

[milessabin]

See comment prior to merge ...

[odersky]

I agree that we still need to discuss this. The analogy between type inference and term inference is spot on I think. I believe from that analogy follows that our syntax must make it clear where a term is missing, so that it can be inferred.

Meanwhile, I was having some new thoughts how to make the proposal less controversial. But that will be a different PR.

[nafg]

Can you link to it here so people subscribed to this ticket can follow the new one?

…

On Tue, Jan 29, 2019 at 11:56 AM odersky ***@***.***> wrote: I agree that we still need to discuss this. The analogy between type inference and term inference is spot on I think. I believe form that analogy follows that our syntax must make it clear where a term is missing, so that it can be inferred. Meanwhile, I was having some new thoughts how to make the proposal less controversial. But that will be a different PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5458 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGAUHSe22qvEGW6Lig41lIqJCjVzwF4ks5vIH1CgaJpZM4Ymu8J> .

[Glavo]

Using instance of Conversion instead of implicit method produces object instances that is not necessary. Does this change cause the class file size grow and introduce unnecessary overhead at runtime?

[odersky]

Using instance of Conversion instead of implicit method produces object instances that is not necessary. Does this change cause the class file size grow and introduce unnecessary overhead at runtime?

It does make class files slightly larger but I don't think it will introduce a sizeable runtime overhead since conversions can be inlined.

[odersky]

@ltpk @b-studios I am experimenting with a tweak which uses given instead of with, and that uses given also for IFT syntax.

[b-studios]

@odersky Great, I would be happy to look into this as soon as you have a draft. Besides the syntactic details, I really love the separation of passing arguments from passing implicit arguments (explicitly). I quite often use value inference to guide type inference; to
be able to tell the difference between the two ways of providing arguments really helps there.

(BTW, I guess you were to mention @LPTK :) )

[drdozer]

@milessabin I think you're entirely right that implicits are term inference, analogous to type inference.

I've been frustrated with earlier incarnations of scala that there was not a sufficient syntactic separation between what was being inferred and what had to be explicitly supplied. It leads to problems like needing to put funcNeedingImplicit.apply all over the place to force the thing to the left-hand-side to summon implicits. The instance with of syntax here does begin to separate out the supplied and inferred values. A bunch of things that used to require a lot of careful boilerplate now Just Work. I'd not like to lose that.

To normalise syntax, perhaps something could be done in the called signatures to flag individual parameters as being ones that are derived, rather than entire parameter lists.

instance MyFoo of Foo { ... }

def foo(a: Int, instance b: Foo, c: String)

I've found it very difficult to juggle implicit vals, def and objects. No end of bugs and of failed refactorings have been down to me mangling the interchange between these. And the def is often a useful fiction of keeping track of type arguments, which (should) evaporate after erasure in any case, resulting in a single implementation instance that can be flyweighetd for all types. But the current commitment to the def verb or the object verb forces object allocation implications on us purely because of chasing compile-time types not relevant to actually running the bytecode.

So I'd welcome a unified syntax where instance introduces an implicit value, be it a constant or a transformation of other values. If we strip out with/of, we get to something terse like:

// declare a "constant" derivable value
instance MyFoo: Foo = new { ... } 

// declare derivables made by a function
instance ListSemigroup[T](instance T: Semigroup[T]): Semigroup[List[T]] = new { ... } 

// declare a derivable by a function that uses @FunctionalInterface sugar
instance ListSemigroup[T](instance T: Semigroup[T]): Semigroup[List[T]] =
   (_: List[T]).zipWith(_).map { (l, r) => T.append(l, r) }

Or lastly, we could use instance to declare an implicit value and deriving to flag the requirement for one.

instance ListSemigroup[T](deriving T: Semigroup[T]): Semigroup[List[T]] = ...

[acjay]

@milessabin Would your earlier comment suggest not labeling arguments as implicit in the function signature at all, but instead having eligibility for inference be elected elsewhere? Presumably at the time of term or type definition? And then inference would just happen for a given ordinary function when called with omitted trailing arguments?

[milessabin]

@acjay Yes, something along those lines.

[acjay]

That sounds more elegant to me than the status quo, on some level. I'd guess the vast majority programs have strictly disjoint sets of implicit and non-implicit types. But I could also see how this could be really confusing. Seems worth considering, though!

FWIW, I don't see this proposal as turning implicit parameters into non-parameters, at all. They'd simply be declared and provided differently from positional parameters. It would be still be straightforward to map to conventional parameter. Kind of like how Int => Int => Int is isomorphic to (Int, Int) => Int.

[odersky]

Evolved version: #5825

[nafg]

@odersky isn't the .apply issue not specific to implicits?

Very contrived illustration:

object O {
  def overloaded(a: Int)(b: String) = 1
  def overloaded(a: Int) = (c: Boolean) => 2
}

Arguably this

def implicitParam(a: Int)(implicit b: String) = (c: Boolean) => 3

is the same thing: a method that has two ways to invoke it, with two parameter lists or with one.

Again, a very intuitive solution is to say that ((m(a))(b) first evaluate m(a) thus being equivalent to `m(a).apply(b).

Last time I suggested it you wrote,

I don't see how that solves the problem. That's just a more obscure way to write f(aa).apply(bb).

I don't see why it's obscure, it's very natural. The fact that right now the parentheses don't do anything was very surprising to me when I first encountered it (a long time ago :) ).

To be honest I don't really get why having to write f(aa).apply(bb) is a problem in the first place. As far as I can tell it's just a mild annoyance because (1) it's a bit more verbose and (2) it loses function call syntax. Making ((f(aa))(bb) work solves both of those. It's not verbose and it's natural function call syntax. Again, the parentheses forcing their insides to be parsed as a standalone expression is very natural, it's what parentheses are always for (e.g. x * (y + z). I don't see why this is any different.

Also,

But, as far as I can see, not just allowing f(aa)(), but forcing you two write f(aa)() if the implicit should be left out.

I really don't see the problem with that, any more than it's a problem with default arguments. Implicits-as-defaults is very elegant, it unifies two overlapping concepts. And if you want you could tackle it for defaults as well, again by parentheses work, as above.