Shapeless 2 on Dotty

[djspiewak]

This is a general meta-issue to aggregate the overall project of getting a version of Shapeless 2 working on Dotty.

Goals

• Rough source-compatibility with the Shapeless 2.x versions of any constructs which can be meaningfully expressed on Dotty
• Full source-compatibility with the Scala 2.13 variants of the same construct in the same release. In other words, it should be possible to move from 2.13 to 3.0 simply by swapping scalaVersion for any code which exclusively uses the supported constructs

Non-Goals

• Perfect binary compatibility with the 2.x line (though, the closer to this we can get, the better)
• Ports of less-used constructs, particularly those which are impossible to render on Dotty

Requirements

Each of these should be split out into a separate issue:

• Generic
• HList
• Coproduct
• Polymorphic functions

Comments and opinions very much desired on all of the above!

To be clear, I'm well aware that most of this is covered by out-of-the-box functionality in Dotty (and shapeless 3 is a nice layer on top of that). However, any project which is cross-publishing between Scala 2 and Scala 3 will benefit considerably from this type of functionality. Additionally, cross-publication is a special case of migration, and any project which is using shapeless already will either need something like this, or it will need to recreate this type of functionality on their own for bespoke cross-building (i.e. what Circe had to do).

[julienrf]

As far as I understood, it would not be possible to implement Lazy with Scala 3 macros. Maybe we need to assess this statement before going further?

For reference, Lazy is implemented using by-name implicits in Scala 2.13. The Scala 2.12 implemantion uses a macro.

Why couldn’t we also implement Lazy in Scala 3.0.0 by using by-name implicits?

If we can’t implement Lazy in Scala 3.0.0, Miles suggested deprecating Lazy from Shapeless in favor of using by-name implicit parameters, which are supported in Scala 2.13 and 3.0.0. So, we would release Shapeless 2.4, where Lazy would be deprecated (in favor of by-name implicit parameters). This version would still be binary compatible with 2.x. Then, it would be possible to publish it for Scala 3.0.0 (Lazy would still be part of the API, but there would be no macros that summon it).

[joroKr21]

Why couldn’t we also implement Lazy in Scala 3.0.0 by using by-name implicits?

It's totally possible. We even have a plugin to enable using by-name implicit syntax on Scala 2.12 which is replaced by Lazy. And on the main branch we already use by-name implicits. See also #1171

This version would still be binary compatible with 2.x

I don't know about that. The biggest problem for binary compatibility is that Shapeless 2.3 is using a weird encoding of Symbol singleton types as labels. I don't know if that can be implemented in Scala 3, but I'm sceptical. And if we can't do that binary compatibility will be broken either way.

[julienrf]

The biggest problem for binary compatibility is that Shapeless 2.3 is using a weird encoding of Symbol singleton types as labels. I don't know if that can be implemented in Scala 3, but I'm skeptical. And if we can't do that binary compatibility will be broken either way.

That’s good to know.

I’m summoning @nicolasstucki to ask whether it is possible or not to express type refinements of the type Symbol in Scala 3, and get something like constValue work with it? Essentially, I would like the following code snippet to compile:

inline def getLabel[L <: Symbol]: String = compiletime.constValue[L].name
inline val bar = Symbol("bar")
println(getLabel[bar.type])

https://scastie.scala-lang.org/xKebkhNYSeWFoxfPmMpylg

[milessabin]

I like us to get off using Symbol for labels irrespective of the Scala version ... it was always a mistake.

That's already done in shapeless 3, but that's Scala 3 only so far. I think we should do the same for shapeless 2.4 ... it'll be a breaking change, but I think we have to live with that.

[julienrf]

I guess it’s OK to switch to String instead of Symbol. Users of shapeless will have to adapt to this change, but hopefully this can be achieved in a binary compatible way.

Typically, I have code like this:

    implicit def consRecord[L <: Symbol, H, T <: HList](implicit
        labelHead: Witness.Aux[L],
        jsonSchemaHead: JsonSchema[H],
        jsonSchemaTail: DerivedGenericRecord[T]
    ): DerivedGenericRecord[FieldType[L, H] :: T] = ...

I will have to change for the following:

    implicit def consRecord[L <: String, H, T <: HList](implicit
        labelHead: Witness.Aux[L],
        jsonSchemaHead: JsonSchema[H],
        jsonSchemaTail: DerivedGenericRecord[T]
    ): DerivedGenericRecord[FieldType[L, H] :: T] = ...

But the erased signature will be the same, so this change will be binary compatible.

[joroKr21]

But the erased signature will be the same, so this change will be binary compatible.

I'm not sure about that because L would erase to its upper bounds which would change from Symbol to String so in general I don't think we can assume binary compatibility will be preserved.

[julienrf]

Oh right, it erases to its bound, so I will have to introduce a separate method.

[Katrix]

Ok, so progress is happening on this finally. So far I've gotten a version of Shapeless 2 compiling on both Scala 2 and 3, but where all macros are dummied out. No idea how well it works after it compiles.

One big discussion which needs to be had is what to do with everything that can't be cleanly ported to Scala 3? This is mostly macros that do things Scala 3 macros can't or types not representable in Scala 3. As I can see, there are a bunch of similar macros, which can be grouped and talked about together.

1. Accesses the outer scope of the macro call (cachedImplicit is one example that does this from a quick glance at it. Also not sure how feasible it would be to implement in Scala 3 with my limited knowledge of what you can do there.)

2. Type providers of all sorts. Think Union, Witness, Record, HList, Coproduct. Problem here is mainly constructing a type from a string, which is not something I think Scala 3 exposes.

3. Things that use untyped trees. Think RecordArgs, FromRecordArgs, NatProductArgs, ProductArgs, FromProductArgs, SingletonProductArgs. These macros generally convert calls to a method into calls to a different method, while manipulating the arguments. For example for RecordArgs, lhs.method(x = 23, y = "foo", z = true) becomes lhs.methodRecord("x" ->> 23 :: "y" ->> "foo", "z" ->> true). I don't think Scala 3 macros offers a way to redirect method calls in such a dynamic manner.

4. Stuff that in some way manipulates source code in ways not exposed by the Scala 3 compiler. Think TypeOf, compileTime. This class of macros in some way serve as a poor mans inline, but manipulates the code as a string. As such I can't see any reasonable way to support it.

There are also the type quantifiers Shapeless exposes, which can be seen here.
https://github.com/milessabin/shapeless/blob/main/core/src/main/scala/shapeless/package.scala#L60

∃ I am fairly certain is not representable in Scala 3. ∀ might still be possible? If it is, it needs a new representation.

It's clear that none of these can be included in the Scala 3 artifact, but should they still be included in the Scala 2 artifact?

My current progress on porting Shapeless 2 to Scala 3 can be found here for anyone interested.
https://github.com/Katrix/shapeless/tree/feature/scala-3-port

[joroKr21]

@Katrix I think we should drop features that we can't port to Scala 3. My reasoning is that Shapeless 2.4 will almost certainly be the last binary breaking release of Shapeless 2, which means that we can't deprecate those features in 2.4 and drop them later. In any case it would be good to deprecate in 2.3 anything we are going to drop in 2.4. I don't think most of these features are particularly important:

1. Those features should be replaced by regular implicit resolution:

• Lazy is replaced with by-name implicits since Scala 2.13
• Strict is not necessary since Scala 2.13 (due to the implicit divergence checker)
• Cached is not really necessary given performance improvements in implicit search
• LowPriority is a failed experiment and can be replaced by standard implicits prioritization
• cachedImplicit is perhaps a bit problematic - worth a try to implement in Scala 3 but if not possible oh well

2. I think the biggest reason for type providers to exist was the lack of singleton types until Scala 2.13. So I think they won't be missed.
3. Yeah those are a bit too magic and go into the direction of changing the language, so I'm fine with dropping them. But just out of curiosity, why do you think they can't be implemented in Scala 3? Is it because we don't have access to the parameter names?
4. Whatever

∃ I am fairly certain is not representable in Scala 3. ∀ might still be possible? If it is, it needs a new representation.

What's the problem with those? Existential types?

[milessabin]

@Katrix I agree with @joroKr21 on all points.

∃ I am fairly certain is not representable in Scala 3. ∀ might still be possible? If it is, it needs a new representation.

Does anyone use these? They date from when shapeless was really just an experiment and they really don't have any practical use. I'm not planning to add anything similar to shapeless 3 and I'd be in favour of dropping them even if they could be ported to Scala 3.

[Katrix]

For those that want to follow along, and provide input and help along the way.

Progress to far:

• Implemented nat.ToInt and Nat implicit conversion
• Changed the encoding of FieldType slightly to work better with Scala 3
• Added basic implementation on Generic using Mirror
• Implemented the

Removals:

• Lazy and Strict removed. Handled by Scala's implicit divergence checker and by-name implicits.
• Removed type providers in: Coproduct, HList, Record, Witness, the, TypeOf, Union
• Removed forwarder traits NatProductArgs, ProductArgs, FromProductArgs, SingletonProductArgs, RecordArgs, FromRecordArgs
• Removed SelectManyAux and HListOps#selectMany as they used NatProductArgs

Questions:

• Record and Union used their type providers to create types of them easily. Should we add a type level ->> to replace this need?
  Instead of

  Record.`"x" -> Int, "y" -> String, "z" -> Boolean`.T
  Union.`"x" -> Int, "y" -> String, "z" -> Boolean`.T

  we could provide

  "x" ->> Int :: "y" ->> String :: "z" ->> Boolean :: HNil 
  "x" ->> Int :+: "y" ->> String :+: "z" ->> Boolean :+: CNil

• What is NatWith? In particular, this type class has macro implicit conversions. What are their purpose?

• What is WitnessWith? In particular, this type class has macro implicit conversions located in Widen. What are their purpose?

• Is Widen still needed?

• The macros for poly use openImplicits for it's macros to prevent diverging implicits. This is something not available in Scala 3 AFAIK. Is this still needed, and if so, how can be be replaced?

• Record and Union use Dynamic to construct instances of them. However, they are very restrictive in the arguments allowed here. Currently macros validate this, but some of these restrictions can be lifted to the method instead. This would however lead to worse error messages as they would just be more generic "wrong type" or "wrong amount of arguments" errors.

  As an example for Record, currently this is what we have

  def applyDynamic(method: String)(rec: Any*): HList = ???
  def applyDynamicNamed(method: String)(rec: Any*): HList = ???

  but it could be this instead.

  def applyDynamic(method: "apply")(): HNil = HNil
  def applyDynamicNamed(method: "apply")(rec: (String, Any)*): HList = ???

  Is this something worth doing, at the cost of slightly worse error messages? For the empty case (applyDynamic) we can just straight up get rid of a macro even in Scala 2 by doing this.

  • As RecordArgs is gone, CopyMethods in alacarte is no longer a thing. Is there anything we can do about this, or should we just remove CopyFacet as well?

  • How do we want to handle Typeable? Shapeless 3's typeable already exists as a seperate dependency and works just fine for Scala 3. Would it be reasonable to just depend on that on Scala 3, and proxy Shapeless 2's Typeable to Shapeless 3's Typeable?

[julienrf]

Here are my 2 cents about some of your questions:

• Record and Union used their type providers to create types of them easily. Should we add a type level ->> to replace this need?

It seems that currently there is no convenient syntax for expressing record types, so I would say yes to this proposal.

• Record and Union use Dynamic to construct instances of them. However, they are very restrictive in the arguments allowed here. Currently macros validate this, but some of these restrictions can be lifted to the method instead. This would however lead to worse error messages as they would just be more generic "wrong type" or "wrong amount of arguments" errors.

Since the purpose of Shapeless 2 on Scala 3 is mainly to ease cross-compiling, I don’t think it is a big issue to have worse error messages in Scala 3 because I expect people to cross-compile with Scala 2 and get the better error messages there. However, if it is possible to do the same validation in a macro in Scala 3 as well, I would go for that.

• How do we want to handle Typeable? Shapeless 3's typeable already exists as a seperate dependency and works just fine for Scala 3. Would it be reasonable to just depend on that on Scala 3, and proxy Shapeless 2's Typeable to Shapeless 3's Typeable?

I would copy Shapeless 3’s implementation in Shapeless 2 instead.

[milessabin]

Should we add a type level ->> to replace [Record and Union]

Yes, I think that's a good idea.

What is NatWith?
What is WitnessWith?

Both of these were intended to improve the ergonomics of using singleton types in the pre-SIP23 world ... they should go.

Is Widen still needed?

I don't know. I suggest taking it out and seeing what breaks.

The macros for poly use openImplicits for it's macros to prevent diverging implicits.

All the poly macros should just go with no replacement.

Record and Union use Dynamic to construct instances of them.

Now that we can assume literal types we should be able come up with some macro-free syntax to replace this, ideally aligned with the ->> type level operator we talked about above.

As RecordArgs is gone, CopyMethods in alacarte is no longer a thing.

I doubt that many people are using alacarte, so I'd drop it altogether, or demote to being an example with CopyFacet removed.

How do we want to handle Typeable?

Copy shapeless 3's.

[johnynek]

Just chiming in here to say I think a shapeless 2 for scala 3 would be very valuable.

I understand we might not get 100% source code compatibility, but getting that number as high as possible will really be a help to projects that crossbuild for scala 2, which I expect we will be dealing with for at least several more years.

[joroKr21]

#1200 does significant progress on this front but the GSoC wasn't enough to complete it.
It needs a new champion unless @Katrix plans to return to that work.

[Katrix]

For my masters thesis I need to do comparisons against Shapeless, so I might get back to it then, but I don't know how much I'll get to complete then, so would be good to look for someone else.

I will say that the majority of the remaining work from what I can see is to minimize Scala 3 bugs and irregularities, where it behaves differently from Scala 2. The library has theoretically been ported, only the tests remain, but those tests might show that something is quite broken currently.

[DmytroMitin]

https://stackoverflow.com/questions/69483366/scala-3-collection-partitioning-with-subtypes
https://stackoverflow.com/questions/74355212/shapeless3-and-annotations

[xuwei-k]

• add eu.timepit.refined.W for Scala 3 fthomas/refined#1246