Would type classes mirroring shapeless.ops.(hlist|record) be welcome?

[mrdziuban]

I maintain an open source library at my company that has made it possible to cross-build a large, shapeless-heavy app for scala 2 and 3. Until recently I had just defined the minimal set of ops needed for my codebase, but I've been working on fleshing out the remaining ones and I'm nearly done (with a few exceptions that don't make sense anymore). I'd be happy to port my changes to this repo and open a pull request if it would be received well!

These use scala.Tuple directly instead of defining a separate HList type. It would cover the ops here:

• https://github.com/milessabin/shapeless/blob/main/core/shared/src/main/scala/shapeless/ops/hlists.scala
• https://github.com/milessabin/shapeless/blob/main/core/shared/src/main/scala/shapeless/ops/records.scala

as well as the syntax that uses those ops here:

• https://github.com/milessabin/shapeless/blob/main/core/shared/src/main/scala/shapeless/syntax/hlists.scala
• https://github.com/milessabin/shapeless/blob/main/core/shared/src/main/scala/shapeless/syntax/records.scala

[joroKr21]

That's quite interesting. I think if we include them, they should be in a separate module. So far the strategy has been to add methods that work with polymorphic lambdas.

How do you manage to cross compile if you're using a different data structure between Scala 2 / Scala 3?

[mrdziuban]

How do you manage to cross compile if you're using a different data structure between Scala 2 / Scala 3?

I have Tuple aliased to HList, EmptyTuple aliased to HNil, and *: aliased to :: -- https://github.com/mblink/typify/blob/more-shapeless-ops/tuple/shared/src/main/scala-2.13/typify/tuple/TuplePackageCompat.scala#L12-L16. So all downstream code looks like it's using Tuples but it's really using shapeless under the hood in scala 2.

I should have clarified originally -- I don't intend to contribute everything to support cross-building, I'm happy to continue maintaining that compatibility layer for my own code. I was just thinking that having the ops type classes live in shapeless would be valuable

[joroKr21]

Wow, that's very clever. I think it could be valuable to even to include the aliases in a separate compat module.

[mrdziuban]

I'm definitely open to including a compat module.

To add another layer of complexity to this, I've discovered that Tuple#tail is significantly less efficient than HList#tail. I asked in discord and the general reasoning is that Tuples are still encoded like they were in scala 2, and have efficient indexing, but inefficient head/tail decomposition. This is especially noticeable with the inductive implicit approach that most shapeless type classes use, where the recursive instance calls tail.

As an example, I wrote a benchmark to test the inductive implicit encoding of shapeless.ops.hlist.Remove. Each method in the benchmark removes an element, 1 through 10, from an HList/Tuple of 10 elements . Both HList and Tuple slow down as you get towards the end of the list, but Tuple performs much worse overall.

HList

remove0   thrpt    5  490473290.904 ± 3129444.721  ops/s
remove1   thrpt    5  457464619.715 ± 1096509.924  ops/s
remove2   thrpt    5   96264525.108 ± 1346279.517  ops/s
remove3   thrpt    5   72316998.768 ±  256008.018  ops/s
remove4   thrpt    5   58088517.695 ±  692081.233  ops/s
remove5   thrpt    5   46468689.588 ±  853738.032  ops/s
remove6   thrpt    5   39467654.695 ±  176281.706  ops/s
remove7   thrpt    5   32712129.452 ±  193437.405  ops/s
remove8   thrpt    5   28621659.869 ±  175725.570  ops/s
remove9   thrpt    5   25141670.252 ±  106827.994  ops/s
remove10  thrpt    5   20571384.988 ±   22875.596  ops/s

Tuple

remove0   thrpt    5  189075036.966 ± 455956.055  ops/s
remove1   thrpt    5   64574108.974 ± 262711.920  ops/s
remove2   thrpt    5   34426588.155 ± 215067.833  ops/s
remove3   thrpt    5   24067830.980 ± 190722.753  ops/s
remove4   thrpt    5   18490813.316 ± 527833.460  ops/s
remove5   thrpt    5   15125573.665 ±  27794.386  ops/s
remove6   thrpt    5   13118493.823 ± 184412.594  ops/s
remove7   thrpt    5   11426432.719 ± 224069.080  ops/s
remove8   thrpt    5   10240794.723 ±  68059.927  ops/s
remove9   thrpt    5    8963642.721 ±  27013.125  ops/s
remove10  thrpt    5    8389338.971 ±  17741.976  ops/s

All this said, do you think it would be better to redefine an HList type for scala 3 and provide conversions to/from native Tuples?

And regardless of the answer to that, should I go ahead and start porting the type classes into shapeless and open a PR when ready? If so, let me know if you have any thoughts on the name of the new module and/or the package structure that the code should follow.

[joroKr21]

In that case it might be better to try and finally finish cross-compiling Shapeless 2 to Scala 3: milessabin/shapeless#1200

Or start from scratch with a less ambitious version. I think Generic from mirrors would be enough.

[Katrix]

Generic from mirrors already exists in the Shapeless 2 for Scala 3 PR, so if something less ambitious is wanted, then one would just have to copy-paste that out. That's roughly what I did for a benchmark for my master thesis, and it worked just fine. IIRC the problem for a more general port was that the Scala 3 compiler didn't work well with some type programming stuff.