change call module

[vtjnash]

currently the code:

module MyModule
  import OtherModule.SomeType
  some_value = SomeType()
  some_value()
end

lowers to:

some_value = MyModule.call(OtherModule.SomeType)
MyModule.call(some_value)

however, i think it may be more backwards compatible with 0.3 and more flexible if it instead lowered to:

some_value = OtherModule.call(OtherModule.SomeType)
OtherModele.call(some_value)

the expected benefit is that I could have a module with a very large number of Types/call methods (coughGtk.jlcough #8745 (comment)) and not need to merge all of those into the global call method

baremodule Gtk
import Core.eval
global call
importall Base.Operators # (ignore the warning about conflicting with Gtk.call)
using Base

export MyType

using OtherLibraries

type MyType(...) ... end
...

this proposal would also fix #11295

[JeffBezanson]

To me, this would be an unacceptably surprising violation of how scoping works. I'm also not sure how to implement it: not every type has an obvious associated module. This seems like trying to claw back class-based OO. Given a type, this would apply some algorithm to convert it to a "class", then dispatch on that.

This is totally orthogonal to #11295. That issue requires call to be qualified somehow, this issue is a specific proposal for how to qualify it.

I think a better way to do this is to "chain" modules together using a definition like this:

Base.call{T<:GtkThing}(::Type{T}, args...) = Gtk.call(T, args...)

This does basically the same thing as your proposal, except the "algorithm for converting a type to a class" is explicit: send any subtype of GtkThing to the Gtk module.

Note this is the same as what we do in essentials.jl to layer Base on top of Core. Of course the problem is that these definitions can get tedious. If we can come up with some way to automate them it could be very helpful for modularity.

[vtjnash]

not every type has an obvious associated module

Why would jl_datatype_t->name->module not be assigned? (except perhaps at the very start of bootstrapping core, but call is also undefined then so it's a moot point)

This is totally orthogonal to #11295. That issue requires call to be qualified somehow, this issue is a specific proposal for how to qualify it.

Agreed. I just mentioned as a cross-link / breadcrumb for related issues.

Note this is the same as what we do in essentials.jl to layer Base on top of Core. Of course the problem is that these definitions can get tedious. If we can come up with some way to automate them it could be very helpful for modularity.

That's what this proposal was intended to do: automate that modularity. I disagree that this is particularly related to OO, since I don't think that usually has much to say about the behavior of calling an object. This also only applies to explicit T(x) syntax, and not call(T,x) which would continue to observe the normal scoping rules. Furthermore, functions are already a bit of a special case in that they always get passed to Core._apply; this proposal would hopefully extend that same behavior to all types (of getting handed off to the call method of ther original module).

This does basically the same thing as your proposal, except the "algorithm for converting a type to a class" is explicit: send any subtype of GtkThing to the Gtk module.

I was aware this was reversible, but thought the result was too absurd to describe originally. Your proposed code cripples Gtk.call so that it is extremely difficult to write the code inside of Gtk.jl (for example, I would have to rewrite array constructors as Base.call(Array, Int, 1) to make a simple vector). Of course, I can work around this by calling this Gtk.gobject_call, but the super type (GObject) actually lives in GLib, so ... back to square one. I find it suspicious that the current options seem to be to be very careful to only have one generic function 'call' function globally for code that expects to interoperate. (Core and Inference are special in that regard because they do not expect to interoperate).

In fairness, one problem I would note is that it becomes necessary to consider explicit chaining in the case where there is no applicable method in the primary module. It might therefore be necessary under this approach to walk up the subtyping hierarchy looking for a call method that is applicable to the given argument type tuple. An example of this usage is the fallback definition in base for turning call into convert for Types.

[JeffBezanson]

jl_datatype_t->name->module only works for DataType, not any type. What about call{T<:Union(A,B)}(::Type{T}, ...), where A and B might be from different modules?
Of course, you'd hardly ever define call for a non-DataType, but the point is to show how this is a bad match for our paradigm.

In class-based OO a call T.m(x) is basically lowered to get_func(T,:m)(T, x). This proposal lowers T(x) to get_func(T,:call)(T, x), which is the exact same idea, except only call is supported.

Walking up the subtyping hierarchy to find methods is even more class-based. This is adding an entirely different, second method resolution system to the language.

[JeffBezanson]

Also you probably remember this thread:
https://groups.google.com/forum/#!topic/julia-users/sk8Gxq7ws3w%5B1-25%5D

where some people did basically want to extend this kind of behavior to all generic functions, not just call, by only exporting functions that "refer" to a type defined in the same module.

[vtjnash]

Also you probably remember this thread:
https://groups.google.com/forum/#!topic/julia-users/sk8Gxq7ws3w%5B1-25%5D

where some people did basically want to extend this kind of behavior to all generic functions, not just call, by only exporting functions that "refer" to a type defined in the same module.

i had seen that, but didn't realize it might be possible to make this applicable in general for all functions. i don't see how that could work in the presence of full dispatch however. is there a useful subclass of type intersection for which dispatch is trivial, such that it might be cheaper to skip the method merge step for #8745? I imagine that the important ones to try to avoid the cost of merging are all the basic Operators: convert, call, getindex, setindex!, start, next, done, ==, +, etc.

[yuyichao]

Just curious, why couldn't we always resolve call into the Base call? This is how all other generic functions is done and call is already implicitly imported to all modules right?

Edit: just realized that expand(:(a[i])) is :(getindex(a, i)) not :(top(getindex(a, i)))

[JeffBezanson]

I've been trying to think of an approach to this issue that addresses the following:

• Retain some degree of modularity and encapsulation. The whole system can't be a single call function.
• Retain the extreme effectiveness of the existing call for constructors and functors.
• Combine anonymous and generic functions and make closures fast.
• Eliminate the need for AddFun et al. typeof(+) needs to give a better answer.
• Fix the awkwardness of needing to call the "right" call function (this issue, change call module #11452).

This is a tall order, but I finally thought of a slightly new angle. In brief, the idea is: combine TypeName and MethodTable. We want to somehow combine types and generic functions, since "calling" types works well now. In fact before call overloading we used to do this by putting a generic function "inside"
each type. However that did not work well, since (1) there were still generic functions that were not types, and (2) figuring out which methods to put in a type was messy due to type parameters: we must use dispatch to select a method for a type, not just pointer indirection.

A possible solution is to have a method table per type "family" (TypeName). A generic function is just an instance of a type that contains the function's method table.

ABI

We now have the nice property that for all x, isa(typeof(x),DataType). This lets us interpret the julia call f(x,y) (at the machine level, in pseudo-C syntax) as:

jl_apply_generic(jl_typeof(f)->name->methtable, {f, x, y}, 3)

Contrast with the current ABI, which is effectively

if (jl_is_function(f))
    f->fptr(f, {x, y}, 2)
else
    jl_apply_generic(call_func, {f, x, y}, 3)

Generic functions

A typical generic function would be the singleton instance of a type created just for that function. Adding methods to it modifies typeof(f).name.methtable.
You can dispatch on ::typeof(+).

Constructors

By the above logic, all constructors would be in the method table for the Core type DataType. This conspicuously fails to address the concern in the OP of this issue that Base.call has too many methods. That's too bad, but this design at least seems better than extending Base.call everywhere. Grouping constructors under DataType arguably makes more sense than a mere convention telling you to use Base.call.

On the minus side, this seems to stick us with the "one huge call function" design for constructors.

Closures

A closure is exactly like any other generic function, except its type has more than zero fields. Can't beat the elegance of that.

At the ABI level, we can ensure that the first argument to a singleton generic function is not actually passed, so simple functions keep their existing C compatibility.

Argument lists

This design is like the "use call overloading for everything" approach in that a function is always explicitly the first argument to itself. This could be quite annoying for reflection --- for convenience, do we sometimes add the first argument for you (e.g. when calling methods or code_typed)?
Not yet fully clear what to do here.

Functors

This is where the design really shines, as generic functions and functors become identical. You can write

abstract ArithFun

type Add <: ArithFun; end

+ = Add()

+(a::Int, b::Int) = # define methods as usual

map(f::typeof(+), a, b) = # special way to add two arrays

However, you cannot write the oft-wished-for

call(f::ScalarFun, a::Array) = map(f, a)

because call would no longer exist!

The way to get this feature would be to allow adding methods to abstract types. This is not the same as defining constructors for abstract types; those all go through DataType. Instead we would need syntax for stuffing methods into e.g. AbstractScalarFun.name.methtable, and then arrange for all subtypes to "inherit" the methods somehow.

This is where I admit that this proposal runs shamelessly afoul of my earlier objection to adding anything that smells like class-based OO. This whole idea adds classes with only one method ("call"). But I think the details here make everything still feel like generic functions. However, adding inheritance of method tables along the type hierarchy might be a step too far, and fortunately it is not strictly required.

Syntax

In this design the syntax f(x) = x can add methods to anything at all, since for all values typeof(f).name leads to a method table. This means you can accidentally add methods to numbers, for example. We'd have to figure out what restrictions, if any, to apply.

Builtins

Doesn't really matter, but builtins could also be generic functions marked as not-extensible. We can stick pointers to their C implementations in their method tables. Some can be removed, some can be moved to julia and implemented with ccall.

Misc

It's interesting to note that both TypeName and MethodTable have fields

jl_sym_t *name;
struct _jl_module_t *module;

plus a field called cache. This convergence might not be a coincidence!

Note: I wrote type.name.methtable above for clarity, but the method table would not necessarily be a separate object from type.name.

[vtjnash]

By the above logic, all constructors would be in the method table for the Core type DataType. This conspicuously fails to address the concern in the OP of this issue that Base.call has too many methods. That's too bad, but this design seems a bit better than simply extending Base.call everywhere. Grouping constructors under DataType arguably makes more sense than a mere convention telling you to use Base.call.

DataTypes are fairly special anyways, maybe we could get away with faking Type{Z}.module to recover modularity?

At the ABI level, we can ensure that the first argument to a singleton generic function is not actually passed, so simple functions keep their existing C compatibility.

we probably already do this, since passing a singleton by pointer would be a huge waste of an argument register. might just need to do a review of codegen to make sure it is fully consistent between isghost and singleton

[JeffBezanson]

faking Type{Z}.module to recover modularity

Very interesting idea. We need to think of a way to add that to the calling sequence efficiently.

[malmaud]

Would I be correct in saying this proposal seems spiritually in the same vein as https://github.com/timholy/FastAnonymous.jl, which also represents functions as instances of a singleton type and closures as extra fields on those types?

[JeffBezanson]

Yes.

[carnaval]

Cool. This looks like a good way of cleaning up both the concepts & the implementation.

I agree that method table inheritance is unacceptable, we already have a symmetric powerful selection mechanism, no need to add a hierarchical second one.

Could it be done somehow like this ? If I understood your idea correctly.

type AutoMap{T}; v::T; end
AutoMap.call{T}(m::AutoMap{T}, args...) = m.v(args...)
AutoMap.call{T}(m::AutoMap{T}, args::Vector...) = map(m.v, args...)

type AddF; end
const + = AddF()
+(x::Int,y::Int) = add_int(x,y)
# IIUC eqv to AddF.call(::AddF,::Int,::Int) = ...

const + = AutoMap(+)

It would probably need a way to make the setup more elegant but at least it is based on the proposed mechanism (and for singleton types like AddFun it should be zero cost ?).

[carnaval]

oh well, it wont work if there is no way to "redirect" method definitions to the contained object, that way we could add definitions later to AddF without polluting all AutoMap functions.

[carnaval]

pushing the insanity : make method definition a generic function. That way you get what I'm saying and sealed methods for free. (methoddef(::typeof(+), meth) = error())

[ScottPJones]

@JeffBezanson Are there any performance considerations with this new approach? What about changes for scoping of methods, in particular, for unexported methods, outside of the defining module?
@carnaval sealed methods?

[carnaval]

sealed methods are just methods you can't add a new definition to.

[JeffreySarnoff]

I like these gentypes.

The growing ball of call is bringing you down; that needs addressing.
Already, you are focused on cross-pollinating two flexible, powerful
and reasonably well understood approaches, types and generics.
+1 for gentypes supporting separable participation and so fixes it.

[yuyichao]

A closure is exactly like any other generic function, except its type has more than zero fields. Can't beat the elegance of that.

So actually a question I have since your talk at JuliaCon. How do you make sure multiple closures created from the same scope see changes made by other closures? By making each of the fields a Ref?

[carnaval]

if I got your question right, then this is only a problem for variable that are assigned to from inner scope. This property is decidable syntactically and those variable get treated specially (isBoxed predicate in our codegen speak, somewhat unfortunate name), and are placed inside a Box object. It's somewhat like a Ref but predates it by a lot.

[yuyichao]

I was just wondering how it would be represented when if we turn closures into callable types with fields as their captured variables.

[carnaval]

@JeffBezanson does this deserves a separate planning issue to discuss it or are you halfway through implementing it already ? :-)