[WIP] Macro extension interface

[joehuchette]

This is a preliminary sketch of an interface into the lower-level macro code in JuMP. It seems that most practical cases can be handled by swapping out the inner "kernel" function and leave all the indexing, looping, etc. untouched. For example, in @defVar the kernel function would be Variable, and in @addConstraint it's either addConstraint or addVectorizedConstraint.

The interface would be very simple: to roll your own macro, simply pass along the expression to the __defVar__ or __addConstraint__ functions, along with a kernelfunc kwarg. So, @defUnc could become

macro defUnc(args...) __defVar__(args...; kernelfunc=:Uncertain) end

Hopefully this relatively simple change will allow us to avoid exposing all the complicated code under the scenes to generate the indexing loops and all that, without losing too much expressiveness.

[joehuchette]

cc @ctjandra

[IainNZ]

Haven't started reading this yet, but here is what is needed by JuMPeR (so not much):
https://github.com/IainNZ/JuMPeR.jl/blob/jump010/src/robustmacro.jl

[mlubin]

In JuMPChance we need to pass keyword arguments through:

@defIndepNormal(m, x, mean=0, var=1)
@defVar(m, z)
@addConstraint(m, z*x >= -1, with_probability=0.95)

[joehuchette]

Should be feasible; we already do that for @addConstraint.

[IainNZ]

So at what point do I block this whole path?

[joehuchette]

Why would you need to?

[IainNZ]

I guess I just throw an error if someone passes in the :SDP type to Variable?

[joehuchette]

Yeah, there should be some mechanism for a nice fallback error message in the case of an unrecognized kwarg or if you only implement a few of the Variable methods implemented or something.

[IainNZ]

Would need a way to change all the error messages too.

I dunno about this, I see what you're getting at but I'd rather just pull out bits as utility functions than try to do it this way.

[mlubin]

I wonder if we could turn start, objective, inconstraints, and coefficients into keyword args for the Variable constructor.

[mlubin]

Also, JuMPChance IndependentNormal types don't have variable bounds or categories. How does this code handle that case?

[joehuchette]

We would need to document which methods are available for you to provide to the macros; if you don't implement it, you would get a no method error. We should also stipulate that you have your type <: AbstractJuMPScalar so that you get vectorized stuff for free, and so that on v0.4 we can use call overloading to provide a helpful error message in the case you don't support a particular method.

[joehuchette]

Also, we could dirty up the JuMP code a bit so that we don't use constructor methods for passing along those things, but manually update Model instead.

[mlubin]

What would the implementation of #481 look like with this new interface? It's hard to see how that can be easily abstracted over.

[joehuchette]

I imagine that would get de-sugared into a bunch of calls to

Variable(..., start[i,j,k])

anyway, so I'm not sure I see the problem, except that JuMP claims certain kwargs (e.g. start, inconstraints, etc.) as its own.

[mlubin]

But is the kernelfunc type supposed to implement a constructor which accepts all of these special keywords even if they're meaningless for the actual type?

[joehuchette]

Not unless it wants helpful error messages. The number of possible methods to implement needs some polishing, and it probably makes sense to change the start keyword into something that calls setValue instead of trying to mash everything through a constructor. That seems like "merely" an engineering problem that can be worked through, though.

[mlubin]

But setValue is also meaningless when applied to an IndepNormal in JuMPChance, for example.

[joehuchette]

Yes, but it's trivial to add a setValue(::AbstractJuMPScalar) = error("My helpful error message here") catch-all method, which works as long as IndepNormal <: AbstractJuMPScalar.

[mlubin]

I feel like we're trying to reuse too much machinery that's specific to JuMP variables, it's not a very clean abstraction.

[joehuchette]

Do you have a more concrete issue? I think inheritance that gets you vectorized operations and helpful error messages for free is a pretty nice abstraction...

[mlubin]

The inheritance aspect is fine. But if we want a solid extension interface at this level, we should be trying to make all of the JuMP-variable-specific processing just an extension itself instead of being baked into the definition of the abstraction layer. We don't want to start breaking extensions as soon as we tweak how we process @defVar.

[ctjandra]

I'd like to add another general use case. One of the things I'm doing is setting attributes to variables. The current interface for that looks like:

for s in SCENARIOS, i in CROPS
    DSPsolver.setVarSubproblem(m, x[s,i], s)
end

from here.

It would be nice if I could use JuMP macros to turn that into something like

@setVarSubproblem(m, x[s=SCENARIOS, i=CROPS], s)

which requires something fairly general.

[mlubin]

Do we have any planned API changes for extensions for 0.10, or just docs? If it's just docs, that doesn't need to hold up the release.

[joehuchette]

Probably just docs at this point.