Allow passing in non-optimizable parameters?

[johnmyleswhite]

Lately I've found myself constructing some pretty complicated closures to work around the current Optim interface.

As such, I'd like to propose a revised protocol for functions passed to Optim. Instead of working with a function f(x::Array) -> Float64, I'd like to work with functions of the form f(x::Array, extra::Any) where extra can be used to provide any additional information needed to evaluate f given x. This would make several things simpler:

• Fitting statistical models that need to be conditioned on observed data
• Implementing alternating minimization
• Working with functions that need to update several buffers during evaluation

What would people think of making this kind of breaking change? It would make Optim much more general, but might also make the interface less intuitive for newcomers since you'd want to do something like f(x, nothing) when your function doesn't depend on additional information.

[StefanKarpinski]

This is exactly what closures are for, it's a shame they are still such a performance hit. I hate to see API changes like this that may become unnecessary in the relatively near future. @JeffBezanson – any thoughts on how soon it might be plausible to get faster closures?

[johnmyleswhite]

I would love to have faster closures.

But I think the use case I'm currently dealing with would be problematic even with faster closures: in these applications, I randomly modify the dataset being closed over on every single iteration of my optimization algorithm, which necessitates a new function definition on every iteration if you use closures to track the current data set being optimized against. Given Julia's approach to JIT-ing functions, it's not clear to me that closures will ever be able to catch up with the interface I'm describing.

[johnmyleswhite]

Since I basically need this functionality to push a project at work forward, I think I'm going to start an Optim2 package to work out how I'd like to see things.

Would love to hear more what to expect from closures in future releases.

[mlubin]

I don't support changing the default interface to do this, it's horribly ugly, not something that I would want to put on a slide during a talk to sell Julia. What we might need is another level of abstraction over the optimization code so that both the current and the f(x,::Any) user interfaces can sit on top of it.

[StefanKarpinski]

It does strike me as not the right default interface.

[johnmyleswhite]

Any suggestions?

[johnmyleswhite]

If we could dispatch on the argument signature of functions, this could be solved by automatically wrapping f(x) with g(x, nothing) = f(x). But I don't know how to emulate that kind of dispatch at the moment.

[johnmyleswhite]

Since I'm on the topic of things I'd like to be able to do, the other functionality I've been needing is the ability to pass in a function that gets executed after every step. That lets you do things like SGD.

[mlubin]

I have a sense that it would be better to break apart the optimization a
bit, instead of trying to stuff everything into callbacks. I'm referring to
an interface where you can "drive" the optimization algorithm at a high
level, choosing when to iterate and providing the necessary information at
each step without needing to get into the details of how the iterations are
performed. A number of Fortran optimization codes have this interface, and
I think that something like this would provide the flexibility that you're
looking for.

[johnmyleswhite]

That sounds reasonable. I should look at a few other libraries and see how they handle this kind of thing.

[johnmyleswhite]

Breaking apart the functionality of optimize might be a good idea at some point, but I've started creating custom code for the two main cases that inspired this issue. Going to close now.