Re-distribute Gurobi in Drake binary releases

[RussTedrake]

Edit: The original issue title was "Can users with a gurobi license (and no snopt license) use gurobi AND snopt?". We've retitled to reflect a new consensus now.

I wasn't able to find any documentation about this on our site. perhaps we can add it?

Can (or does) the binary distribution include gurobi bindings, but fail if the gurobi libraries aren't found?

I believe the alternative (having someone build from source and get snopt, despite not having the snopt source) is much harder to achieve?

@jamiesnape -- tagging you to get the current status first.

[jamiesnape]

No, we don't support providing your own commercial solver licenses or libraries at present. Enabling Gurobi support is currently a compile-time decision. We could add support for loading commercial solvers at runtime if available, but it would be an architectural change.

[jwnimmer-tri]

... and to further clarify the "add support for loading commercial solvers at runtime if available":

That would entail linking or loading solvers/gurobi_solver.pic.o, not just libgurobi80.so. Even with the delayed load of gurobi, we cannnot ship solvers/gurobi_solver.pic.o in any binary release, because it is a derived work of gurobi_c.h and we do not have (to my knowledge) permission to redistribute derived works of this form.

[RussTedrake]

i understand. thanks.

just to collect all of these thoughts in one place... the other alternative, I would think, is to have gurobi_solver.o always in the binary release, and have it check the license file (instead of loading the obj file) to confirm availability. would this be better? (conditional on us obtaining permission for redistributing it)

[jamiesnape]

Even with the delayed load of gurobi, we cannnot ship solvers/gurobi_solver.pic.o in any binary release, because it is a derived work of gurobi_c.h and we do not have (to my knowledge) permission to redistribute derived works of this form.

My reading of EULA suggests that part would not necessarily be a problem, but someone would need to check for any (other) constraints.

[jamiesnape]

just to collect all of these thoughts in one place... the other alternative, I would think, is to have gurobi_solver.o always in the binary release, and have it check the license file (instead of loading the obj file) to confirm availability. would this be better? (conditional on us obtaining permission for redistributing it)

I think we probably want a license check as well regardless to be nice to the end-user.

[jwnimmer-tri]

@RussTedrake The easiest solution along those lines would be if we have permission to redistribute both the gurobi80.so library as well as our object code derived works of gurobi_c.h such as solvers/gurobi_solver.pic.o, subject to Gurobi's default built-in license-checking. In that case, we can just enable it in binary releases by default with minimal code changes -- I guess just check the license prior to ChooseBestSolver making a decision, and tuning up any install glue code.

If we only have permission to redistribute solvers/gurobi_solver.pic.o, then we cannot unconditionally link gurobi80.so in the default release -- so either we'd need to make a new binary release flavor, or split out a -ldrake_solvers_gurobi mini-library, or use dlopen, or etc.

[jamiesnape]

My thoughts were along the lines of a dlopen of gurobi80.so, but certainly plenty of reasons to want to avoid that, if at all possible.

[EricCousineau-TRI]

Just to check: Can I ask what situation this accommodates, or what use case?
Is this to check alternatives against SNOPT (given the current version / config ping pong that we're playing)?

[jwnimmer-tri]

My impression of the user story was "Most people are happy to just use Drake binary releases. If they have a (site) license to Gurobi, it would be nice if that Just Worked out of the box. The need is especially acute for users without a site license to SNOPT -- in that case, even if they compile from source in order to use Gurobi, they then lose the ability to use SNOPT".

[RussTedrake]

correct. and that happens to account for most of the academic world, as gurobi is free for academics, but SNOPT is not.

[tri-ltyyu]

Use case understood. Is there a research group/class actively blocked by this to gauge timing? Wondering if there's a work around such as using Colab or asking the dept. to get a license for $600.

[jwnimmer-tri]

Since the next action on this issue is to see if MIT has the right to redistribute Gurobi stuff (per details above), maybe the easy answer is to reassign this issue to Russ, and he can pursue that question with MIT legal on whatever pace it requires?

[jwnimmer-tri]

I guess @tri-ltyyu could also own it and chase down MIT legal. All I'm really saying is that @jamiesnape isn't an appropriate assignee anymore.

[tri-ltyyu]

Happy to chase someone down to find out, but want to confirm if it's MIT legal or Gurobi legal that we want to get permission from? From Gurobi's EULA, I see the following, which makes me think we need permission from them first?
"You may not use, copy, modify, or distribute the Product, or make any copy, adaptation, transcription, or merge any portion thereof, unless expressly authorized by Gurobi in a separate written agreement."

[jamiesnape]

Ultimately, it would need to be Gurobi. Whether it is you or Russ or MIT who makes the request to Gurobi is a slightly different question. The license on CI is officially assigned to MIT CSAIL with Russ as the administrator.

[jwnimmer-tri]

... just check the license prior to ChooseBestSolver making a decision ...

FYI #13279 has implemented this now.

[RussTedrake]

Just noticed this for the first time: https://www.gurobi.com/license-center/for-online-courses/ I wonder if that changes anything?

[jamiesnape]

Difficult to say for certain. Probably not, but it could give you a different avenue to talking to Gurobi. The redistribute part of the license is the problem, and I do not suppose the new license changes that. I doubt it changes anything for Colab, for example.

[jwnimmer-tri]

Another option here might be to write a SolverInterface implementation in pure Python that calls into gurobipy (which is Gurobi's first-party python module). That might avoid the kinds of linker headaches anticipated above, even to the point where we could distribute that binding directly as part of pip install drake. It would not be in pydrake.all, and thus would only conditionally depend in import gurobipy being available.

[RussTedrake]

Just stumbled on this, which might be relevant for the "license on deepnote / colab (if that ever comes back)": https://support.gurobi.com/hc/en-us/articles/4409582394769-Google-Colab-Installation-and-Licensing

[jwnimmer-tri]

@RussTedrake with the uptick in some of the new optimal planning code in Drake and Drake-adjacent projects, it seems like everyone wants Gurobi, so then rebuilds Drake from source, but then loses SNOPT.

I'm thinking about resurrecting this work and trying to reach out to Gurobi people for a redistribution license.

Do you have any input before I start that? (email is fine too)

[RussTedrake]

No updates from me. I agree with your assessment!

[jwnimmer-tri]

To expand upon my proposal above (#10804 (comment))...

I'm proposing adding a new class pydrake.solvers.GurobiPySolver that implements SolverInterface in Python.

• Its implementation of "is available" would try to import gurobipy and return true iff that succeeds.
• Its implementation of "is enabled" would check whether os.environ["GRB_LICENSE_FILE"] is set to non-empty.
• Its implementation of "program attributes" stuff would delegate to the C++ pydrake.solvers.GurobiSolver.
  • That code in C++ is in gurobi_solver_common.cc so is compiled unconditionally.
• Its implementation of "solve" would be a re-implementation of gurobi_solver.cc in Python (~1300 lines of C++).
  • For helper functions like SetLinearConstraintDualSolutions that don't touch any Gurobi data types, we could move them to gurobi_solver_common.cc and bind them privately into pydrake, so that the GurobiPySolver could reuse them instead of re-implementing them.
• When the pydrake bindings are loaded, it would automatically insert GurobiPySolver into the ChooseBestSolver suite. If only C++ is being used (no pydrake), this new solver would not be part of ChooseBestSolver.

That would allow Python users of Drake to easily use Gurobi to solve their optimization programs with only a license key, without recompilation. C++ users of Gurobi would still need to rebuild Drake from source.

The ongoing maintenance burden would be that we need to maintain two copies of the gurobi_solver implementation indefinitely. With some careful refactoring for reuse, I hope we could get that down to only ~500 lines of code that's duplicated.

@hongkai-dai could you offer any general feedback or advice about this proposal?

[jwnimmer-tri]

I should also add -- since the program is convex the only Python/C++ calling cost overhead would be while iterating through the costs and constraints. During the solve, the are no nonlinear constraints that would require per-step callbacks crossing the C++/Python GIL boundary.

[jwnimmer-tri]

@RussTedrake @hongkai-dai I would be curious to hear your thoughts about the proposal upthread (see the prior two messages).

My proposal is that we solve this packaging issue for Python users only. Python users would be able to pip install drake gurobipy and then Gurobi would be usable (modulo a license key) for MathematicalProgram. C++ users would still need to rebuild from source. Drake C++ code that calls Solve() would be able to call gurobipy.

After the Python part is working, my proposal is to close this issue, and not attempt any C++ integration for our prepackaged releases.

[hongkai-dai]

Sorry I somehow missed your previous message.

I think implementing GurobiPySolver should work. I have been using gurobi's python interface in my personal project and got relatively familiar with it. I can work on this in my spare time. What is the timeline for adding GurobiPySolver?

[jwnimmer-tri]

I don't have any specific goal in mind for a release date of the feature. I'm sure many users would enjoy having it, but I haven't gotten any recent requests myself. (Russ would probably know more on that front.)

The main purpose of my question was to able to remove the "redistribute libgurobi.so" from my build system task list. If we can agree to give up on C++ support, I can focus my energy elsewhere.

[RussTedrake]

I think it will be an additional support burden (especially for Hongkai). But I agree it seems the best solution.

[RussTedrake]

Do we expect it could result in a significant performance hit?

[jwnimmer-tri]

#10804 (comment) is my current guess re: performance impact. We'll have to see how it turns out, though.

[jwnimmer-tri]

Actually, we may not need to use pygurobi for solving, only for providing the shared library. The beautiful https://github.com/yugr/Implib.so tool lets us do lazy loading of shared libraries the first time they are called, instead of only lazy binding. It seems like Mach-O might already have a similar feature built-in. I will continue to investigate, time permitting.

[RussTedrake]

That would be fantastic!