Implement alternative back-end to Pyomo (e.g. linopy) #1064
Comments
|
Fast model building would also be beneficial for rolling horizon dispatch planning! I will try to make a minimal implementation into solph and open a PR. |
|
Here is a first implementation draft: https://github.com/oemof/oemof-solph/blob/feature/add-linopy-backend/solph-with-linopy-concept.py It is not beautiful, but it may be a good point to start a discussion on if and how to move forward. |
|
Hello, @fwitte! I am the author of PyOptInterface, an efficient modeling interface for mathematical optimization in Python. Its performance is quite competitive compared with existing solutions (faster than vendored Python bindings of some optimizers). PyOptInterface provides user-friendly expression-based API to formulate problems with different optimizers, and the lightweight handles of variables and constraints can be stored freely in built-in multidimensional container or Numpy ndarray. By the way, PyOptInterface supports efficient incremental modification of the model (adding/removing variables or constraints, modifying the coefficient and RHS of linear constraints, change the objective, etc). You can refer to the documentation of deleting variables, deleting constraints, modifying constraints and modifying objective. This might be very useful for rolling dispatch problems. As far as I know, the time to modify and solve in linopy is quite expensive because it needs to re-export the model again. If you are concerned with the performance to construct models, welcome to take a look at PyOptInterface and try it! I have prepared a benchmark between linopy and PyOptInterface where we construct a model, then modify and solve it in iterations. import time
import pandas as pd
import linopy
import pyoptinterface as poi
from pyoptinterface import gurobi
def linopy_main(N, K):
m = linopy.Model()
hours = pd.Index(range(N))
flow_names = ["source_b1", "b1_converter", "converter_b2", "b2_sink"]
flows = m.add_variables(lower=0, name="all-flows", coords=[hours, flow_names])
fixed_demand_con = m.add_constraints(
flows.loc[:, "b2_sink"] == 1, name="fixed demand"
)
m.add_constraints(
flows.loc[:, "b1_converter"] == flows.loc[:, "converter_b2"],
name="converter_efficiency",
)
m.add_constraints(
flows.loc[:, "source_b1"] == flows.loc[:"b1_converter"], name="b1 balance"
)
m.add_constraints(
flows.loc[:, "converter_b2"] == flows.loc[:, "b2_sink"], name="b2 balance"
)
variable_costs = 1
m.objective = variable_costs * flows.loc[:, "source_b1"]
m.solve(solver_name="gurobi")
for k in range(K):
fixed_demand_con.rhs.loc[k] = 2.0
m.solve(solver_name="gurobi")
def poi_main(N, K):
m = gurobi.Model()
hours = range(N)
flow_names = ["source_b1", "b1_converter", "converter_b2", "b2_sink"]
flows = m.add_variables(hours, flow_names, lb=0, name="all-flows")
def fixed_demand(h):
return m.add_linear_constraint(
flows[h, "b2_sink"], poi.Eq, 1.0, name=f"fixed demand {h}"
)
fixed_demand_con = poi.make_tupledict(hours, rule=fixed_demand)
def converter_efficiency(h):
return m.add_linear_constraint(
flows[h, "b1_converter"] - flows[h, "converter_b2"],
poi.Eq,
0.0,
name=f"converter efficiency {h}",
)
converter_efficiency_con = poi.make_tupledict(hours, rule=converter_efficiency)
def b1_balance(h):
return m.add_linear_constraint(
flows[h, "source_b1"] - flows[h, "b1_converter"],
poi.Eq,
0.0,
name=f"b1 balance {h}",
)
b1_balance_con = poi.make_tupledict(hours, rule=b1_balance)
def b2_balance(h):
return m.add_linear_constraint(
flows[h, "converter_b2"] - flows[h, "b2_sink"],
poi.Eq,
0.0,
name=f"b2 balance {h}",
)
b2_balance_con = poi.make_tupledict(hours, rule=b2_balance)
variable_costs = 1
objective = poi.quicksum(variable_costs * flows[h, "source_b1"] for h in hours)
m.set_objective(objective)
m.optimize()
for k in range(K):
m.set_normalized_rhs(fixed_demand_con[k], 2.0)
m.optimize()
def main(N, K):
linopy_times = {}
for k in range(K):
start = time.time()
linopy_main(N, k)
end = time.time()
linopy_times[k] = end - start
poi_times = {}
for k in range(K):
start = time.time()
poi_main(N, k)
end = time.time()
poi_times[k] = end - start
for k in range(K):
print(f"Run ({N}, {k}):")
linopy_time = linopy_times[k]
poi_time = poi_times[k]
print(f"\tlinopy: {linopy_time:.2f} seconds")
print(f"\tpoi: {poi_time:.2f} seconds")
if __name__ == "__main__":
main(8760, 10)On my computer, the output is: As the result shows, the one-shot time of PyOptInterface is nearly 1/4-1/3 of linopy. The time of linopy increases linearly as the number of "modify and solve" iterations |
|
I checked other boundary conditions:
|
|
Thanks for your feedback! For the third concern, the design of PyOptInterface is a minimalistic and thin wrapper of C API with no magic, so its implementations are essentially calling corresponding C API. You can look at https://github.com/metab0t/PyOptInterface/blob/master/lib/gurobi_model.cpp as an example. The logic of each function is quite clear and explicit. On the Python side, we only provide one built-in container type tupledict following the similar semantics of tupledict in gurobipy. It is up to the users to use high-level structures to store these variables and constraints freely, such as As for the maturity of PyOptInterface, some academic researchers I know (including myself) have been using it in research and engineering on power systems and have positive experience. The minimalistic design of PyOptInterface also significantly reduces its maintenance burden because the C API of optimizers are very stable even across major versions. Of course, I am looking forward to a bigger community of users and contributors as it gets more adoptions. |
|
The third point was no objection, rather a reason why checking code quality might be very important. Indeed, even on the long run, a slim one-person-project can be better maintained than a big, established one. Running the benchmark you posted on my local machine with HiGHS as a solver resulted in the following output: At least without adjusting the setup, linopy outperforms poi by far. Interestingly, the time to solve the poi model decreases for higher number of iterations. It feels like something strange is happening there. |
p-snft
changed the title
|
I can reproduce this result. Thank you very much for the test on HiGHS. I have figured out that there are four HiGHS API functions that slow down PyOptInterface significantly. I have a local fixed version that makes HiGHS fast again, and I will release a new version soon. |
|
@p-snft I have pushed a new release v0.2.2. Use This is the result of HiGHS on my computer: The first time is a little slower because PyOptInterface needs to load the dynamic library of HiGHS in the initialization step. |
|
The |
Or, potentially even replace pyomo (also see https://forum.openmod.org/t/linopy-for-oemof/4638).
I have recently used oemof-solph in a small dispatch optimization problem featuring two
Converters, aStorageand aGenericCHP. I profiled the execution times and roughly found:(not the actual times, the point is, that only half of the time is spent on solving)
There are a couple of issues with this for my use case:
Since I loose so much time in tasks which is not solving, I would like to bring this topic to the table.
I tested a very simple model (Source -> Bus -> Converter -> Bus -> Sink) in oemof-solph and linopy and got the following results (based on a single execution, I will retry a couple of times, so maybe do not look at the absolutes but more at the relative differences):
oemof-solph
linopy
The text was updated successfully, but these errors were encountered: