Error while trying to execute [solver] #581

EnricoSteez · 2022-09-25T21:19:03Z

EnricoSteez
Sep 25, 2022

I used PuLP on the same sort of problem for months now and I recently switched to a more complex version of the problem, with a lot of decision variables.
I am running the problem several (1260 to be precise) times, with all possible combination of the parameters that at the moment you see hardcoded in the simplified version.
When using 1000 items (N=1000), I get this error: pulp.apis.core.PulpSolverError: PuLP: Error while trying to execute gurobi_cl.
If I go down to 200 items , I get the error only after 27 iterations.
I think it has to to with the size of the problem, but I am not sure.
The size of the main decision variable X is a 3D binary matrix of size [1000,95,100]
The size of the secondary decision variable Z is a 2D binary matrix of size [95,100]

Do you think this is "too much"?

I cannot paste a minimal RUNNABLE version of the problem, since it is a research project and I cannot disclose some details, I hope this bit of code can help telling whether the size of the problem is too large for the free version of gurobi

item_ids = [ k for k in range(N)]
vm_volume_pairs = [vector of length 95]
logical_nodes = [vector of size 100]

x = pulp.LpVariable.dicts(
    "Placement",
    indices=(item_ids, vm_volume_pairs, logical_nodes),
    cat=constants.LpBinary,
)

z = pulp.LpVariable.dicts(
    "Which logical instances per node type are instantiated",
    indices=(vm_volume_pairs, logical_nodes),
    cat=constants.LpBinary,
)

solver = pulp.getSolver("GUROBI_CMD")

problem = pulp.LpProblem("Placement", pulp.LpMinimize)

# objective function
problem += (
    lpSum(
        [
            z[j][k] * RF * vm_costs[j.split("-")[0]]
            for j in vm_volume_pairs
            for k in logical_nodes
        ]
    )
    + lpSum(
        [
            z[j][k] * RF * max_storage * cost_volume_storage[j.split("-")[1]]
            for j in vm_volume_pairs
            for k in logical_nodes
        ]
    )
    + lpSum(
        [
            x[i][j][k]
            * (t_r[i] + t_w[i] * RF)
            * s[i]
            * params.IO_FACTOR[j.split("-")[1]]
            * cost_volume_iops[j.split("-")[1]]
            for i in range(N)
            for j in vm_volume_pairs
            for k in logical_nodes
        ]
    )
    + lpSum(
        [
            x[i][j][k] * (t_r[i] + t_w[i] * RF) * s[i] * cost_volume_tp[j.split("-")[1]]
            for i in range(N)
            for j in vm_volume_pairs
            for k in logical_nodes
        ]
    ),
    "Minimization ...",
)

for i in range(N):
    problem += lpSum([x[i][j][k] for j in vm_volume_pairs for k in logical_nodes]) == 1

for j in vm_volume_pairs:
    for k in logical_nodes:
        vmtype = j.split("-")[0]
        volumetype = j.split("-")[1]

        problem += lpSum([x[i][j][k] * s[i] for i in range(N)]) <= max_storage

        problem += (
            lpSum([x[i][j][k] * (t_r[i] + t_w[i] * RF) * s[i] for i in range(N)])
            * params.VM_IO_FACTOR
            <= max_vm_IOPS[vmtype] * RF
        )

        problem += (
            lpSum([x[i][j][k] * (t_r[i] + t_w[i] * RF) * s[i] for i in range(N)])
            * params.IO_FACTOR[volumetype]
            <= max_volume_IOPS[volumetype] * RF
        )

        problem += (
            lpSum([x[i][j][k] * (t_r[i] + t_w[i] * RF) * s[i] for i in range(N)])
            <= max_vm_bandwidths[vmtype] * RF
        )

        problem += (
            lpSum([x[i][j][k] * (t_r[i] + t_w[i] * RF) * s[i] for i in range(N)])
            <= max_volume_bandwidths[volumetype] * RF
        )

        problem += lpSum([x[i][j][k] for i in range(N)]) <= M * z[j][k]  # (16g)
        problem += lpSum([x[i][j][k] for i in range(N)]) * M >= z[j][k]  # (16h)

problem.solve(solver)

These are the information from the solver during the crashed run:

Thread count: 8 physical cores, 8 logical processors, using up to 8 threads
Optimize a model with 66700 rows, 1909500 columns and 15219000 nonzeros
Model fingerprint: 0x4e83d578
Variable types: 0 continuous, 1909500 integer (1909500 binary)
Coefficient statistics:
  Matrix range     [8e-02, 1e+05]
  Objective range  [2e-01, 5e+03]
  Bounds range     [1e+00, 1e+00]
  RHS range        [1e+00, 4e+06]
Found heuristic solution: objective 255257.61527
Presolve removed 38300 rows and 1900 columns (presolve time = 5s) ...
Presolve removed 40300 rows and 1900 columns (presolve time = 10s) ...
Presolve removed 40400 rows and 1900 columns (presolve time = 15s) ...
Presolve removed 40400 rows and 1900 columns (presolve time = 20s) ...
Presolve removed 40400 rows and 1900 columns
Presolve time: 22.60s
Presolved: 26300 rows, 1907600 columns, 7131400 nonzeros
Variable types: 0 continuous, 1907600 integer (1907600 binary)
Found heuristic solution: objective 149584.51385

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Ordering time: 0.00s

Barrier statistics:
 AA' NZ     : 1.882e+04
 Factor NZ  : 2.925e+04 (roughly 4 MB of memory)
 Factor Ops : 3.502e+06 (less than 1 second per iteration)
 Threads    : 1
                                                                                                                                                       [75/1932]
                  Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
   0   4.66990765e+04 -6.63266090e+06  2.29e+00 1.05e+03  7.46e+02    33s
   1   1.35953632e+04 -3.07756214e+06  6.18e-01 1.58e+03  2.69e+02    33s
   2   1.62351581e+03 -1.89090791e+05  7.29e-16 8.64e-12  9.87e+00    33s
   3   1.53798088e+03 -5.40001075e+02  6.94e-17 4.40e-12  1.08e-01    33s
   4   6.94741682e+02  1.99033923e+02  6.25e-17 7.28e-12  2.57e-02    33s
   5   2.60051838e+02  1.78239120e+02  8.33e-17 7.28e-12  4.24e-03    33s
   6   2.33779033e+02  2.15804680e+02  5.55e-17 7.28e-12  9.31e-04    33s
   7   2.22261599e+02  2.20426923e+02  8.33e-17 3.64e-12  9.50e-05    33s
   8   2.20986559e+02  2.20974916e+02  1.98e-15 3.64e-12  6.03e-07    33s
   9   2.20979459e+02  2.20979448e+02  2.80e-15 1.82e-12  6.03e-10    33s
  10   2.20979452e+02  2.20979452e+02  7.77e-16 7.28e-12  6.03e-13    33s

Barrier solved model in 10 iterations and 32.63 seconds (21.97 work units)
Optimal objective 2.20979452e+02


Root crossover log...

      92 DPushes remaining with DInf 0.0000000e+00                33s
       0 DPushes remaining with DInf 0.0000000e+00                33s

       0 PPushes remaining with PInf 0.0000000e+00                33s

  Push phase complete: Pinf 0.0000000e+00, Dinf 2.5650593e-12     33s


Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
      49    2.2097945e+02   0.000000e+00   0.000000e+00     33s
      49    2.2097945e+02   0.000000e+00   0.000000e+00     33s

Use crossover to convert LP symmetric solution to basic solution...

Root crossover log...

       0 DPushes remaining with DInf 0.0000000e+00                33s

   19801 PPushes remaining with PInf 0.0000000e+00                33s
       0 PPushes remaining with PInf 0.0000000e+00                33s
Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
   26855    2.2097945e+02   0.000000e+00   0.000000e+00     33s
   26855    2.2097945e+02   0.000000e+00   0.000000e+00     34s
Concurrent spin time: 0.39s

Solved with primal simplex

Root relaxation: objective 2.209795e+02, 27909 iterations, 9.35 seconds (5.93 work units)
Total elapsed time = 35.80s

Nodes    |    Current Node    |     Objective Bounds    |   Work
 Expl Unexpl  |   Obj  Depth IntInf  |   Incumbent BestBd Gap |        It/Node Time

     0     0  220.97945    0    6 149584.514  220.97945   100%     -   41s
H    0     0                     967.8900000  220.97945  77.2%     -   42s
H    0     0                     825.2950000  220.97945  73.2%     -   74s
     0     0  220.97945    0   12  825.29500  220.97945  73.2%     -   89s
     0     0  220.97945    0   22  825.29500  220.97945  73.2%     -  153s
H    0     0                     806.5750000  220.97945  72.6%     -  155s
     0     0  220.97945    0  296  806.57500  220.97945  72.6%     -  180s
     0     0  220.97945    0  278  806.57500  220.97945  72.6%     -  231s
     0     0  220.97945    0  273  806.57500  220.97945  72.6%     -  235s
     0     0  220.97945    0  230  806.57500  220.97945  72.6%     -  245s
     0     0  220.97945    0  247  806.57500  220.97945  72.6%     -  251s
     0     0  220.97945    0   10  806.57500  220.97945  72.6%     -  260s
     0     0  220.97945    0   14  806.57500  220.97945  72.6%     -  263s
     0     0  220.97945    0   34  806.57500  220.97945  72.6%     -  270s
     0     0  220.97945    0   41  806.57500  220.97945  72.6%     -  274s
     0     0  220.97945    0   22  806.57500  220.97945  72.6%     -  282s
     0     0  220.97945    0   26  806.57500  220.97945  72.6%     -  301s
     0     0  220.97945    0    7  806.57500  220.97945  72.6%     -  307s
     0     0  220.97945    0   22  806.57500  220.97945  72.6%     -  311s
     0     0  220.97945    0    9  806.57500  220.97945  72.6%     -  319s
     0     0  220.97945    0    9  806.57500  220.97945  72.6%     -  327s
     0     2  220.98000    0    9  806.57500  220.98000  72.6%     -  484s
     1     4  227.56667    1   41  806.57500  220.98000  72.6%   219  500s
     3     8  230.32877    2   17  806.57500  220.98000  72.6%   265  512s
     7    16  236.66667    3   30  806.57500  222.75800  72.4%   887  547s
    15    24  236.66667    4   29  806.57500  223.50377  72.3%   552  575s
    23    32  241.30573    4   26  806.57500  225.21500  72.1%   423  603s
    31    40  237.82643    5   22  806.57500  226.46755  71.9%   345  877s
H   32    40                     729.5900000  226.46755  69.0%   334  877s
    39    48  240.00000    6   27  729.59000  228.81500  68.6%   308  904s
    47    60  252.18578    7   35  729.59000  241.30573  66.9%   385  920s
    60    71  256.14116    8   37  729.59000  241.30573  66.9%   324  953s
    72    82  256.62710    9   45  729.59000  241.30573  66.9%   721 1050s
    86    99  256.74085   10   45  729.59000  241.30573  66.9%   625 1062s
   107   115  269.40014   12   30  729.59000  241.30573  66.9%   541 1078s
   126   126  274.36049   13   52  729.59000  241.30573  66.9%   579 1125s
   139   138  278.20082   14   48  729.59000  241.30573  66.9%   703 1150s
   155   146  287.98768   15   43  729.59000  241.30573  66.9%   713 1163s
   170   147  292.66367   16   46  729.59000  241.30573  66.9%   730 1436s
H  177   147                     727.6650000  241.30573  66.8%   726 1436s
   184   155  293.56672   17   46  727.66500  241.30573  66.8%   732 1451s
   204   163  335.02273   21   42  727.66500  241.30573  66.8%   735 1484s
Traceback (most recent call last):
  File "/home/vagrant/HybridDB/problem3D.py", line 522, in <module>
    main(N, size, throughput, skew, read_percent, filename, dist)
  File "/home/vagrant/HybridDB/problem3D.py", line 182, in main
    problem.solve(solver)
  File "/home/vagrant/.local/lib/python3.10/site-packages/pulp/pulp.py", line 1913, in solve
    status = solver.actualSolve(self, **kwargs)
File "/home/vagrant/.local/lib/python3.10/site-packages/pulp/apis/gurobi_api.py", line 397, in actualSolve
    raise PulpSolverError("PuLP: Error while trying to execute " + self.path)
pulp.apis.core.PulpSolverError: PuLP: Error while trying to execute gurobi_cl
Error, stopping

BrianWangDL · 2023-10-20T10:46:30Z

BrianWangDL
Oct 20, 2023

Today I met the same problem.Have you solved it?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while trying to execute [solver] #581

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Error while trying to execute [solver] #581

EnricoSteez Sep 25, 2022

Replies: 1 comment

BrianWangDL Oct 20, 2023

EnricoSteez
Sep 25, 2022

BrianWangDL
Oct 20, 2023