Skip to content

Handle out of memory and out of time limit issues gracefully #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 2 commits into from
May 21, 2025
Merged

Handle out of memory and out of time limit issues gracefully #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ab8d9d7
Add timelimit to compute related variables and catch out of memory ex…
rg20 May 20, 2025
98411dd
Make sure to resize to zero during early exit of finding related vari…
rg20 May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cpp/src/linear_programming/solve.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,11 @@ optimization_problem_solution_t<i_t, f_t> solve_lp(optimization_problem_t<i_t, f
} catch (const cuopt::logic_error& e) {
CUOPT_LOG_ERROR("Error in solve_lp: %s", e.what());
return optimization_problem_solution_t<i_t, f_t>{e, op_problem.get_handle_ptr()->get_stream()};
} catch (const std::bad_alloc& e) {
CUOPT_LOG_ERROR("Error in solve_lp: %s", e.what());
return optimization_problem_solution_t<i_t, f_t>{
cuopt::logic_error("Memory allocation failed", cuopt::error_type_t::RuntimeError),
Copy link
Contributor

@chris-maes chris-maes May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should a bad_alloc be a cuopt::logic_error? We might want to create a different cuopt exception for this.

Copy link
Contributor Author

@rg20 rg20 May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bad_alloc is thrown by rmm, unless we catch them somewhere and convert them to logic_error, which is what I am doing here

op_problem.get_handle_ptr()->get_stream()};
}
}

Expand Down
16 changes: 12 additions & 4 deletions cpp/src/mip/problem/problem.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,10 @@ void problem_t<i_t, f_t>::recompute_auxilliary_data(bool check_representation)
{
compute_n_integer_vars();
compute_binary_var_table();
compute_related_variables();

// TODO: speedup compute related variables
const double time_limit = 30.;
compute_related_variables(time_limit);
if (check_representation) check_problem_representation(true);
}

Expand Down Expand Up @@ -761,7 +764,7 @@ void problem_t<i_t, f_t>::compute_binary_var_table()
}

template <typename i_t, typename f_t>
void problem_t<i_t, f_t>::compute_related_variables()
void problem_t<i_t, f_t>::compute_related_variables(double time_limit)
{
auto pb_view = view();

Expand All @@ -788,6 +791,7 @@ void problem_t<i_t, f_t>::compute_related_variables()

i_t output_offset = 0;
i_t related_var_offset = 0;
auto start_time = std::chrono::high_resolution_clock::now();
for (i_t i = 0;; ++i) {
i_t slice_size = min(max_slice_size, n_variables - i * max_slice_size);
if (slice_size <= 0) break;
Expand Down Expand Up @@ -816,10 +820,14 @@ void problem_t<i_t, f_t>::compute_related_variables()
i_t related_var_base = related_variables.size();
related_variables.resize(related_variables.size() + array_size, handle_ptr->get_stream());

auto current_time = std::chrono::high_resolution_clock::now();
// if the related variable array would wind up being too large for available memory, abort
// TODO this used to be 1e9
if (related_variables.size() > 1e9) {
CUOPT_LOG_DEBUG("Computing the related variable array would use too much memory, aborting\n");
if (related_variables.size() > 1e9 ||
std::chrono::duration_cast<std::chrono::seconds>(current_time - start_time).count() >
time_limit) {
CUOPT_LOG_DEBUG(
"Computing the related variable array would use too much memory or time, aborting\n");
related_variables.resize(0, handle_ptr->get_stream());
related_variables_offsets.resize(0, handle_ptr->get_stream());
return;
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/mip/problem/problem.cuh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ class problem_t {
void recompute_auxilliary_data(bool check_representation = true);
void compute_n_integer_vars();
void compute_binary_var_table();
void compute_related_variables();
void compute_related_variables(double time_limit);
void fix_given_variables(problem_t<i_t, f_t>& original_problem,
rmm::device_uvector<f_t>& assignment,
const rmm::device_uvector<i_t>& variables_to_fix,
Expand Down
5 changes: 5 additions & 0 deletions cpp/src/mip/solve.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,11 @@ mip_solution_t<i_t, f_t> solve_mip(optimization_problem_t<i_t, f_t>& op_problem,
} catch (const cuopt::logic_error& e) {
CUOPT_LOG_ERROR("Error in solve_mip: %s", e.what());
return mip_solution_t<i_t, f_t>{e, op_problem.get_handle_ptr()->get_stream()};
} catch (const std::bad_alloc& e) {
CUOPT_LOG_ERROR("Error in solve_mip: %s", e.what());
return mip_solution_t<i_t, f_t>{
cuopt::logic_error("Memory allocation failed", cuopt::error_type_t::RuntimeError),
op_problem.get_handle_ptr()->get_stream()};
}
}

Expand Down
5 changes: 5 additions & 0 deletions cpp/src/routing/solve.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ assignment_t<i_t> solve(data_model_view_t<i_t, f_t> const& data_model,
} catch (const cuopt::logic_error& e) {
CUOPT_LOG_ERROR("Error in solve: %s", e.what());
return assignment_t<i_t>(e, data_model.get_handle_ptr()->get_stream());
} catch (const std::bad_alloc& e) {
CUOPT_LOG_ERROR("Error in solve: %s", e.what());
return assignment_t<i_t>(
cuopt::logic_error("Memory allocation failed", cuopt::error_type_t::RuntimeError),
data_model.get_handle_ptr()->get_stream());
}
}

Expand Down