Skip to content

Change max size of DP table #1423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pavelvelikhov merged 4 commits into from
Jan 30, 2024
Merged

Change max size of DP table #1423

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b37345
Change max size of DP table
Dark-Avery Jan 30, 2024
4652438
Change size of bitset for graph joins
Dark-Avery Jan 30, 2024
10f2a02
minor fix
Dark-Avery Jan 30, 2024
7cc0377
minor fix
Dark-Avery Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions ydb/library/yql/dq/opt/dq_opt_join_cost_based.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1014,7 +1014,7 @@ std::shared_ptr<TJoinOptimizerNode> OptimizeSubtree(const std::shared_ptr<TJoinO
return PickBestNonReorderabeJoin(joinTree->LeftArg, joinTree->RightArg, joinTree->JoinConditions, joinTree->JoinType, ctx);
}

TGraph<64> joinGraph;
TGraph<128> joinGraph;
TVector<std::shared_ptr<IBaseOptimizerNode>> rels;
std::set<std::pair<TJoinColumn, TJoinColumn>> joinConditions;

Expand All @@ -1024,10 +1024,10 @@ std::shared_ptr<TJoinOptimizerNode> OptimizeSubtree(const std::shared_ptr<TJoinO
joinGraph.AddNode(i, rels[i]->Labels());
}

// Check if we have more rels than DPccp can handle (64)
// Check if we have more rels than DPccp can handle (128)
// If that's the case - don't optimize the plan and just return it with
// computed statistics
if (rels.size() >= 64) {
if (rels.size() >= 128) {
ComputeStatistics(joinTree, ctx);
return joinTree;
}
Expand Down Expand Up @@ -1055,7 +1055,7 @@ std::shared_ptr<TJoinOptimizerNode> OptimizeSubtree(const std::shared_ptr<TJoinO
YQL_CLOG(TRACE, CoreDq) << str.str();
}

TDPccpSolver<64> solver(joinGraph, rels, ctx);
TDPccpSolver<128> solver(joinGraph, rels, ctx);

// Check that the dynamic table of DPccp is not too big
// If it is, just compute the statistics for the join tree and return it
Expand Down Expand Up @@ -1172,7 +1172,7 @@ class TOptimizerNative: public IOptimizer {

TOutput JoinSearch() override {
auto dummyProviderCtx = TDummyProviderContext();
TDPccpSolver<64> solver(JoinGraph, Rels, dummyProviderCtx);
TDPccpSolver<128> solver(JoinGraph, Rels, dummyProviderCtx);
std::shared_ptr<TJoinOptimizerNode> result = solver.Solve();
if (Log) {
std::stringstream str;
Expand Down Expand Up @@ -1282,7 +1282,7 @@ class TOptimizerNative: public IOptimizer {
const std::function<void(const TString&)> Log;

TVector<std::shared_ptr<IBaseOptimizerNode>> Rels;
TGraph<64> JoinGraph;
TGraph<128> JoinGraph;
};

IOptimizer* MakeNativeOptimizer(const IOptimizer::TInput& input, const std::function<void(const TString&)>& log) {
Expand Down
2 changes: 1 addition & 1 deletion ydb/library/yql/providers/dq/common/yql_dq_settings.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ struct TDqSettings {
static constexpr ETaskRunnerStats TaskRunnerStats = ETaskRunnerStats::Basic;
static constexpr ESpillingEngine SpillingEngine = ESpillingEngine::Disable;
static constexpr ui32 CostBasedOptimizationLevel = 0;
static constexpr ui32 MaxDPccpDPTableSize = 10000U;
static constexpr ui32 MaxDPccpDPTableSize = 16400U;

};

Expand Down