Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hashpartition leak #2874

Closed
shfshihuafeng opened this issue Jan 23, 2024 · 3 comments
Closed

hashpartition leak #2874

shfshihuafeng opened this issue Jan 23, 2024 · 3 comments
Labels

Comments

@shfshihuafeng
Copy link
Contributor

Before submitting a bug report, please verify that you are using the most current version of Drill.

Describe the bug
hashpartition leak when allocate memory exception with OutOfMemoryException

To Reproduce
Steps to reproduce the behavior:

  1. prepare data for tpch 1s
  2. 20 concurrent for tpch sql8
  3. set direct memory 5g
  4. when it had OutOfMemoryException , stopped all sql.
  5. finding memory leak

Expected behavior
(1)i set {DRILL_MAX_DIRECT_MEMORY:-"5G"}
(2) i run sql8 (sql detail as Additional context) with 20 concurrent
(3) it had OutOfMemoryException when create hashPartion

Error detail, log output or screenshots
Unable to allocate buffer of size 262144 (rounded from 262140) due to memory limit (41943040). Current allocation: 20447232

Drill version
drill 2.0

Additional context
Add any other context about the problem here.

select  
o_year,  
sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share  
from ( 
select  
extract(year from o_orderdate) as o_year,  
l_extendedprice * 1.0 as volume,  
n2.n_name as nation  
from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region 
where  
p_partkey = l_partkey  
and s_suppkey = l_suppkey  
and l_orderkey = o_orderkey  
and o_custkey = c_custkey  
and c_nationkey = n1.n_nationkey  
and n1.n_regionkey = r_regionkey  
and r_name = 'ASIA'  
and s_nationkey = n2.n_nationkey  
and o_orderdate between date '1995-01-01'  
and date '1996-12-31'  
and p_type = 'LARGE BRUSHED BRASS') as all_nations  
group by o_year  
order by o_year
@shfshihuafeng
Copy link
Contributor Author

I fixed the bug see attachment
0001-bug-10227615-hashPartition-leak.patch

@cgivre
Copy link
Contributor

cgivre commented Jan 23, 2024

@shfshihuafeng Can you please submit this as a pull request?

@shfshihuafeng
Copy link
Contributor Author

ok @cgivre

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants