Skip to content

Conversation

@zhangstar333
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
the bitmap union maybe could call add_batch firstly, and the call add function,
so if add_batch function not update is_first variable, it's will be loss data in union.

    static void add(BitmapValue& res, const BitmapValue& data, bool& is_first) {
        if (UNLIKELY(is_first)) {
            res = data;
            is_first = false;
        } else {
            res |= data;
        }
    }

    static void add_batch(BitmapValue& res, std::vector<const BitmapValue*>& data, bool& is_first) {
        res.fastunion(data);
        is_first = false;  // before not set this
    }

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34060 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b59f9a1a320d3e7d2435b277677e6c8435bd449b, data reload: false

------ Round 1 ----------------------------------
q1	17632	5334	4974	4974
q2	1953	290	182	182
q3	10358	1341	728	728
q4	10309	1008	549	549
q5	8767	2433	2338	2338
q6	195	166	136	136
q7	925	746	629	629
q8	9330	1295	1158	1158
q9	6755	5132	5150	5132
q10	6949	2393	1987	1987
q11	502	297	273	273
q12	351	354	229	229
q13	17792	3685	3057	3057
q14	231	231	217	217
q15	568	490	470	470
q16	442	442	375	375
q17	609	893	387	387
q18	7689	7138	7069	7069
q19	1474	957	589	589
q20	355	346	227	227
q21	3865	3168	2408	2408
q22	1020	982	946	946
Total cold run time: 108071 ms
Total hot run time: 34060 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5182	5128	5125	5125
q2	241	323	225	225
q3	2123	2656	2331	2331
q4	1417	1828	1315	1315
q5	4249	4293	4415	4293
q6	218	173	132	132
q7	2016	1914	1729	1729
q8	2591	2758	2545	2545
q9	7099	7144	7113	7113
q10	3080	3257	2810	2810
q11	578	495	498	495
q12	704	778	629	629
q13	3560	3901	3310	3310
q14	297	303	289	289
q15	518	471	470	470
q16	480	501	462	462
q17	1170	1611	1319	1319
q18	7346	7232	7168	7168
q19	839	861	913	861
q20	1918	2049	1789	1789
q21	4874	4444	4313	4313
q22	1057	991	947	947
Total cold run time: 51557 ms
Total hot run time: 49670 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185134 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b59f9a1a320d3e7d2435b277677e6c8435bd449b, data reload: false

query1	960	406	393	393
query2	6522	1797	1815	1797
query3	6751	222	222	222
query4	26144	23401	22906	22906
query5	4341	645	496	496
query6	326	225	214	214
query7	4623	501	290	290
query8	291	238	219	219
query9	8611	2634	2634	2634
query10	476	356	270	270
query11	15416	15100	14815	14815
query12	167	109	111	109
query13	1658	535	426	426
query14	8852	6485	6317	6317
query15	205	203	168	168
query16	7338	641	481	481
query17	1214	719	597	597
query18	2027	413	308	308
query19	195	219	170	170
query20	120	118	121	118
query21	218	135	125	125
query22	4059	4105	4024	4024
query23	33890	32986	33036	32986
query24	8424	2395	2407	2395
query25	536	469	404	404
query26	934	268	153	153
query27	2722	520	344	344
query28	4297	2126	2116	2116
query29	731	567	450	450
query30	287	224	190	190
query31	905	804	746	746
query32	74	66	62	62
query33	546	363	330	330
query34	796	847	552	552
query35	797	834	732	732
query36	948	995	891	891
query37	118	104	76	76
query38	4099	4114	4063	4063
query39	1494	1439	1409	1409
query40	210	120	108	108
query41	70	57	60	57
query42	128	113	108	108
query43	482	506	483	483
query44	1337	826	827	826
query45	184	176	168	168
query46	863	1035	639	639
query47	1727	1786	1686	1686
query48	388	443	322	322
query49	709	490	393	393
query50	675	709	415	415
query51	4136	4180	4155	4155
query52	114	110	106	106
query53	242	270	183	183
query54	587	578	506	506
query55	96	90	88	88
query56	325	320	299	299
query57	1180	1203	1111	1111
query58	262	275	268	268
query59	2617	2687	2536	2536
query60	341	328	301	301
query61	125	122	135	122
query62	806	708	661	661
query63	226	199	192	192
query64	3606	1021	698	698
query65	4279	4158	4240	4158
query66	1032	422	319	319
query67	15733	15634	15112	15112
query68	8670	904	542	542
query69	493	308	292	292
query70	1224	1131	1126	1126
query71	488	335	306	306
query72	5527	4630	4599	4599
query73	691	585	353	353
query74	9116	8987	8620	8620
query75	4013	3200	2725	2725
query76	3619	1201	768	768
query77	779	386	306	306
query78	9941	10229	9325	9325
query79	1938	824	608	608
query80	611	511	450	450
query81	476	254	222	222
query82	424	133	102	102
query83	292	265	230	230
query84	292	105	90	90
query85	774	424	327	327
query86	334	305	302	302
query87	4507	4596	4313	4313
query88	2918	2270	2270	2270
query89	382	324	286	286
query90	1911	217	214	214
query91	144	144	112	112
query92	76	65	57	57
query93	1115	921	595	595
query94	680	420	273	273
query95	379	295	280	280
query96	486	565	282	282
query97	2720	2779	2647	2647
query98	227	224	203	203
query99	1400	1377	1291	1291
Total cold run time: 271239 ms
Total hot run time: 185134 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b59f9a1a320d3e7d2435b277677e6c8435bd449b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.45	0.40	0.42
query6	1.15	0.67	0.67
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.51
query10	0.58	0.56	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.65	0.61	0.62
query14	0.80	0.83	0.81
query15	0.90	0.88	0.87
query16	0.39	0.38	0.40
query17	1.08	1.05	1.05
query18	0.22	0.21	0.20
query19	2.03	1.88	1.88
query20	0.01	0.01	0.01
query21	15.41	0.89	0.54
query22	0.74	1.24	0.66
query23	14.90	1.38	0.64
query24	7.84	0.80	0.55
query25	0.50	0.13	0.11
query26	0.66	0.16	0.13
query27	0.06	0.06	0.05
query28	9.12	0.89	0.44
query29	12.68	4.01	3.31
query30	0.25	0.09	0.07
query31	2.83	0.62	0.39
query32	3.23	0.55	0.46
query33	3.00	3.07	3.08
query34	16.10	5.42	4.73
query35	4.81	4.81	4.82
query36	0.70	0.52	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.48 s
Total hot run time: 29.22 s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.61% (15117/26702)
Line Coverage 45.39% (135457/298424)
Region Coverage 44.49% (68094/153039)
Branch Coverage 39.06% (34957/89502)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.36% (16119/26268)
Line Coverage 50.75% (151313/298143)
Region Coverage 48.19% (86683/179860)
Branch Coverage 41.46% (42360/102170)

Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhangstar333 zhangstar333 merged commit ac6e694 into apache:master Jun 24, 2025
27 of 29 checks passed
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jul 9, 2025
…pache#52033)

Problem Summary:
the bitmap union maybe could call add_batch firstly, and the call add
function,
so if add_batch function not update is_first variable, it's will be loss
data in union.
```
    static void add(BitmapValue& res, const BitmapValue& data, bool& is_first) {
        if (UNLIKELY(is_first)) {
            res = data;
            is_first = false;
        } else {
            res |= data;
        }
    }

    static void add_batch(BitmapValue& res, std::vector<const BitmapValue*>& data, bool& is_first) {
        res.fastunion(data);
        is_first = false;  // before not set this
    }
```
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jul 9, 2025
…pache#52033)

Problem Summary:
the bitmap union maybe could call add_batch firstly, and the call add
function,
so if add_batch function not update is_first variable, it's will be loss
data in union.
```
    static void add(BitmapValue& res, const BitmapValue& data, bool& is_first) {
        if (UNLIKELY(is_first)) {
            res = data;
            is_first = false;
        } else {
            res |= data;
        }
    }

    static void add_batch(BitmapValue& res, std::vector<const BitmapValue*>& data, bool& is_first) {
        res.fastunion(data);
        is_first = false;  // before not set this
    }
```
@dataroaring dataroaring added the usercase Important user case type label label Jul 9, 2025
dataroaring pushed a commit that referenced this pull request Jul 9, 2025
…y sql (#52033) (#52999)

### What problem does this PR solve?
Problem Summary:
cherry-pick from master  (#52033)

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants