[YSQL] Allow remote filters in YB Bitmap Scans #21141

timothy-e · 2024-02-22T16:40:08Z

Description

CREATE TABLE test_remote (a INT, b INT, c INT);
CREATE INDEX ON test_remote (a ASC);
INSERT INTO test_remote SELECT i, i * 2, i * 3 FROM generate_series(1, 10000) i;

Observe that a Index Only Scan can use a remote filter to avoid fetching extra rows from the index:

EXPLAIN (COSTS OFF) SELECT * FROM test_remote WHERE a < 100 AND a % 2 = 0 AND c % 2 = 0;
                         QUERY PLAN
------------------------------------------------------------
 Index Only Scan using test_remote_a_b_c_idx on test_remote
   Index Cond: (a < 100)
   Remote Filter: (((a % 2) = 0) AND ((c % 2) = 0))
(3 rows)

But a bitmap scan cannot:

/*+ BitmapScan(test_remote) */ EXPLAIN (COSTS OFF) SELECT * FROM test_remote WHERE a < 100 AND a % 2 = 0 AND c & 2 = 0;
                    QUERY PLAN
--------------------------------------------------
 YB Bitmap Table Scan on test_remote
   Filter: (((a % 2) = 0) AND ((c & 2) = 0))
   ->  Bitmap Index Scan on test_remote_a_b_c_idx
         Index Cond: (a < 100)
(4 rows)

Ideally the behaviour could look like:

/*+ BitmapScan(test_remote) */ EXPLAIN (COSTS OFF) SELECT * FROM test_remote WHERE a < 100 AND a % 2 = 0 AND c & 2 = 0;
                    QUERY PLAN
--------------------------------------------------
 YB Bitmap Table Scan on test_remote
   ->  Bitmap Index Scan on test_remote_a_b_c_idx
         Index Cond: (a < 100)
         Remote Filter: (((a % 2) = 0) AND ((c & 2) = 0))
(4 rows)

Issue Type

kind/enhancement

Warning: Please confirm that this issue does not contain any sensitive information

I confirm this issue does not contain any sensitive information.

Summary: Pushing down filters prevents us from fetching unnecessary rows. In the bitmap scan case, if we can push down a filter at the Bitmap Index Scan level, it saves us from 1. retrieving the ybctid 2. using that ybctid in any bitmap operations (AND, OR) 3. retrieving the row corresponding to the ybctid. With this diff, both YB Bitmap Table Scans and Bitmap Index Scans are capable of pushing down filters. == YB Bitmap Table Scan pushdown == YB Bitmap Table Scans can push down any clause. The pushdown logic is similar to the recheck logic - it only pushes down a filter if it is necessary. For example, ```lang=sql /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=6.91..11.21 rows=10 width=12) -> BitmapOr (cost=6.91..6.91 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (b < 1000) (6 rows) ``` does not require a remote or local filter because it knows that the index scans will return only correct results. ``` /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000 AND c < 1; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=7.06..11.41 rows=10 width=12) Remote Filter: ((a < 1000) OR ((b < 1000) AND (c < 1))) -> BitmapOr (cost=7.06..7.06 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (b < 1000) (7 rows) ``` This requires a filter, and this filter can be pushed down. == Bitmap Index Scan pushdown == There are two cases where filters can be pushed down for bitmap index scans. === 1. If the filter clause is at the same nesting level as the index qual in the boolean operation tree === During planning, when the OR is broken up into separate indexes, the pushdown conditions are extracted for each branch of the OR. This allows us to get useful filters for pushdown, knowing that a particular index scan is concerned only by a portion of the whole condition. For example, in the index below, the first Bitmap Index Scan has an index condition `unique1 < 5` and a remote filter `unique1 % 2 = 0`. ```lang=sql /*+ BitmapScan(tenk1) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM tenk1 WHERE (unique1 < 5 AND unique1 % 2 = 0) OR unique2 < 3; QUERY PLAN ----------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual rows=6 loops=1) Remote Filter: (((unique1 < 5) AND ((unique1 % 2) = 0)) OR (unique2 < 3)) Storage Table Read Requests: 1 Storage Table Rows Scanned: 6 -> BitmapOr (actual rows=6 loops=1) -> Bitmap Index Scan on tenk1_unique1 (actual rows=3 loops=1) Index Cond: (unique1 < 5) Remote Index Filter: ((unique1 % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 5 -> Bitmap Index Scan on tenk1_unique2 (actual rows=3 loops=1) Index Cond: (unique2 < 3) Storage Index Read Requests: 1 Storage Index Rows Scanned: 3 Storage Read Requests: 3 Storage Rows Scanned: 14 Storage Write Requests: 0 Storage Flush Requests: 0 (18 rows) ``` When creating a bitmap scan plan, the planner breaks the OR into two parts. a: `(unique1 < 5 AND unique1 % 2 = 0)` b: `unique2 < 3` If it can find an index scan for each part, then a bitmap scan is a valid plan. (If there was a third clause on an unindexed field, c: `string1 = 'hi'`, then there would not be an index to answer each part of the OR so we cannot use a bitmap scan here). The default PG behaviour is to extract index clauses from each branch, so PG would identify `unique1 < 5` for a and `unique2 < 3` for b. With this diff, Yugabyte goes a step further. If a condition is not a valid index condition (e.g. `unique1 % 2 = 0`, then we test if it can be pushed down using the index columns. If yes, then we store it as a valid pushdown filter for a bitmap index scan. So we identify: | | index qual | filter qual | | a | `unique1 < 5` | `unique1 % 2 = 0` | | b | `unique2 < 3` | | Since we found an index path for each branch of the OR, we've found a valid bitmap scan path. === 2. All columns referenced by the top-level condition are contained in the index === In this case, we can push down the entire condition (or a portion of the top-level AND clause) to the index with the included columns. We cannot partially pushdown top-level ORs because this index doesn't know enough about the other branches, but that case was already handled above. For example, in the example below, the first Bitmap Index Scan pushes down the entire condition `(((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0)))` on `multi_c_a_idx` because the every column in the condition can be checked by this index. ```lang=sql /*+ BitmapScan(multi) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM multi WHERE (a < 50 AND a % 2 = 0) AND (c <= 100 AND c % 3 = 0); QUERY PLAN ------------------------------------------------------------------------------------------- YB Bitmap Table Scan on multi (actual time=3.655..3.666 rows=16 loops=1) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.014 ms Storage Table Rows Scanned: 16 -> BitmapAnd (actual time=2.492..2.492 rows=16 loops=1) -> Bitmap Index Scan on multi_c_a_idx (actual time=2.550..2.550 rows=16 loops=1) Index Cond: (c <= 100) Storage Index Filter: ((a < 50) AND ((a % 2) = 0) AND ((c % 3) = 0)) Storage Index Read Requests: 1 Storage Index Read Execution Time: 1.778 ms Storage Index Rows Scanned: 33 -> Bitmap Index Scan on multi_pkey (actual time=1.288..1.288 rows=24 loops=1) Index Cond: (a < 50) Storage Index Filter: ((a % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.164 ms Storage Table Rows Scanned: 49 Planning Time: 15.273 ms Execution Time: 4.362 ms Storage Read Requests: 2 Storage Read Execution Time: 2.942 ms Storage Rows Scanned: 82 Storage Write Requests: 0 Catalog Read Requests: 19 Catalog Read Execution Time: 31.753 ms Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 34.695 ms Peak Memory Usage: 119 kB (26 rows) ``` === 3. All columns referenced by the one of the top-level clauses (implicitly ANDed) are contained in the index === Consider the `tenk1` table: ```lang=sql \d tenk1 Table "public.tenk1" ... Indexes: "tenk1_hundred" lsm (hundred ASC) "tenk1_thous_tenthous" lsm (thousand ASC, tenthous ASC) "tenk1_unique1" lsm (unique1 ASC) "tenk1_unique2" lsm (unique2 ASC) ``` Then the query can push down the `tenthous % 2 = 0` condition to the `thousand` bitmap index scan. Even though it will still need to be validated at the Table Scan layer (to check the results from the `unique2` bitmap index scan, it is efficient to reduce the number of rows retrieved. ``` explain (analyze, costs off, dist) /*+ bitmapscan(tenk1) */ select * from tenk1 where thousand < 10 and unique2 < 1000 and tenthous % 2 = 0; QUERY PLAN -------------------------------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual time=17.917..17.929 rows=4 loops=1) Storage Filter: ((tenthous % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.170 ms Storage Table Rows Scanned: 4 -> BitmapAnd (actual time=16.422..16.422 rows=4 loops=1) -> Bitmap Index Scan on tenk1_unique2 (actual time=13.315..13.315 rows=1000 loops=1) Index Cond: (unique2 < 1000) Storage Index Read Requests: 1 Storage Index Read Execution Time: 10.981 ms Storage Index Rows Scanned: 1000 -> Bitmap Index Scan on tenk1_thous_tenthous (actual time=2.845..2.845 rows=50 loops=1) Index Cond: (thousand < 10) Storage Index Filter: ((tenthous % 2) = 0) Storage Index Read Requests: 1 Storage Index Read Execution Time: 2.407 ms Storage Index Rows Scanned: 100 Planning Time: 0.894 ms Execution Time: 18.702 ms Storage Read Requests: 3 Storage Read Execution Time: 14.559 ms Storage Rows Scanned: 1104 Storage Write Requests: 0 Catalog Read Requests: 0 Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 14.559 ms Peak Memory Usage: 180 kB (28 rows) ``` Jira: DB-10083 Test Plan: Jenkins: build platform: alma8 ``` ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressYbBitmapScans' ``` Tested with random query generator Reviewers: amartsinchyk, tnayak Reviewed By: tnayak Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D32651

…p Scans Summary: Original commit: 773869c / D32651 Pushing down filters prevents us from fetching unnecessary rows. In the bitmap scan case, if we can push down a filter at the Bitmap Index Scan level, it saves us from 1. retrieving the ybctid 2. using that ybctid in any bitmap operations (AND, OR) 3. retrieving the row corresponding to the ybctid. With this diff, both YB Bitmap Table Scans and Bitmap Index Scans are capable of pushing down filters. == YB Bitmap Table Scan pushdown == YB Bitmap Table Scans can push down any clause. The pushdown logic is similar to the recheck logic - it only pushes down a filter if it is necessary. For example, ```lang=sql /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=6.91..11.21 rows=10 width=12) -> BitmapOr (cost=6.91..6.91 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (b < 1000) (6 rows) ``` does not require a remote or local filter because it knows that the index scans will return only correct results. ``` /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000 AND c < 1; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=7.06..11.41 rows=10 width=12) Remote Filter: ((a < 1000) OR ((b < 1000) AND (c < 1))) -> BitmapOr (cost=7.06..7.06 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (b < 1000) (7 rows) ``` This requires a filter, and this filter can be pushed down. == Bitmap Index Scan pushdown == There are two cases where filters can be pushed down for bitmap index scans. === 1. If the filter clause is at the same nesting level as the index qual in the boolean operation tree === During planning, when the OR is broken up into separate indexes, the pushdown conditions are extracted for each branch of the OR. This allows us to get useful filters for pushdown, knowing that a particular index scan is concerned only by a portion of the whole condition. For example, in the index below, the first Bitmap Index Scan has an index condition `unique1 < 5` and a remote filter `unique1 % 2 = 0`. ```lang=sql /*+ BitmapScan(tenk1) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM tenk1 WHERE (unique1 < 5 AND unique1 % 2 = 0) OR unique2 < 3; QUERY PLAN ----------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual rows=6 loops=1) Remote Filter: (((unique1 < 5) AND ((unique1 % 2) = 0)) OR (unique2 < 3)) Storage Table Read Requests: 1 Storage Table Rows Scanned: 6 -> BitmapOr (actual rows=6 loops=1) -> Bitmap Index Scan on tenk1_unique1 (actual rows=3 loops=1) Index Cond: (unique1 < 5) Remote Index Filter: ((unique1 % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 5 -> Bitmap Index Scan on tenk1_unique2 (actual rows=3 loops=1) Index Cond: (unique2 < 3) Storage Index Read Requests: 1 Storage Index Rows Scanned: 3 Storage Read Requests: 3 Storage Rows Scanned: 14 Storage Write Requests: 0 Storage Flush Requests: 0 (18 rows) ``` When creating a bitmap scan plan, the planner breaks the OR into two parts. a: `(unique1 < 5 AND unique1 % 2 = 0)` b: `unique2 < 3` If it can find an index scan for each part, then a bitmap scan is a valid plan. (If there was a third clause on an unindexed field, c: `string1 = 'hi'`, then there would not be an index to answer each part of the OR so we cannot use a bitmap scan here). The default PG behaviour is to extract index clauses from each branch, so PG would identify `unique1 < 5` for a and `unique2 < 3` for b. With this diff, Yugabyte goes a step further. If a condition is not a valid index condition (e.g. `unique1 % 2 = 0`, then we test if it can be pushed down using the index columns. If yes, then we store it as a valid pushdown filter for a bitmap index scan. So we identify: | | index qual | filter qual | | a | `unique1 < 5` | `unique1 % 2 = 0` | | b | `unique2 < 3` | | Since we found an index path for each branch of the OR, we've found a valid bitmap scan path. === 2. All columns referenced by the top-level condition are contained in the index === In this case, we can push down the entire condition (or a portion of the top-level AND clause) to the index with the included columns. We cannot partially pushdown top-level ORs because this index doesn't know enough about the other branches, but that case was already handled above. For example, in the example below, the first Bitmap Index Scan pushes down the entire condition `(((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0)))` on `multi_c_a_idx` because the every column in the condition can be checked by this index. ```lang=sql /*+ BitmapScan(multi) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM multi WHERE (a < 50 AND a % 2 = 0) AND (c <= 100 AND c % 3 = 0); QUERY PLAN ------------------------------------------------------------------------------------------- YB Bitmap Table Scan on multi (actual time=3.655..3.666 rows=16 loops=1) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.014 ms Storage Table Rows Scanned: 16 -> BitmapAnd (actual time=2.492..2.492 rows=16 loops=1) -> Bitmap Index Scan on multi_c_a_idx (actual time=2.550..2.550 rows=16 loops=1) Index Cond: (c <= 100) Storage Index Filter: ((a < 50) AND ((a % 2) = 0) AND ((c % 3) = 0)) Storage Index Read Requests: 1 Storage Index Read Execution Time: 1.778 ms Storage Index Rows Scanned: 33 -> Bitmap Index Scan on multi_pkey (actual time=1.288..1.288 rows=24 loops=1) Index Cond: (a < 50) Storage Index Filter: ((a % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.164 ms Storage Table Rows Scanned: 49 Planning Time: 15.273 ms Execution Time: 4.362 ms Storage Read Requests: 2 Storage Read Execution Time: 2.942 ms Storage Rows Scanned: 82 Storage Write Requests: 0 Catalog Read Requests: 19 Catalog Read Execution Time: 31.753 ms Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 34.695 ms Peak Memory Usage: 119 kB (26 rows) ``` === 3. All columns referenced by the one of the top-level clauses (implicitly ANDed) are contained in the index === Consider the `tenk1` table: ```lang=sql \d tenk1 Table "public.tenk1" ... Indexes: "tenk1_hundred" lsm (hundred ASC) "tenk1_thous_tenthous" lsm (thousand ASC, tenthous ASC) "tenk1_unique1" lsm (unique1 ASC) "tenk1_unique2" lsm (unique2 ASC) ``` Then the query can push down the `tenthous % 2 = 0` condition to the `thousand` bitmap index scan. Even though it will still need to be validated at the Table Scan layer (to check the results from the `unique2` bitmap index scan, it is efficient to reduce the number of rows retrieved. ``` explain (analyze, costs off, dist) /*+ bitmapscan(tenk1) */ select * from tenk1 where thousand < 10 and unique2 < 1000 and tenthous % 2 = 0; QUERY PLAN -------------------------------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual time=17.917..17.929 rows=4 loops=1) Storage Filter: ((tenthous % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.170 ms Storage Table Rows Scanned: 4 -> BitmapAnd (actual time=16.422..16.422 rows=4 loops=1) -> Bitmap Index Scan on tenk1_unique2 (actual time=13.315..13.315 rows=1000 loops=1) Index Cond: (unique2 < 1000) Storage Index Read Requests: 1 Storage Index Read Execution Time: 10.981 ms Storage Index Rows Scanned: 1000 -> Bitmap Index Scan on tenk1_thous_tenthous (actual time=2.845..2.845 rows=50 loops=1) Index Cond: (thousand < 10) Storage Index Filter: ((tenthous % 2) = 0) Storage Index Read Requests: 1 Storage Index Read Execution Time: 2.407 ms Storage Index Rows Scanned: 100 Planning Time: 0.894 ms Execution Time: 18.702 ms Storage Read Requests: 3 Storage Read Execution Time: 14.559 ms Storage Rows Scanned: 1104 Storage Write Requests: 0 Catalog Read Requests: 0 Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 14.559 ms Peak Memory Usage: 180 kB (28 rows) ``` Jira: DB-10083 Test Plan: ``` ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressYbBitmapScans' ``` Tested with random query generator Reviewers: amartsinchyk, tnayak Reviewed By: tnayak Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34006

…ter-merge Merge YB master commit 773869c titled [#21141,21036] YSQL: Enable remote filters for Bitmap Scans and committed 2024-04-11T00:27:43+00:00 into YB pg15. Since there is only one commit being merged in from YB master since the last merge, I will refer to it simply as "YB master". YB pg15 initial merge refers to 55782d5. - executor/Makefile: - OBJS: YB master adds nodeYbBitmapIndexscan.o; upstream PG changes style. - execAmi.c: - includes: YB master adds nodeYbBitmapIndexscan.h include in same location upstream PG fa2cf164aaf91e074be653c28e035f65d80eb666 adds extensible.h include. YB pg15 initial merge creates a new Yugabyte includes section, so move this there. - nodeYbBitmapIndexscan.c: - ExecEndYbBitmapIndexScan: YB master adds this new file with ExecCloseScanRelation. As before in YB pg15 ad2fedc, upstream PG 9ddef36278a9f676c07d0b4d9f33fa22e48ce3b5 removes ExecCloseScanRelation, so get rid of the logic introduced by YB master. - nodeYbBitmapTablescan.c: - function declarations: YB master removes param currentRelation from CreateYbBitmapTableScanDesc. YB pg15 b728fbd never had that param to begin with. Ignore the incoming master change. - YbBitmapTableNext: YB master moves some lines lower; YB pg15 ad2fedc and b728fbd make adjustments to those lines. Tricky merge. - CreateYbBitmapTableScanDesc: major merge with YB pg15 b728fbd. Of particular note is - local variable "plan" is typed YbBitmapTableScan. - is_internal_scan changes from false to true. - rs_temp_snap and rs_cblock are not set since they are not part of TableScanDesc. - switch SO_TYPE_SEQSCAN to SO_TYPE_BITMAPSCAN. - ExecReScanYbBitmapTableScan: YB master expands a comment for why old scan is set to null and sets it to null. YB pg15 ad2fedc already set it to null. Bring in the comment from master since it seems useful, but besides that, stick with YB pg15's version. - indxpath.c: - includes: YB master adds two includes interleaved with PG includes, and this causes conflict with YB pg15 changes to the PG includes. Move those includes down to the Yugabyte includes section which was added by YB pg15 initial merge. - function declarations: YB master adds new parameter yb_bitmap_idx_pushdowns to four functions. YB pg15 changes indentation of those functions. Trivial merge. - build_index_paths - top comment: adjacent lines conflict between upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf and YB master. - create_index_path calls (x4): upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf removes indexclausecols parameter from create_index_path, conflicting with YB master which adds a parameter in the same location. - build_paths_for_OR: both YB master and 74dfe58a5927b22c744b29534e67bfdd203ac028 change signature of match_clauses_to_index. Merge for two calls. - match_restriction_clauses_to_index: (same as build_paths_for_OR) - match_join_clauses_to_index: (same as build_paths_for_OR) - match_eclass_clauses_to_index: (same as build_paths_for_OR) - match_clauses_to_index: (same as build_paths_for_OR) - match_clause_to_index: (same as build_paths_for_OR) - createplan.c: - function declarations: YB master adds parameters (mistakenly without YB prefix) to some PG functions. YB pg15 changes indentation. - create_bitmap_subplan: YB master adds indexpushdownquals setting after indexqual setting, and upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf moves the setting lower. Move the YB setting lower as well. Take the opportunity to detach it from PG code better. - planner.c: - plan_cluster_use_sort: YB master adds NIL argument to create_index_path_call. Adjust to the change in parameters by YB pg15. - setrefs.c: - set_plan_refs: - T_YbBitmapIndexScan: YB pg15 ad2fedc adds extra parameter to fix_scan_list, so do so accordingly on top of code added by YB master. Upstream PG 41efb8340877e8ffd0023bb6b2ef22ffd1ca014d adds extra parameter to fix_upper_expr, so add the extra argument like other places in the same file. - T_YbBitmapTableScan: YB master significantly changes this. YB pg15 ad2fedc adds extra parameter to fix_scan_list, so do so accordingly on top of master. - pathnode.c: - create_index_path: - top comment: adjacent lines conflict between upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf and YB master. - signature: (same) - body: same upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf removes indexquals and indexqualcols settings. YB master adds yb_bitmap_idx_pushdowns setting in the same location. - ruleutils.c: - set_deparse_planstate: upstream PG 6ef77cf46e81f45716ec981cb08781d426181378 changes "ps->plan" to "plan". YB master adds new code with "ps->plan". Change it to "plan". - find_param_referent: YB master removes trailing whitespace on code which no longer has trailing whitespace in YB pg15. Ignore the incoming YB master change. - execnodes.h: - BitmapIndexScanState: adjacent line conflict between YB master modifying biss_result line and YB pg15 modifying biss_ScanKeys line. - YbBitmapIndexScanState: change type of biss_ScanKeys and biss_ScanDesc according to how it was changed in BitmapIndexScanState by upstream PG 0944ec54de389b4b8a471ca1f40f1b9d81de1f30. - pathnodes.h: - IndexPath: upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf changes indexclauses and removes indexquals and indexqualcols. YB master adds yb_bitmap_idx_pushdowns in the same area. - pathnode.h: - create_index_path: (same as pathnode.c create_index_path signature) - pg_dml.cc: - PgDml::AppendTargetPB: YB pg15 5e6edbf removes has_system_targets_ and is_system logic. YB master adds logic to set has_regular_targets_ using is_system. Bring back is_system but not has_system_targets_ from 5e6edbf, then use it for this merge. - pggate.cc: - PgApiImpl::DmlHasRegularTargets: again, adjacent lines conflict with YB pg15 5e6edbf that removes PgApiImpl::DmlHasSystemTargets. - pggate.h: - DmlHasRegularTargets: (same) - ybc_pggate.cc: - YBCPgDmlHasRegularTargets: (same) - ybc_pggate.h: - YBCPgDmlHasRegularTargets: (same) - pg_expr.cc: - PgColumnRef::is_system: (same as pg_dml.cc) - pg_expr.h: - is_system: (same as pg_dml.cc) - pg15_tests/test_yb_bitmap_scans.sh: - Update diff line numbers since YB master changes content in the file. - Only check up to line 786 of the file since, after that, the output is quite wrong.

…2792304' into pg15 Summary: Merge YB master commit 773869c titled [#21141,21036] YSQL: Enable remote filters for Bitmap Scans and committed 2024-04-11T00:27:43+00:00 into YB pg15. Since there is only one commit being merged in from YB master since the last merge, I will refer to it simply as "YB master". YB pg15 initial merge refers to 55782d5. - executor/Makefile: - OBJS: YB master adds nodeYbBitmapIndexscan.o; upstream PG changes style. - execAmi.c: - includes: YB master adds nodeYbBitmapIndexscan.h include in same location upstream PG fa2cf164aaf91e074be653c28e035f65d80eb666 adds extensible.h include. YB pg15 initial merge creates a new Yugabyte includes section, so move this there. - nodeYbBitmapIndexscan.c: - ExecEndYbBitmapIndexScan: YB master adds this new file with ExecCloseScanRelation. As before in YB pg15 ad2fedc, upstream PG 9ddef36278a9f676c07d0b4d9f33fa22e48ce3b5 removes ExecCloseScanRelation, so get rid of the logic introduced by YB master. - nodeYbBitmapTablescan.c: - function declarations: YB master removes param currentRelation from CreateYbBitmapTableScanDesc. YB pg15 b728fbd never had that param to begin with. Ignore the incoming master change. - YbBitmapTableNext: YB master moves some lines lower; YB pg15 ad2fedc and b728fbd make adjustments to those lines. Tricky merge. - CreateYbBitmapTableScanDesc: major merge with YB pg15 b728fbd. Of particular note is - local variable "plan" is typed YbBitmapTableScan. - is_internal_scan changes from false to true. - rs_temp_snap and rs_cblock are not set since they are not part of TableScanDesc. - switch SO_TYPE_SEQSCAN to SO_TYPE_BITMAPSCAN. - ExecReScanYbBitmapTableScan: YB master expands a comment for why old scan is set to null and sets it to null. YB pg15 ad2fedc already set it to null. Bring in the comment from master since it seems useful, but besides that, stick with YB pg15's version. - indxpath.c: - includes: YB master adds two includes interleaved with PG includes, and this causes conflict with YB pg15 changes to the PG includes. Move those includes down to the Yugabyte includes section which was added by YB pg15 initial merge. - function declarations: YB master adds new parameter yb_bitmap_idx_pushdowns to four functions. YB pg15 changes indentation of those functions. Trivial merge. - build_index_paths - top comment: adjacent lines conflict between upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf and YB master. - create_index_path calls (x4): upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf removes indexclausecols parameter from create_index_path, conflicting with YB master which adds a parameter in the same location. - build_paths_for_OR: both YB master and 74dfe58a5927b22c744b29534e67bfdd203ac028 change signature of match_clauses_to_index. Merge for two calls. - match_restriction_clauses_to_index: (same as build_paths_for_OR) - match_join_clauses_to_index: (same as build_paths_for_OR) - match_eclass_clauses_to_index: (same as build_paths_for_OR) - match_clauses_to_index: (same as build_paths_for_OR) - match_clause_to_index: (same as build_paths_for_OR) - createplan.c: - function declarations: YB master adds parameters (mistakenly without YB prefix) to some PG functions. YB pg15 changes indentation. - create_bitmap_subplan: YB master adds indexpushdownquals setting after indexqual setting, and upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf moves the setting lower. Move the YB setting lower as well. Take the opportunity to detach it from PG code better. - planner.c: - plan_cluster_use_sort: YB master adds NIL argument to create_index_path_call. Adjust to the change in parameters by YB pg15. - setrefs.c: - set_plan_refs: - T_YbBitmapIndexScan: YB pg15 ad2fedc adds extra parameter to fix_scan_list, so do so accordingly on top of code added by YB master. Upstream PG 41efb8340877e8ffd0023bb6b2ef22ffd1ca014d adds extra parameter to fix_upper_expr, so add the extra argument like other places in the same file. - T_YbBitmapTableScan: YB master significantly changes this. YB pg15 ad2fedc adds extra parameter to fix_scan_list, so do so accordingly on top of master. - pathnode.c: - create_index_path: - top comment: adjacent lines conflict between upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf and YB master. - signature: (same) - body: same upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf removes indexquals and indexqualcols settings. YB master adds yb_bitmap_idx_pushdowns setting in the same location. - ruleutils.c: - set_deparse_planstate: upstream PG 6ef77cf46e81f45716ec981cb08781d426181378 changes "ps->plan" to "plan". YB master adds new code with "ps->plan". Change it to "plan". - find_param_referent: YB master removes trailing whitespace on code which no longer has trailing whitespace in YB pg15. Ignore the incoming YB master change. - execnodes.h: - BitmapIndexScanState: adjacent line conflict between YB master modifying biss_result line and YB pg15 modifying biss_ScanKeys line. - YbBitmapIndexScanState: change type of biss_ScanKeys and biss_ScanDesc according to how it was changed in BitmapIndexScanState by upstream PG 0944ec54de389b4b8a471ca1f40f1b9d81de1f30. - pathnodes.h: - IndexPath: upstream PG 1a8d5afb0dfc5d0dcc6eda0656a34cb1f0cf0bdf changes indexclauses and removes indexquals and indexqualcols. YB master adds yb_bitmap_idx_pushdowns in the same area. - pathnode.h: - create_index_path: (same as pathnode.c create_index_path signature) - pg_dml.cc: - PgDml::AppendTargetPB: YB pg15 5e6edbf removes has_system_targets_ and is_system logic. YB master adds logic to set has_regular_targets_ using is_system. Bring back is_system but not has_system_targets_ from 5e6edbf, then use it for this merge. - pggate.cc: - PgApiImpl::DmlHasRegularTargets: again, adjacent lines conflict with YB pg15 5e6edbf that removes PgApiImpl::DmlHasSystemTargets. - pggate.h: - DmlHasRegularTargets: (same) - ybc_pggate.cc: - YBCPgDmlHasRegularTargets: (same) - ybc_pggate.h: - YBCPgDmlHasRegularTargets: (same) - pg_expr.cc: - PgColumnRef::is_system: (same as pg_dml.cc) - pg_expr.h: - is_system: (same as pg_dml.cc) - pg15_tests/test_yb_bitmap_scans.sh: - Update diff line numbers since YB master changes content in the file. - Only check up to line 786 of the file since, after that, the output is quite wrong. Test Plan: Jenkins: rebase: pg15 Reviewers: telgersma Reviewed By: telgersma Subscribers: kfranz, fizaa, tfoucher, yql Differential Revision: https://phorge.dev.yugabyte.com/D35112

timothy-e added area/ysql Yugabyte SQL (YSQL) status/awaiting-triage Issue awaiting triage labels Feb 22, 2024

timothy-e self-assigned this Feb 22, 2024

yugabyte-ci added kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue labels Feb 22, 2024

sushantrmishra removed the status/awaiting-triage Issue awaiting triage label Feb 22, 2024

yugabyte-ci assigned sushantrmishra and unassigned timothy-e Feb 22, 2024

timothy-e assigned timothy-e and unassigned sushantrmishra Feb 27, 2024

yugabyte-ci closed this as completed Apr 11, 2024

timothy-e mentioned this issue Jun 3, 2024

[YSQL] Add Bitmap Scan support in YugabyteDB #22653

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YSQL] Allow remote filters in YB Bitmap Scans #21141

[YSQL] Allow remote filters in YB Bitmap Scans #21141

timothy-e commented Feb 22, 2024 •

edited

Loading

[YSQL] Allow remote filters in YB Bitmap Scans #21141

[YSQL] Allow remote filters in YB Bitmap Scans #21141

Comments

timothy-e commented Feb 22, 2024 • edited Loading

Description

Issue Type

Warning: Please confirm that this issue does not contain any sensitive information

timothy-e commented Feb 22, 2024 •

edited

Loading