diff --git a/in-memory-21c/aim-high/aim-high.md b/in-memory-21c/aim-high/aim-high.md index 660ef7c09..92b5e0bc9 100644 --- a/in-memory-21c/aim-high/aim-high.md +++ b/in-memory-21c/aim-high/aim-high.md @@ -1,4 +1,4 @@ -# Automatic In-Memory High +# Automatic In-Memory Level High ## Introduction Watch the video below to get an overview of Automatic In-Memory: @@ -31,9 +31,7 @@ In Oracle Database 18c a feature called Automatic In-Memory (AIM) was added. The This Lab will explore the new AIM level HIGH and how it works. A new schema will be used, the AIM schema with small, medium and large tables. This will make it easier to show how AIM works as the column store experiences "memory pressure" (i.e. gets full). The LINEORDER table in the SSB schema will be used to help "fill up" the IM column store and then the AIM tables will be used to show how AIM can manage the total number of objects for maximum benefit. -## Task 1: Verify Directory Definitions - -In this Lab we will be populating external data from a local directory and we will need to define a database directory to use in our external table definitions to point the database to our external data. +## Task 1: AIM Level High Let's switch to the aim-high folder and log back in to the PDB: @@ -320,7 +318,7 @@ SQL> SQL> ``` - Verify that all of the LINEORDER partitions are populated before continuing on with the lab. + Verify that all of the LINEORDER partitions are populated before continuing on with the lab. You may have to re-run this script to verify that the partitions are populated since it does take a little bit of time to fully populate the partitions. 5. Let's re-check the inmemory status of the objects in the SSB and AIM schemas: @@ -807,7 +805,7 @@ SQL> SQL> ``` -11. Let's take a look at the Heat Map statistics for the segments. +11. Let's take a look at the Heat Map statistics for the segments. Although Heat Map is not used directly by AIM, and does not have to be enabled for AIM to work, it does give us an easy way to look at the usage statistics that AIM does base its decisions on. Run the script *11\_hm\_stats.sql* @@ -889,6 +887,8 @@ SQL> SQL> ``` + Note that your values may be different than what is shown above. The values shown will be based on the usage that has occurred in your database. + 12. Now let's see if we can figure out what has happened with the AIM processing. First we will look at the tasks that are running as part of AIM. Run the script *12\_aim\_tasks.sql* @@ -932,7 +932,7 @@ SQL> SQL> ``` - Make note of the last task_id. We will use this as input in the next step. Also note that the tasks are being run approximately every 2 minutes. As was described in Lab13 on AIM level LOW and MEDIUM, AIM tasks will continue to be scheduled during each IMCO cycle when under memory pressure. Again, this means that it may take a couple of cycles before an object is populated by AIM in the IM column store. + Make note of the last task_id. We will use this as input in the next step. Also note that the tasks are being run approximately every 2 minutes. AIM tasks will be scheduled during each IMCO cycle, which is approximately every 2 minutes, when the IM column store is under memory pressure. This means that it may take a couple of cycles before an object is populated by AIM in the IM column store. 13. Now let's look at the AIM task details, or what actually happened. @@ -998,7 +998,7 @@ SQL> SQL> ``` - As a reminder, take a look at the OBJECT_NAME and the ACTION. Now that the IM column store is under memory pressure AIM has taken over control of population and there is a lot going on. Based on usage statistics AIM will populate the objects that will result in the most benefit to queries being run. You may want to take a look at some of the other task details to get a better picture of what has happened. Also note that now that AIM is controlling population the PRIORITY level will be ignored and AIM will decide which objects to populate and which to evict. + Take a look at the OBJECT_NAME and the ACTION. Now that the IM column store is under memory pressure AIM has taken over control of population and there is a lot going on. Based on usage statistics AIM will populate the objects that will result in the most benefit to queries being run. You may want to take a look at some of the other task details to get a better picture of what has happened. Also note that now that AIM is controlling population the PRIORITY level will be ignored and AIM will decide which objects to populate and which to evict. 14. Now let's turn AIM off and see what happens. @@ -1141,7 +1141,7 @@ SQL> SQL> ``` - Notice that all of the objects that were enabled for inmemory when the AIM level was set to high have now been disabled. However, the LINEORDER partitions that we manually enabled are still enabled. + Notice that all of the objects that were enabled for inmemory when the AIM level was set to HIGH have now been disabled. However, the LINEORDER partitions that we manually enabled are still enabled. 16. What has happened to the populated segments in the IM column store? @@ -1284,7 +1284,7 @@ SQL> ## Conclusion -This lab demonstrated how then new INMEMORY\_AUTOMATIC\_LEVEL = HIGH feature works and how AIM level high can enable the automatic management of the contents of IM column store. This means no more having to try and figure out which objects would get the most benefit from being populated. Now the database will do it for you. +This lab demonstrated how the new INMEMORY\_AUTOMATIC\_LEVEL = HIGH feature works and how AIM level high can enable the automatic management of the contents of IM column store. This means no more having to try and figure out which objects would get the most benefit from being populated. Now the database will do it for you. You may now **proceed to the next lab**. diff --git a/in-memory-21c/aim-high/files/01_aim_attributes.sql b/in-memory-21c/aim-high/files/01_aim_attributes.sql new file mode 100644 index 000000000..8299125d6 --- /dev/null +++ b/in-memory-21c/aim-high/files/01_aim_attributes.sql @@ -0,0 +1,30 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set tab off + +column owner format a10; +column table_name format a20; +column partition_name format a15; +column inmemory format a10; +column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; +column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; +column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; +set echo on + +-- Show table attributes + +select owner, table_name, NULL as partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tables +where owner in ('AIM','SSB') +UNION ALL +select table_owner as owner, table_name, partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tab_partitions +where table_owner in ('AIM','SSB') +order by owner, table_name, partition_name; + +set echo off + diff --git a/in-memory-21c/aim-high/files/02_disable_tables.sql b/in-memory-21c/aim-high/files/02_disable_tables.sql new file mode 100644 index 000000000..d0bd9b465 --- /dev/null +++ b/in-memory-21c/aim-high/files/02_disable_tables.sql @@ -0,0 +1,36 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 + +set serveroutput on; +declare + v_ddl varchar2(1000); +begin + for tab_cursor in ( + select owner, table_name + from dba_tables + where owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||tab_cursor.owner||'.'||tab_cursor.table_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; + -- + for part_cursor in ( + select table_owner, table_name, partition_name + from dba_tab_partitions + where table_owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||part_cursor.table_owner||'.'||part_cursor.table_name|| + ' modify partition '||part_cursor.partition_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; +end; +/ + diff --git a/in-memory-21c/aim-high/files/03_pop_ssb_tables.sql b/in-memory-21c/aim-high/files/03_pop_ssb_tables.sql new file mode 100644 index 000000000..858d1256a --- /dev/null +++ b/in-memory-21c/aim-high/files/03_pop_ssb_tables.sql @@ -0,0 +1,12 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set echo on + +alter table ssb.lineorder inmemory; + +exec dbms_inmemory.populate('SSB','LINEORDER'); + +set echo off + diff --git a/in-memory-21c/aim-high/files/04_im_populated.sql b/in-memory-21c/aim-high/files/04_im_populated.sql new file mode 100644 index 000000000..9da3776b5 --- /dev/null +++ b/in-memory-21c/aim-high/files/04_im_populated.sql @@ -0,0 +1,28 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 +set tab off + +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +where owner not in ('AUDSYS','SYS') +order by owner, segment_name, partition_name; + +select * from v$inmemory_area; + +set echo off diff --git a/in-memory-21c/aim-high/files/05_aim_attributes.sql b/in-memory-21c/aim-high/files/05_aim_attributes.sql new file mode 100644 index 000000000..8299125d6 --- /dev/null +++ b/in-memory-21c/aim-high/files/05_aim_attributes.sql @@ -0,0 +1,30 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set tab off + +column owner format a10; +column table_name format a20; +column partition_name format a15; +column inmemory format a10; +column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; +column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; +column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; +set echo on + +-- Show table attributes + +select owner, table_name, NULL as partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tables +where owner in ('AIM','SSB') +UNION ALL +select table_owner as owner, table_name, partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tab_partitions +where table_owner in ('AIM','SSB') +order by owner, table_name, partition_name; + +set echo off + diff --git a/in-memory-21c/aim-high/files/06_aim_high.sql b/in-memory-21c/aim-high/files/06_aim_high.sql new file mode 100644 index 000000000..890ac1829 --- /dev/null +++ b/in-memory-21c/aim-high/files/06_aim_high.sql @@ -0,0 +1,12 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 + +show parameters inmemory_automatic_level + +alter system set inmemory_automatic_level=high; + +show parameters inmemory_automatic_level + + diff --git a/in-memory-21c/aim-high/files/07_aim_attributes.sql b/in-memory-21c/aim-high/files/07_aim_attributes.sql new file mode 100644 index 000000000..8299125d6 --- /dev/null +++ b/in-memory-21c/aim-high/files/07_aim_attributes.sql @@ -0,0 +1,30 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set tab off + +column owner format a10; +column table_name format a20; +column partition_name format a15; +column inmemory format a10; +column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; +column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; +column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; +set echo on + +-- Show table attributes + +select owner, table_name, NULL as partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tables +where owner in ('AIM','SSB') +UNION ALL +select table_owner as owner, table_name, partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tab_partitions +where table_owner in ('AIM','SSB') +order by owner, table_name, partition_name; + +set echo off + diff --git a/in-memory-21c/aim-high/files/08_im_populated.sql b/in-memory-21c/aim-high/files/08_im_populated.sql new file mode 100644 index 000000000..9da3776b5 --- /dev/null +++ b/in-memory-21c/aim-high/files/08_im_populated.sql @@ -0,0 +1,28 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 +set tab off + +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +where owner not in ('AUDSYS','SYS') +order by owner, segment_name, partition_name; + +select * from v$inmemory_area; + +set echo off diff --git a/in-memory-21c/aim-high/files/09_pop_aim_tables.sql b/in-memory-21c/aim-high/files/09_pop_aim_tables.sql new file mode 100644 index 000000000..d83c1ef18 --- /dev/null +++ b/in-memory-21c/aim-high/files/09_pop_aim_tables.sql @@ -0,0 +1,26 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set echo on + +select count(*) from lrgtab1; +select count(*) from lrgtab1; + +select count(*) from lrgtab2; +select count(*) from lrgtab2; +select count(*) from lrgtab2; + +select count(*) from lrgtab3; +select count(*) from medtab1; +select count(*) from medtab1; +select count(*) from medtab1; +select count(*) from medtab2; +select count(*) from medtab2; +select count(*) from medtab2; + +select count(*) from lrgtab4; +select count(*) from lrgtab4; + +set echo off + diff --git a/in-memory-21c/aim-high/files/10_im_populated.sql b/in-memory-21c/aim-high/files/10_im_populated.sql new file mode 100644 index 000000000..9da3776b5 --- /dev/null +++ b/in-memory-21c/aim-high/files/10_im_populated.sql @@ -0,0 +1,28 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 +set tab off + +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +where owner not in ('AUDSYS','SYS') +order by owner, segment_name, partition_name; + +select * from v$inmemory_area; + +set echo off diff --git a/in-memory-21c/aim-high/files/11_hm_stats.sql b/in-memory-21c/aim-high/files/11_hm_stats.sql new file mode 100644 index 000000000..3223cc391 --- /dev/null +++ b/in-memory-21c/aim-high/files/11_hm_stats.sql @@ -0,0 +1,47 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 +set echo off +set tab off + +-- Heat Map query +-- (Flush Heat Map data: exec dbms_ilm.flush_all_segments) +-- + +col owner format a10; +col object_name format a20; +col subobject_name format a15; +col track_time format a16; +col segment_write heading 'SEG|WRITE' format a10; +col segment_read heading 'SEG|READ' format a10; +col full_scan heading 'FULL|SCAN' format a10; +col lookup_scan heading 'LOOKUP|SCAN' format a10; +col n_fts heading 'NUM FULL|SCAN' format 99999999; +col n_lookup heading 'NUM LOOKUP|SCAN' format 99999999; +col n_write heading 'NUM SEG|WRITE' format 99999999; +-- +select + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME, + to_char(TRACK_TIME,'MM/DD/YYYY HH24:MI') track_time, + SEGMENT_WRITE, + SEGMENT_READ, + FULL_SCAN, + LOOKUP_SCAN, + N_FTS, + N_LOOKUP, + N_WRITE +from + sys."_SYS_HEAT_MAP_SEG_HISTOGRAM" h, + dba_objects o +where + o.object_id = h.obj# + and track_time >= sysdate-1 +order by + track_time, + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME; + diff --git a/in-memory-21c/aim-high/files/12_aimtasks.sql b/in-memory-21c/aim-high/files/12_aimtasks.sql new file mode 100644 index 000000000..b8c77a327 --- /dev/null +++ b/in-memory-21c/aim-high/files/12_aimtasks.sql @@ -0,0 +1,12 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set tab off + +select task_id, to_char(creation_time,'DD-MON-YY hh24:mi:ss') as create_time, state +from dba_inmemory_aimtasks +where creation_time > sysdate -1 +order by task_id +/ + diff --git a/in-memory-21c/aim-high/files/13_aimtaskdetails.sql b/in-memory-21c/aim-high/files/13_aimtaskdetails.sql new file mode 100644 index 000000000..316c7ee07 --- /dev/null +++ b/in-memory-21c/aim-high/files/13_aimtaskdetails.sql @@ -0,0 +1,12 @@ +@../aim_login.sql + +set lines 150 +set pages 9999 + +col object_owner format a15; +col object_name format a30; +col subobject_name format a30; +select * from dba_inmemory_aimtaskdetails where task_id = &1 +and object_owner not in ('SYS','AUDSYS') +order by object_owner, object_name, subobject_name, action; + diff --git a/in-memory-21c/aim-high/files/14_aim_off.sql b/in-memory-21c/aim-high/files/14_aim_off.sql new file mode 100644 index 000000000..718f7fd86 --- /dev/null +++ b/in-memory-21c/aim-high/files/14_aim_off.sql @@ -0,0 +1,12 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 + +show parameters inmemory_automatic_level + +alter system set inmemory_automatic_level=off; + +show parameters inmemory_automatic_level + + diff --git a/in-memory-21c/aim-high/files/15_aim_attributes.sql b/in-memory-21c/aim-high/files/15_aim_attributes.sql new file mode 100644 index 000000000..8299125d6 --- /dev/null +++ b/in-memory-21c/aim-high/files/15_aim_attributes.sql @@ -0,0 +1,30 @@ +@../aim_login.sql + +set pages 999 +set lines 200 +set tab off + +column owner format a10; +column table_name format a20; +column partition_name format a15; +column inmemory format a10; +column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; +column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; +column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; +set echo on + +-- Show table attributes + +select owner, table_name, NULL as partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tables +where owner in ('AIM','SSB') +UNION ALL +select table_owner as owner, table_name, partition_name, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from dba_tab_partitions +where table_owner in ('AIM','SSB') +order by owner, table_name, partition_name; + +set echo off + diff --git a/in-memory-21c/aim-high/files/16_im_populated.sql b/in-memory-21c/aim-high/files/16_im_populated.sql new file mode 100644 index 000000000..9da3776b5 --- /dev/null +++ b/in-memory-21c/aim-high/files/16_im_populated.sql @@ -0,0 +1,28 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 +set tab off + +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +where owner not in ('AUDSYS','SYS') +order by owner, segment_name, partition_name; + +select * from v$inmemory_area; + +set echo off diff --git a/in-memory-21c/aim-high/files/17_aim-high_cleanup.sql b/in-memory-21c/aim-high/files/17_aim-high_cleanup.sql new file mode 100644 index 000000000..f9c82cf30 --- /dev/null +++ b/in-memory-21c/aim-high/files/17_aim-high_cleanup.sql @@ -0,0 +1,39 @@ +@../imlogin.sql + +set pages 999 +set lines 200 + +alter system set inmemory_automatic_level=off; +alter table lineorder no inmemory; + +set serveroutput on; +declare + v_ddl varchar2(1000); +begin + for tab_cursor in ( + select owner, table_name + from dba_tables + where owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||tab_cursor.owner||'.'||tab_cursor.table_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; + -- + for part_cursor in ( + select table_owner, table_name, partition_name + from dba_tab_partitions + where table_owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||part_cursor.table_owner||'.'||part_cursor.table_name|| + ' modify partition '||part_cursor.partition_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; +end; +/ + diff --git a/in-memory-21c/joins-aggregations/files/01_join_im.sql b/in-memory-21c/joins-aggregations/files/01_join_im.sql new file mode 100644 index 000000000..338957604 --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/01_join_im.sql @@ -0,0 +1,28 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set numwidth 16 +set timing on +set echo on + +-- Demonstrate an in-memory join query + +Select sum(lo_extendedprice * lo_discount) revenue +From LINEORDER l, DATE_DIM d +Where l.lo_orderdate = d.d_datekey +And l.lo_discount between 2 and 3 +And l.lo_quantity < 24 +And d.d_date='December 24, 1996'; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/joins-aggregations/files/02_join_buffer.sql b/in-memory-21c/joins-aggregations/files/02_join_buffer.sql new file mode 100644 index 000000000..cfa4c0296 --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/02_join_buffer.sql @@ -0,0 +1,29 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set numwidth 16 +set timing on +set echo on + +-- Buffer Cache query with the column store disabled using a NO_INMEMORY hint +-- + +SELECT /*+ NO_INMEMORY NO_VECTOR_TRANSFORM */ + SUM(lo_extendedprice * lo_discount) revenue +FROM lineorder l, + date_dim d +WHERE l.lo_orderdate = d.d_datekey +AND d.d_date='December 24, 1996'; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/joins-aggregations/files/03_3join_im.sql b/in-memory-21c/joins-aggregations/files/03_3join_im.sql new file mode 100644 index 000000000..f29a54f32 --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/03_3join_im.sql @@ -0,0 +1,36 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set numwidth 16 + +set timing on +set echo on + +-- Demonstrate an in-memory join query + +SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, p.p_brand1,SUM(lo_revenue) rev +FROM lineorder l, + date_dim d, + part p, + supplier s +WHERE l.lo_orderdate = d.d_datekey +AND l.lo_partkey = p.p_partkey +AND l.lo_suppkey = s.s_suppkey +AND p.p_category = 'MFGR#12' +AND s.s_region = 'AMERICA' +AND d.d_year = 1997 +GROUP BY d.d_year,p.p_brand1; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/joins-aggregations/files/04_3join_buffer.sql b/in-memory-21c/joins-aggregations/files/04_3join_buffer.sql new file mode 100644 index 000000000..e092fc760 --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/04_3join_buffer.sql @@ -0,0 +1,38 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set numwidth 16 + +set timing on +set echo on + +alter session set inmemory_query = disable; + +-- Demonstrate a buffer cache three way join query + +SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, p.p_brand1,SUM(lo_revenue) rev +FROM lineorder l, + date_dim d, + part p, + supplier s +WHERE l.lo_orderdate = d.d_datekey +AND l.lo_partkey = p.p_partkey +AND l.lo_suppkey = s.s_suppkey +AND p.p_category = 'MFGR#12' +AND s.s_region = 'AMERICA' +AND d.d_year = 1997 +GROUP BY d.d_year,p.p_brand1; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/joins-aggregations/files/05_join_nl_im.sql b/in-memory-21c/joins-aggregations/files/05_join_nl_im.sql new file mode 100644 index 000000000..61270f1bb --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/05_join_nl_im.sql @@ -0,0 +1,33 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set echo on + +-- Show the use of in-memory with a Nested Loops join + +-- Enable the use of invisible indexes + +alter session set optimizer_use_invisible_indexes=true; + +-- Execute query + +set timing on + +select /*+ NO_VECTOR_TRANSFORM INDEX(l, lineorder_i1) */ + sum(lo_extendedprice * lo_discount) revenue +from LINEORDER l, DATE_DIM d +where l.lo_orderdate = d.d_datekey +and d.d_date='December 24, 1996'; + +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +alter session set optimizer_use_invisible_indexes=false; + +set echo off + diff --git a/in-memory-21c/joins-aggregations/files/06_vgb_im.sql b/in-memory-21c/joins-aggregations/files/06_vgb_im.sql new file mode 100644 index 000000000..37398906e --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/06_vgb_im.sql @@ -0,0 +1,29 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on +set numwidth 20 + +-- In-Memory query with In-Memory Aggregation enabled + +select d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit +from LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C +where l.lo_orderdate = d.d_datekey +and l.lo_partkey = p.p_partkey +and l.lo_suppkey = s.s_suppkey +and l.lo_custkey = c.c_custkey +and s.s_region = 'AMERICA' +and c.c_region = 'AMERICA' +group by d.d_year, c.c_nation +order by d.d_year, c.c_nation; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/joins-aggregations/files/07_novgb_im.sql b/in-memory-21c/joins-aggregations/files/07_novgb_im.sql new file mode 100644 index 000000000..4c4d5d25c --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/07_novgb_im.sql @@ -0,0 +1,29 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- In-Memory query with In-Memory Aggregation disabled + +SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + Where l.lo_orderdate = d.d_datekey + And l.lo_partkey = p.p_partkey + And l.lo_suppkey = s.s_suppkey + And l.lo_custkey = c.c_custkey + And s.s_region = 'AMERICA' + And c.c_region = 'AMERICA' + Group by d.d_year, c.c_nation + Order by d.d_year, c.c_nation; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/joins-aggregations/files/08_vgb_buffer.sql b/in-memory-21c/joins-aggregations/files/08_vgb_buffer.sql new file mode 100644 index 000000000..da99ebe5e --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/08_vgb_buffer.sql @@ -0,0 +1,31 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +alter session set inmemory_query = disable; + +-- Query with In-Memory Aggregation enabled and in-memory disabled + +SELECT + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + Where l.lo_orderdate = d.d_datekey + And l.lo_partkey = p.p_partkey + And l.lo_suppkey = s.s_suppkey + And l.lo_custkey = c.c_custkey + And s.s_region = 'AMERICA' + And c.c_region = 'AMERICA' + Group by d.d_year, c.c_nation + Order by d.d_year, c.c_nation; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/joins-aggregations/files/09_novgb_buffer.sql b/in-memory-21c/joins-aggregations/files/09_novgb_buffer.sql new file mode 100644 index 000000000..215458a1b --- /dev/null +++ b/in-memory-21c/joins-aggregations/files/09_novgb_buffer.sql @@ -0,0 +1,31 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +alter session set inmemory_query = disable; + +-- Query with In-Memory Aggregation enabled and in-memory disabled + +SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + Where l.lo_orderdate = d.d_datekey + And l.lo_partkey = p.p_partkey + And l.lo_suppkey = s.s_suppkey + And l.lo_custkey = c.c_custkey + And s.s_region = 'AMERICA' + And c.c_region = 'AMERICA' + Group by d.d_year, c.c_nation + Order by d.d_year, c.c_nation; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/joins-aggregations/joins-aggregations.md b/in-memory-21c/joins-aggregations/joins-aggregations.md index 8cba949d9..38ff02d4b 100644 --- a/in-memory-21c/joins-aggregations/joins-aggregations.md +++ b/in-memory-21c/joins-aggregations/joins-aggregations.md @@ -1,9 +1,6 @@ # In-Memory Joins and Aggregations ## Introduction -[//]: # ( Watch the video below to get an overview of joins using Database In-Memory. ) - -[//]: # ( [YouTube video](youtube:y3tQeVGuo6g) ) Watch the video below for a quick walk-through of the In-memory Joins and Aggregations lab: @@ -186,6 +183,8 @@ SQL> Database In-Memory has no problem executing a query with a join, and in fact can optimize hash joins by being able to take advantage of Bloom filters. It’s easy to identify Bloom filters in the execution plan. They will appear in two places, at creation time (i.e. JOIN FILTER CREATE) and again when they are applied (i.e. JOIN FILTER USE). Look at Id 3 and Id 6 in the plan above. You can also see what join condition was used to build the Bloom filter by looking at the predicate information under the plan. + You may also note that there is another use of a Bloom filter at line Id 4. This is not a Database In-Memory feature but simply Oracle Database optimizing access to the partitions that make up the LINEORDER table. A takeaway here is that the use of Database In-Memory does not prevent the use of the other Oracle Database features. + 2. Let's run the query using the buffer cache. Run the script *02\_join\_buffer.sql* @@ -762,7 +761,7 @@ SQL> SQL> ``` - Notice that we have told the optimizer that it can use invisible indexes and we just happen to have an index that can be used on the LINEORDER table. This results in the optimizer choosing to perform a nested loops join by first accessing the DATE\_DIM table in-memory and then accessing the LINEORDER table through an index. The optimizer chooses this join, rather than a hash join, based on cost. This is another big advantage with Database In-Memory, the ability of the optimizer to choose the lowest cost methods to run queries with or without accessing object in-memory. + Notice that we have told the optimizer that it can use invisible indexes and we have added an index hint that can be used on the LINEORDER table. This results in the optimizer choosing to perform a nested loops join by first accessing the DATE\_DIM table in-memory and then accessing the LINEORDER table through an index. In this example, the cost is lower to access the LINEORDER table in the IM column store, but we wanted to show you that it is possible for the optimizer to choose different join types when using in-memory. If you're adventerous you can edit the script and remove the index hint. Don't worry, this is your own environment so it won't affect anyone else. If you then run the query again you can compare the cost to accessing the data through the index with a nested loops join. The thing to remember is the ability of the optimizer to choose the lowest cost methods to run queries with or without accessing object(s) in-memory. 6. Up until this point we have been focused on joins and how the IM column store can execute them incredibly efficiently. Let’s now turn our attention to more OLAP style “What If” queries. In this case our query examines the yearly profits from a specific region and manufacturer over our complete data set. @@ -1093,9 +1092,9 @@ SQL> SQL> ``` - Notice how much slower this second query ran even though it still ran in-memory, and even took advantage of Bloom filters. This is why we say that you can expect at least a 3-8x performance improvement with In-Memory Aggregation. + Notice how much slower this second query ran even though it still ran in-memory, and even took advantage of Bloom filters. In this Lab our tables are pretty small, but as the tables grow in size the performance difference is typically much greater. This is why we say that you can normally expect at least a 3-8x performance improvement with In-Memory Aggregation. -8. As we mentioned earlier, with Database In-Memory enabled the optimizer can even take advantage of vector transformation when the tables are not in-memory. To see this in action execute the same query against the buffer cache. +8. With Database In-Memory enabled the optimizer can even take advantage of vector transformation when the tables are not in-memory. To see this in action execute the same query against the buffer cache. Run the script *08\_vgb\_buffer.sql* @@ -1447,11 +1446,11 @@ SQL> ## Conclusion -This lab saw our performance comparison expanded to queries with both joins and aggregations. You had an opportunity to see just how efficiently a hash join, that is automatically converted to a Bloom filter, can be executed in the IM column store. +This lab saw our performance comparison expanded to queries with both joins and aggregations. You had an opportunity to see just how efficiently an in-memory hash join with Bloom filters can be executed in the IM column store. You also got to see just how sophisticated the Oracle Optimizer has become over the last 30 plus years, when it used a combination of complex query transformations to find the optimal execution plan for a star query. -Oracle Database adds In-Memory database functionality to existing databases, and transparently accelerates analytics by orders of magnitude while simultaneously speeding up mixed-workload OLTP. With Oracle Database In-Memory, users get immediate answers to business questions that previously took hours. +Oracle Database adds in-memory database functionality to existing databases, and transparently accelerates analytics by orders of magnitude while simultaneously speeding up mixed-workload OLTP. With Oracle Database In-Memory, users get immediate answers to business questions that previously took hours or days to run. You may now **proceed to the next lab**. diff --git a/in-memory-21c/json/files/01_disable_tables.sql b/in-memory-21c/json/files/01_disable_tables.sql new file mode 100644 index 000000000..b00c16bdc --- /dev/null +++ b/in-memory-21c/json/files/01_disable_tables.sql @@ -0,0 +1,37 @@ +@../aim_login.sql + +set pages 9999 +set lines 150 + +set serveroutput on; +declare + v_ddl varchar2(1000); +begin + for tab_cursor in ( + select owner, table_name + from dba_tables + where owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||tab_cursor.owner||'.'||tab_cursor.table_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; + -- + for part_cursor in ( + select table_owner, table_name, partition_name + from dba_tab_partitions + where table_owner not in ('AUDSYS','SYS') + and inmemory = 'ENABLED' + ) + loop + v_ddl := 'alter table '||part_cursor.table_owner||'.'||part_cursor.table_name|| + ' modify partition '||part_cursor.partition_name||' no inmemory'; + dbms_output.put_line(v_ddl); + execute immediate v_ddl; + end loop; +end; +/ +-- exec dbms_ilm_admin.CLEAR_HEAT_MAP_ALL; + diff --git a/in-memory-21c/json/files/02_json_prereq.sql b/in-memory-21c/json/files/02_json_prereq.sql new file mode 100644 index 000000000..b6dec18c7 --- /dev/null +++ b/in-memory-21c/json/files/02_json_prereq.sql @@ -0,0 +1,15 @@ +@../imlogin.sql + +set pages 9999 +set lines 200 + +-- Shows the SGA init.ora parameters + +show parameter compatible + +show parameter max_string_size + +show parameter inmemory_expressions_usage + +show parameter inmemory_virtual_columns + diff --git a/in-memory-21c/json/files/03_j_purchaseorder.sql b/in-memory-21c/json/files/03_j_purchaseorder.sql new file mode 100644 index 000000000..5136ded4d --- /dev/null +++ b/in-memory-21c/json/files/03_j_purchaseorder.sql @@ -0,0 +1,28 @@ +@../imlogin.sql + +set pages 9999 + +-- List table + +set lines 80 +desc J_PURCHASEORDER + +set lines 150 +col table_name format a20; +col column_name format a30; +col data_type format a30; +col data_length format 999999; +col data_default format a30 WRAPPED; +select TABLE_NAME, COLUMN_NAME, DATA_TYPE, data_length, DATA_DEFAULT from user_tab_cols +where table_name = 'J_PURCHASEORDER' +order by column_id; + +-- List constraints + +col constraint_name format a20; +col search_condition format a30; +select table_name, constraint_name, constraint_type, search_condition +from user_constraints +where table_name = 'J_PURCHASEORDER'; + + diff --git a/in-memory-21c/json/files/04_enable_json.sql b/in-memory-21c/json/files/04_enable_json.sql new file mode 100644 index 000000000..00cf257e0 --- /dev/null +++ b/in-memory-21c/json/files/04_enable_json.sql @@ -0,0 +1,30 @@ +@../imlogin.sql + +set pages 999 +set lines 150 + +set echo on; + +alter table j_purchaseorder add constraint ensure_json check (po_document is json); + +set echo off + +-- List table columns + +col table_name format a20; +col column_name format a30; +col data_type format a30; +col data_length format 999999; +col data_default format a30 WRAPPED; +select TABLE_NAME, COLUMN_NAME, DATA_TYPE, data_length, DATA_DEFAULT from user_tab_cols +where table_name = 'J_PURCHASEORDER' +order by column_id; + +-- List constraints + +col constraint_name format a20; +col search_condition format a30; +select table_name, constraint_name, constraint_type, search_condition +from user_constraints +where table_name = 'J_PURCHASEORDER'; + diff --git a/in-memory-21c/json/files/05_json_query.sql b/in-memory-21c/json/files/05_json_query.sql new file mode 100644 index 000000000..3d23bdf37 --- /dev/null +++ b/in-memory-21c/json/files/05_json_query.sql @@ -0,0 +1,36 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- JSON query + +col costcenter format a20; +col revenue format 999,999,999,999; +SELECT json_value(po_document, '$.CostCenter') as costcenter, + sum(round(jt.UnitPrice * jt.Quantity)) as revenue + FROM j_purchaseorder po, + json_table(po.po_document + COLUMNS (NESTED LineItems[*] + COLUMNS (ItemNumber NUMBER, + UnitPrice PATH Part.UnitPrice, + Quantity NUMBER) + ) + ) AS "JT" + group by json_value(po_document, '$.CostCenter') + order by revenue desc; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/json/files/06_json_pop.sql b/in-memory-21c/json/files/06_json_pop.sql new file mode 100644 index 000000000..9564ad811 --- /dev/null +++ b/in-memory-21c/json/files/06_json_pop.sql @@ -0,0 +1,11 @@ +@../imlogin.sql + +set pages 999 +set lines 150 +set echo on + +alter table j_purchaseorder inmemory; + +exec dbms_inmemory.populate(USER, 'J_PURCHASEORDER'); + +set echo off diff --git a/in-memory-21c/json/files/07_im_populated.sql b/in-memory-21c/json/files/07_im_populated.sql new file mode 100644 index 000000000..0f17b64a8 --- /dev/null +++ b/in-memory-21c/json/files/07_im_populated.sql @@ -0,0 +1,23 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +order by owner, segment_name, partition_name; + +set echo off diff --git a/in-memory-21c/json/files/08_ime_usage.sql b/in-memory-21c/json/files/08_ime_usage.sql new file mode 100644 index 000000000..6f1dbc6c2 --- /dev/null +++ b/in-memory-21c/json/files/08_ime_usage.sql @@ -0,0 +1,35 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +col owner heading "Owner" format a10; +col object_name heading "Object" format a20; +col partition_name heading "Partition|Name" format a20; +col column_name heading "Column|Name" format a15; +col t_imeu heading "Total|IMEUs" format 999999; +col space heading "Used|Space(MB)" format 999,999,999; + +set echo on + +-- This query displays what objects are in the In-Memory Column Store + +select + o.owner, + o.object_name, + o.subobject_name as partition_name, + i.column_name, + count(*) t_imeu, + sum(i.length)/1024/1024 space +from + v$im_imecol_cu i, + dba_objects o +where + i.objd = o.object_id +group by + o.owner, + o.object_name, + o.subobject_name, + i.column_name; + +set echo off diff --git a/in-memory-21c/json/files/09_json_query.sql b/in-memory-21c/json/files/09_json_query.sql new file mode 100644 index 000000000..3d23bdf37 --- /dev/null +++ b/in-memory-21c/json/files/09_json_query.sql @@ -0,0 +1,36 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- JSON query + +col costcenter format a20; +col revenue format 999,999,999,999; +SELECT json_value(po_document, '$.CostCenter') as costcenter, + sum(round(jt.UnitPrice * jt.Quantity)) as revenue + FROM j_purchaseorder po, + json_table(po.po_document + COLUMNS (NESTED LineItems[*] + COLUMNS (ItemNumber NUMBER, + UnitPrice PATH Part.UnitPrice, + Quantity NUMBER) + ) + ) AS "JT" + group by json_value(po_document, '$.CostCenter') + order by revenue desc; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/json/files/10_json_purchaseorder.sql b/in-memory-21c/json/files/10_json_purchaseorder.sql new file mode 100644 index 000000000..22fd8a814 --- /dev/null +++ b/in-memory-21c/json/files/10_json_purchaseorder.sql @@ -0,0 +1,25 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +-- List JSON table + +col table_name format a20; +col column_name format a30; +col data_type format a30; +col data_length format 999999; +col data_default format a30 WRAPPED; +select TABLE_NAME, COLUMN_NAME, DATA_TYPE, data_length, DATA_DEFAULT from user_tab_cols +where table_name = 'JSON_PURCHASEORDER' +order by column_id; + +-- List constraints + +col table_name format a20; +col constraint_name format a20; +col search_condition format a30; +select table_name, constraint_name, constraint_type, search_condition +from user_constraints +where table_name = 'JSON_PURCHASEORDER'; + diff --git a/in-memory-21c/json/files/11_json_pop.sql b/in-memory-21c/json/files/11_json_pop.sql new file mode 100644 index 000000000..651934eb5 --- /dev/null +++ b/in-memory-21c/json/files/11_json_pop.sql @@ -0,0 +1,13 @@ +@../imlogin.sql + +set pages 999 +set lines 150 +set echo on + +alter table j_purchaseorder no inmemory; + +alter table json_purchaseorder inmemory; + +exec dbms_inmemory.populate(USER, 'JSON_PURCHASEORDER'); + +set echo off diff --git a/in-memory-21c/json/files/12_im_populated.sql b/in-memory-21c/json/files/12_im_populated.sql new file mode 100644 index 000000000..2006501fc --- /dev/null +++ b/in-memory-21c/json/files/12_im_populated.sql @@ -0,0 +1,23 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +order by inmemory_size; + +set echo off diff --git a/in-memory-21c/json/files/13_json_query.sql b/in-memory-21c/json/files/13_json_query.sql new file mode 100644 index 000000000..df83153fd --- /dev/null +++ b/in-memory-21c/json/files/13_json_query.sql @@ -0,0 +1,36 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- JSON query + +col costcenter format a20; +col revenue format 999,999,999,999; +SELECT json_value(po_document, '$.CostCenter') as costcenter, + sum(round(jt.UnitPrice * jt.Quantity)) as revenue + FROM json_purchaseorder po, + json_table(po.po_document + COLUMNS (NESTED LineItems[*] + COLUMNS (ItemNumber NUMBER, + UnitPrice PATH Part.UnitPrice, + Quantity NUMBER) + ) + ) AS "JT" + group by json_value(po_document, '$.CostCenter') + order by revenue desc; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/json/files/14_json_cleanup.sql b/in-memory-21c/json/files/14_json_cleanup.sql new file mode 100644 index 000000000..946d285f4 --- /dev/null +++ b/in-memory-21c/json/files/14_json_cleanup.sql @@ -0,0 +1,11 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set echo on + +alter table json_purchaseorder no inmemory; + +alter table j_purchaseorder drop constraint ensure_json; + +set echo off diff --git a/in-memory-21c/json/json.md b/in-memory-21c/json/json.md index fc854bb94..ad6648044 100644 --- a/in-memory-21c/json/json.md +++ b/in-memory-21c/json/json.md @@ -326,7 +326,7 @@ This task will focusing on using JSON data in the IM column store prior to 21c. Note that not only have we created a check constraint on the PO\_DOCUMENT column, but that has also resulted in the creation of a new virtual column that starts with the name SYS\_IME\_OSON. Oracle Database has created a special binary version of the PO\_DOCUMENT column. -3. Next we will run a query on the JSON data to show the functionality and peformance. Remember that we have not yet enabled in-memory processing for the JSON data. +3. Next we will run a query on the JSON data to show the functionality and performance. Remember that we have not yet enabled in-memory processing for the JSON data. Run the script *05\_json\_query.sql* @@ -538,19 +538,9 @@ This task will focusing on using JSON data in the IM column store prior to 21c. In-Memory Bytes OWNER SEGMENT_NAME PARTITION_NAME POPULATE_STATUS Disk Size Size Not Populated ---------- -------------------- --------------- --------------- ---------------- ---------------- ---------------- - SSB CUSTOMER COMPLETED 24,928,256 23,199,744 0 - SSB DATE_DIM COMPLETED 122,880 1,179,648 0 SSB J_PURCHASEORDER COMPLETED 793,198,592 1,069,285,376 0 - SSB LINEORDER PART_1994 COMPLETED 563,601,408 585,236,480 0 - SSB LINEORDER PART_1995 COMPLETED 563,470,336 585,236,480 0 - SSB LINEORDER PART_1996 COMPLETED 565,010,432 587,333,632 0 - SSB LINEORDER PART_1997 COMPLETED 563,314,688 585,236,480 0 - SSB LINEORDER PART_1998 COMPLETED 329,015,296 342,556,672 0 - SSB PART COMPLETED 56,893,440 21,168,128 0 - SSB SUPPLIER COMPLETED 1,769,472 2,228,224 0 - - 10 rows selected. - + + SQL> ``` @@ -631,7 +621,7 @@ This task will focusing on using JSON data in the IM column store prior to 21c. SQL> ``` - Not very much space, but like some of the other features that we've seen in this Lab it does require some additional space in the IM column store in-memory optimized JSON data. + Not very much space, but it does require some additional space in the IM column store in order to store in-memory optimized JSON data. 7. Now let's run the JSON query again. This is the same query from step 3. @@ -775,6 +765,8 @@ This task will focusing on using JSON data in the IM column store prior to 21c. SQL> ``` + A couple of things to note here. The first is that the query ran in about half the time. That is a pretty good speed up. The second is take a look at the statistics. Notice that there are now "IM scan EU" statistics. That's because binary JSON data leverages another Database In-Memory feature called In-Memory Expressions. + ## Task 3: Populating JSON data in the IM column using the JSON data type in 21c 1. Now let's switch to a new table with the PO\_DOCUMENT column defined as a JSON data type. Recall that this is new in Oracle Database 21c. @@ -925,19 +917,9 @@ This task will focusing on using JSON data in the IM column store prior to 21c. Partition In-Memory Bytes Owner SEGMENT_NAME Name POPULATE_STATUS Disk Size Size Not Populated ---------- -------------------- --------------- --------------- ---------------- ---------------- ---------------- - SSB DATE_DIM COMPLETED 122,880 1,179,648 0 - SSB SUPPLIER COMPLETED 1,769,472 2,228,224 0 - SSB PART COMPLETED 56,893,440 21,168,128 0 - SSB CUSTOMER COMPLETED 24,928,256 23,199,744 0 - SSB LINEORDER PART_1998 COMPLETED 329,015,296 342,556,672 0 - SSB LINEORDER PART_1995 COMPLETED 563,470,336 585,236,480 0 - SSB LINEORDER PART_1994 COMPLETED 563,601,408 585,236,480 0 - SSB LINEORDER PART_1997 COMPLETED 563,314,688 585,236,480 0 - SSB LINEORDER PART_1996 COMPLETED 565,010,432 587,333,632 0 SSB JSON_PURCHASEORDER COMPLETED 691,912,704 816,513,024 0 - 10 rows selected. - + SQL> ``` diff --git a/in-memory-21c/queries/files/01_im_query_stats.sql b/in-memory-21c/queries/files/01_im_query_stats.sql new file mode 100644 index 000000000..7d2f6f9f0 --- /dev/null +++ b/in-memory-21c/queries/files/01_im_query_stats.sql @@ -0,0 +1,26 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- In-Memory query + +select + max(lo_ordtotalprice) most_expensive_order, + sum(lo_quantity) total_items +from LINEORDER; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/queries/files/02_buffer_query_stats.sql b/in-memory-21c/queries/files/02_buffer_query_stats.sql new file mode 100644 index 000000000..6c5f59645 --- /dev/null +++ b/in-memory-21c/queries/files/02_buffer_query_stats.sql @@ -0,0 +1,26 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- Buffer Cache query with the column store disabled via NO_INMEMORY hint + +select /*+ NO_INMEMORY */ + max(lo_ordtotalprice) most_expensive_order, + sum(lo_quantity) total_items +from LINEORDER; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/queries/files/03_single_key_im.sql b/in-memory-21c/queries/files/03_single_key_im.sql new file mode 100644 index 000000000..cd18a46fc --- /dev/null +++ b/in-memory-21c/queries/files/03_single_key_im.sql @@ -0,0 +1,21 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- In-Memory Column Store query + +select lo_orderkey, lo_custkey, lo_revenue +from LINEORDER +where lo_orderkey = 5000000; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/queries/files/04_single_key_buffer.sql b/in-memory-21c/queries/files/04_single_key_buffer.sql new file mode 100644 index 000000000..ca15fa455 --- /dev/null +++ b/in-memory-21c/queries/files/04_single_key_buffer.sql @@ -0,0 +1,22 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- Buffer Cache query with the column store disables via INMEMORY_QUERY parameter + +select /*+ NO_INMEMORY */ + lo_orderkey, lo_custkey, lo_revenue +from LINEORDER +where lo_orderkey = 5000000; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + diff --git a/in-memory-21c/queries/files/05_index_comparison.sql b/in-memory-21c/queries/files/05_index_comparison.sql new file mode 100644 index 000000000..69644d11b --- /dev/null +++ b/in-memory-21c/queries/files/05_index_comparison.sql @@ -0,0 +1,28 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set echo on + +-- Show query using an index based on cost + +-- Enable the use of invisible indexes + +alter session set optimizer_use_invisible_indexes=true; + +set timing on + +Select /* With index */ lo_orderkey, lo_custkey, lo_revenue +From LINEORDER +Where lo_orderkey = 5000000; + +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +alter session set optimizer_use_invisible_indexes=false; + +set echo off diff --git a/in-memory-21c/queries/files/06_storage_index.sql b/in-memory-21c/queries/files/06_storage_index.sql new file mode 100644 index 000000000..faee5c471 --- /dev/null +++ b/in-memory-21c/queries/files/06_storage_index.sql @@ -0,0 +1,25 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- Demonstrate the use of In-Memory Storage Indexes + +select lo_orderkey, lo_custkey, lo_revenue +from LINEORDER +where lo_orderkey = 5000000; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/queries/files/07_multi_preds.sql b/in-memory-21c/queries/files/07_multi_preds.sql new file mode 100644 index 000000000..942bb55f2 --- /dev/null +++ b/in-memory-21c/queries/files/07_multi_preds.sql @@ -0,0 +1,27 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- In-Memory query + +select lo_orderkey, lo_custkey, lo_revenue +from LINEORDER +where lo_custkey = 5641 +and lo_shipmode = 'SHIP' +and lo_orderpriority = '5-LOW'; + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/queries/files/08_multi_col.sql b/in-memory-21c/queries/files/08_multi_col.sql new file mode 100644 index 000000000..f41b8dda7 --- /dev/null +++ b/in-memory-21c/queries/files/08_multi_col.sql @@ -0,0 +1,32 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 + +set timing on +set echo on + +-- In-Memory query + +Select lo_orderkey, lo_revenue +From LINEORDER +Where lo_revenue = (Select min(lo_revenue) + From LINEORDER + Where lo_supplycost = (Select max(lo_supplycost) + From LINEORDER + Where lo_quantity > 10) + And lo_shipmode LIKE 'TRUCK%' + And lo_discount between 2 and 5 + ); + +set echo off +set timing off + +pause Hit enter ... + +select * from table(dbms_xplan.display_cursor()); + +pause Hit enter ... + +@../imstats.sql + diff --git a/in-memory-21c/queries/queries.md b/in-memory-21c/queries/queries.md index 0139ab9da..e07f84e03 100644 --- a/in-memory-21c/queries/queries.md +++ b/in-memory-21c/queries/queries.md @@ -421,7 +421,7 @@ SQL> 2 - filter("LO_ORDERKEY"=5000000) ``` -5. Think indexing lo\_orderkey would provide the same performance as the IM column store? There is an invisible index already created on the lo\_orderkey column of the LINEORDER table. By using the parameter OPTIMIZER\_USE\_INVISIBLE\_INDEXES we can compare the performance of the IM column store to using an index. Let's see how well the index performs. +5. Think indexing the LO\_ORDERKEY column would provide the same performance as the IM column store? There is an invisible index already created on the LO\_ORDERKEY column of the LINEORDER table. By using the parameter OPTIMIZER\_USE\_INVISIBLE\_INDEXES we can compare the performance of the IM column store to using an index. Let's see how well the index performs. Run the script *05\_index\_comparison.sql* @@ -614,7 +614,7 @@ SQL> Note that we are now back to an inmemory query. This time we included the session statistics for the query. Take note of two key statistics. The first is "IM scan CUs memcompress for query low". This tells us how many IMCUs the data is populated into. The second important statistic is "IM scan CUs pruned". Notice that this number is almost as large as the total number of IMCUs. This means that Database In-Memory was able to avoid scanning almost all of the data. This is because at population time In-Memory storage indexes are created for each set of column values in each IMCU with the MIN and MAX values. During the scan these MIN and MAX values can be compared with filter predicates and can possibly result in not having to scan the actual columnar data thereby improving performance. After all, the fastest way to do something is to not do it at all. -7. Analytical queries typically have more than one WHERE clause predicate. What happens when there are multiple single column predicates on a table? Traditionally you would create a multi-column index. Can storage indexes compete with that? +7. Analytical queries typically have more than one WHERE clause predicate. What happens when there are multiple single column predicates on a table? Traditionally you would create a multi-column index. Can th IM column store compete with that? Let’s change our query to look for a specific line item in an order and monitor the session statistics: diff --git a/in-memory-21c/setup/files/01_show_parms.sql b/in-memory-21c/setup/files/01_show_parms.sql new file mode 100644 index 000000000..9882d5d29 --- /dev/null +++ b/in-memory-21c/setup/files/01_show_parms.sql @@ -0,0 +1,17 @@ +@../imlogin.sql + +set pages 9999 +set lines 200 +set echo on + +-- Shows the SGA init.ora parameters + +show parameter sga + +show parameter db_keep_cache_size + +show parameter heat_map + +show parameter inmemory_size + +set echo off diff --git a/in-memory-21c/setup/files/02_show_sga.sql b/in-memory-21c/setup/files/02_show_sga.sql new file mode 100644 index 000000000..1b47e6ce6 --- /dev/null +++ b/in-memory-21c/setup/files/02_show_sga.sql @@ -0,0 +1,12 @@ +@../imlogin.sql + +set pages 9999 +set lines 200 +set echo on +set numwidth 20 + +-- Show SGA memory allocation + +show sga + +set echo off diff --git a/in-memory-21c/setup/files/03_im_usage.sql b/in-memory-21c/setup/files/03_im_usage.sql new file mode 100644 index 000000000..7c5d8e884 --- /dev/null +++ b/in-memory-21c/setup/files/03_im_usage.sql @@ -0,0 +1,15 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set echo on +column pool format a10; +column alloc_bytes format 999,999,999,999,999 +column used_bytes format 999,999,999,999,999 + +-- Show total column store usage + +SELECT pool, alloc_bytes, used_bytes, populate_status, con_id +FROM v$inmemory_area; + +set echo off diff --git a/in-memory-21c/setup/files/04_im_alter_table.sql b/in-memory-21c/setup/files/04_im_alter_table.sql new file mode 100644 index 000000000..f22618e20 --- /dev/null +++ b/in-memory-21c/setup/files/04_im_alter_table.sql @@ -0,0 +1,15 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set echo on + +-- Enable tables for in-memory + +alter table LINEORDER inmemory; +alter table PART inmemory; +alter table CUSTOMER inmemory; +alter table SUPPLIER inmemory; +alter table DATE_DIM inmemory; + +set echo off diff --git a/in-memory-21c/setup/files/05_im_attributes.sql b/in-memory-21c/setup/files/05_im_attributes.sql new file mode 100644 index 000000000..3ff69beea --- /dev/null +++ b/in-memory-21c/setup/files/05_im_attributes.sql @@ -0,0 +1,29 @@ +@../imlogin.sql + +set pages 999 +set lines 200 +set tab off + +column table_name format a12; +column partition_name format a15; +column buffer_pool format a11; +column compression heading 'DISK|COMPRESSION' format a11; +column compress_for format a12; +column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; +column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; +column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; +set echo on + +-- Show table attributes + +select table_name, NULL as partition_name, buffer_pool, compression, compress_for, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from user_tables +where table_name in ('DATE_DIM','PART','SUPPLIER','CUSTOMER') +UNION ALL +select table_name, partition_name, buffer_pool, compression, compress_for, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression +from user_tab_partitions +where table_name = 'LINEORDER'; + +set echo off diff --git a/in-memory-21c/setup/files/06_im_start_pop.sql b/in-memory-21c/setup/files/06_im_start_pop.sql new file mode 100644 index 000000000..bb516f0b0 --- /dev/null +++ b/in-memory-21c/setup/files/06_im_start_pop.sql @@ -0,0 +1,15 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set echo on + +-- Access tables enabled for in-memory to start population + +select /*+ full(LINEORDER) noparallel(LINEORDER) */ count(*) from LINEORDER; +select /*+ full(PART) noparallel(PART) */ count(*) from PART; +select /*+ full(CUSTOMER) noparallel(CUSTOMER) */ count(*) from CUSTOMER; +select /*+ full(SUPPLIER) noparallel(SUPPLIER) */ count(*) from SUPPLIER; +select /*+ full(DATE_DIM) noparallel(DATE_DIM) */ count(*) from DATE_DIM; + +set echo off diff --git a/in-memory-21c/setup/files/07_populate_wait.sql b/in-memory-21c/setup/files/07_populate_wait.sql new file mode 100644 index 000000000..b15b2747d --- /dev/null +++ b/in-memory-21c/setup/files/07_populate_wait.sql @@ -0,0 +1,19 @@ +@../imlogin.sql + +set pages 999 +set lines 200 + +set echo on + +-- Show populate_wait query + +-- Return code: +-- -1 = POPULATE_TIMEOUT +-- 0 = POPULATE_SUCCESS +-- 1 = POPULATE_OUT_OF_MEMORY +-- 2 = POPULATE_NO_INMEMORY_OBJECTS +-- 3 = POPULATE_INMEMORY_SIZE_ZERO + +select dbms_inmemory_admin.populate_wait(priority=>'NONE',percentage=>100,timeout=>60) pop_status from dual; + +set echo off diff --git a/in-memory-21c/setup/files/08_im_populated.sql b/in-memory-21c/setup/files/08_im_populated.sql new file mode 100644 index 000000000..5450a601d --- /dev/null +++ b/in-memory-21c/setup/files/08_im_populated.sql @@ -0,0 +1,25 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set tab off + +column owner format a10; +column segment_name format a20; +column partition_name format a15; +column populate_status format a15; +column bytes heading 'Disk Size' format 999,999,999,999 +column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 +column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 +set echo on + +-- Query the view v$IM_SEGMENTS to shows what objects are in the column store +-- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 +-- it indicates the entire table was populated. + +select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated +from v$im_segments +order by owner, segment_name, partition_name; + +set echo off diff --git a/in-memory-21c/setup/files/09_im_usage.sql b/in-memory-21c/setup/files/09_im_usage.sql new file mode 100644 index 000000000..7c5d8e884 --- /dev/null +++ b/in-memory-21c/setup/files/09_im_usage.sql @@ -0,0 +1,15 @@ +@../imlogin.sql + +set pages 9999 +set lines 150 +set echo on +column pool format a10; +column alloc_bytes format 999,999,999,999,999 +column used_bytes format 999,999,999,999,999 + +-- Show total column store usage + +SELECT pool, alloc_bytes, used_bytes, populate_status, con_id +FROM v$inmemory_area; + +set echo off diff --git a/in-memory-21c/setup/setup.md b/in-memory-21c/setup/setup.md index 52d85ff04..367e592df 100644 --- a/in-memory-21c/setup/setup.md +++ b/in-memory-21c/setup/setup.md @@ -72,7 +72,7 @@ SQL> set lines 150 SQL> ``` -1. Database In-Memory is integrated into Oracle Database 12c (12.1.0.2) and higher. The IM column store is not enabled by default, but can be easily enabled via a few steps. Before you enable it, let's take a look at the default configuration. Set your oracle environment +1. Database In-Memory is integrated into Oracle Database 12c (12.1.0.2) and higher. The IM column store is not enabled by default, but can be easily enabled via a few steps. Before you enable it, let's take a look at the default configuration. Run the script *01\_show\_parms.sql* @@ -134,7 +134,7 @@ SQL> These parameters have already been set for this Lab. The IM column store is not enabled by default (i.e. INMEMORY\_SIZE=0), but we have set it to a size that will work for this Lab. HEAT\_MAP defaults to OFF, but it has been enabled for one of the later labs. The KEEP pool (i.e. DB\_KEEP\_CACHE\_SIZE) is set to 0 by default. We have defined it for this Lab so that you can compare the performance of objects populated in the IM column store with the same objects fully cached in the buffer cache and compare the difference in performance for yourself. -2. Database In-Memory is integrated into Oracle Database 12c (12.1.0.2) and higher. The IM column store is allocated within the System Global Area (SGA) and can be easily displayed using normal database commands. +2. Database In-Memory is fully integrated into Oracle Database. The IM column store is allocated within the System Global Area (SGA) and can be easily displayed using normal database commands. Run the script *02\_show\_sga.sql* @@ -406,6 +406,8 @@ SQL> SQL> ``` + Note the FULL and NOPARALLEL hints. These have been added to ensure that the table data is also read into the KEEP pool that was defined. This is only done for this Lab so that we can show you a true memory based comparison of the performance of the Database In-Memory columnar format versus the traditional row format. This is not required to initiate Database In-Memory population. + 7. There is a function available that enables the ability to programatically check if the IM column store has been populated. The function, dbms\_inmemory\_admin.populate_wait returns a code based on populate priority and percentage of population: ``` @@ -610,7 +612,7 @@ SQL> ## Conclusion -In this lab you saw that the IM column store is configured by setting the initialization parameter INMEMORY_SIZE. The IM column store is a new static pool in the SGA, and once allocated it can be resized dynamically, but it is not managed by either of the automatic SGA memory features. +In this lab you saw that the IM column store is configured by setting the initialization parameter INMEMORY_SIZE. The IM column store is a new static pool in the SGA, and once allocated it can be increased in size dynamically, but it is not managed by either of the automatic SGA memory features. You also had an opportunity to populate and view objects in the IM column store and to see how much memory they use. In this lab we populated five tables into the IM column store, and the LINEORDER table is the largest of the tables populated with over 41 million rows. You may have noticed that it is also a partitioned table. We will be using that attribute in later labs.