feat: Data source cache #2164

scsmithr · 2023-11-24T19:09:50Z

Adds caching of external database table info.

> CREATE EXTERNAL DATABASE my_pg
::: FROM postgres OPTIONS (
:::   host = 'pg.demo.glaredb.com',
:::   port = '5432',
:::   user = 'demo',
:::   password = 'demo',
:::   database = 'postgres',
::: );
Database created
> select * from cache_external_database_tables();
┌────────────────────────────────┬─────────────────────────┬─────────────────────────┐
│ system_operation_name          │ started_at              │ finished_at             │
│ ──                             │ ──                      │ ──                      │
│ Utf8                           │ Date64                  │ Date64                  │
╞════════════════════════════════╪═════════════════════════╪═════════════════════════╡
│ cache_external_database_tables │ 2023-12-05T03:10:17.086 │ 2023-12-05T03:10:42.756 │
└────────────────────────────────┴─────────────────────────┴─────────────────────────┘
> select * from glare_catalog.cached_external_database_tables;
┌──────────────┬─────────────┬──────────────────┬───────────────────────┬────────────────────────────────────┐
│ database_oid │ schema_name │ table_name       │ column_name           │ data_type                          │
│           ── │ ──          │ ──               │ ──                    │ ──                                 │
│        Int32 │ Utf8        │ Utf8             │ Utf8                  │ Utf8                               │
╞══════════════╪═════════════╪══════════════════╪═══════════════════════╪════════════════════════════════════╡
│        20000 │ pg_catalog  │ pg_stat_archiver │ archived_count        │ Int64                              │
│        20000 │ pg_catalog  │ pg_stat_archiver │ last_archived_wal     │ Utf8                               │
│        20000 │ pg_catalog  │ pg_stat_archiver │ last_archived_time    │ Timestamp(Nanosecond, Some("UTC")) │
│        20000 │ pg_catalog  │ pg_stat_archiver │ failed_count          │ Int64                              │
│        20000 │ pg_catalog  │ pg_stat_archiver │ last_failed_wal       │ Utf8                               │
│        20000 │ pg_catalog  │ pg_stat_archiver │ last_failed_time      │ Timestamp(Nanosecond, Some("UTC")) │
│        20000 │ pg_catalog  │ pg_stat_archiver │ stats_reset           │ Timestamp(Nanosecond, Some("UTC")) │
│        20000 │ pg_catalog  │ pg_stat_bgwriter │ checkpoints_timed     │ Int64                              │
│        20000 │ pg_catalog  │ pg_stat_bgwriter │ checkpoints_req       │ Int64                              │
│        20000 │ pg_catalog  │ pg_stat_bgwriter │ checkpoint_write_time │ Float64                            │
│            … │ …           │ …                │ …                     │ …                                  │
│        20000 │ public      │ orders           │ o_comment             │ Utf8                               │
│        20000 │ public      │ part             │ p_partkey             │ Int32                              │
│        20000 │ public      │ part             │ p_name                │ Utf8                               │
│        20000 │ public      │ part             │ p_mfgr                │ Utf8                               │
│        20000 │ public      │ part             │ p_brand               │ Utf8                               │
│        20000 │ public      │ part             │ p_type                │ Utf8                               │
│        20000 │ public      │ part             │ p_size                │ Int32                              │
│        20000 │ public      │ part             │ p_container           │ Utf8                               │
│        20000 │ public      │ part             │ p_retailprice         │ Decimal128(38, 9)                  │
│        20000 │ public      │ part             │ p_comment             │ Utf8                               │
└──────────────┴─────────────┴──────────────────┴───────────────────────┴────────────────────────────────────┘
 131 rows (20 shown)

tychoish · 2023-11-25T08:22:32Z

crates/sqlbuiltins/src/builtins.rs

+///
+/// This means we have ~3600 OIDs to play with for builtin objects. Note that
+/// once a builtin object is given a stable OID, it **must not** be changed ever
+/// (unless you're the person willing to write a migration system).


which we should do in a while?

Likely, just want to try to delay it as much as possible to better understand what needs to happen for a system like that.

1000% agree.

tychoish · 2023-11-25T08:25:02Z

crates/sqlbuiltins/src/builtins.rs

+/// First glaredb builtin OID: 16384
+/// First user object OID: 20000
+///
+/// This means we have ~3600 OIDs to play with for builtin objects. Note that


why do they have to be sequentially grouped?

They don't, it's just easier to track what the next OID should be when adding a table/schema/whatever. If we have more than 16 builtin schemas (unlikely), we can just jump forward to some other available OID.

tychoish · 2023-12-04T09:13:09Z

What's the relationship of this pr to #2201

scsmithr · 2023-12-04T14:57:01Z

What's the relationship of this pr to #2201

I expect Cloud to use the simple query interface to call the cache function(s) in this PR for updating the cache, but they're two independent features.

universalmind303 · 2023-12-05T02:29:09Z

crates/datafusion_ext/src/system.rs

+///
+/// This should be focused on operations that do not require user interactions
+/// (e.g. background operations like optimizing tables or running caching jobs).
+pub trait SystemOperation: Sync + Send {


Would we ever have an unknown number of SystemOperation's, or custom operations? Considering we are dealing with an exhaustive list of operations, I'm wondering if it'd make sense to use an enum here instead?

Fair, will change

universalmind303 · 2023-12-05T02:40:47Z

crates/sqlbuiltins/src/functions/table/system/cache_external_tables.rs

+impl ConstBuiltinFunction for CacheExternalDatabaseTables {
+    const NAME: &'static str = "cache_external_database_tables";
+    const DESCRIPTION: &'static str = "Cache tables from external databases.";
+    const EXAMPLE: &'static str = "select * from cache_external_database_tables();";


Something for another day, but it'd be cool to support the CALL syntax for this.

while valid, select * from cache_external_database_tables(); seems weird to me as im selecting from an action.

CALL seems fitting for something that is just executing a function call.

CALL cache_external_database_tables();`

I agree, I originally wanted to use CALL, but sqlparser doesn't support it.

scsmithr · 2023-12-05T15:30:47Z

Force merging, CI failure due to #2168

tychoish reviewed Nov 25, 2023

View reviewed changes

scsmithr added 2 commits November 30, 2023 13:21

Change reset

12a44ae

Merge remote-tracking branch 'origin/main' into sean/cache

7d94623

scsmithr force-pushed the sean/cache branch from 112c05d to 7d94623 Compare November 30, 2023 19:22

scsmithr added 2 commits November 30, 2023 15:54

making progress

531b732

plan

45a053a

scsmithr added 5 commits December 4, 2023 17:12

Merge remote-tracking branch 'origin/main' into sean/cache

471492b

wip use new traits

01d3b74

Merge remote-tracking branch 'origin/main' into sean/cache

f57a5c1

Move exec plans from sql exec

2267b47

Add columns

e4e890d

universalmind303 reviewed Dec 5, 2023

View reviewed changes

scsmithr added 2 commits December 4, 2023 20:47

Add tests

086d913

use enum

c8d5ab3

scsmithr requested review from tychoish, universalmind303 and vrongmeal December 5, 2023 03:08

scsmithr marked this pull request as ready for review December 5, 2023 03:08

Remove unused module

c930394

tychoish approved these changes Dec 5, 2023

View reviewed changes

Merge branch 'main' into sean/cache

6281dba

scsmithr enabled auto-merge (squash) December 5, 2023 15:05

scsmithr disabled auto-merge December 5, 2023 15:30

scsmithr merged commit 82b9d81 into main Dec 5, 2023
8 of 9 checks passed

scsmithr deleted the sean/cache branch December 5, 2023 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Data source cache #2164

feat: Data source cache #2164

scsmithr commented Nov 24, 2023 •

edited

Loading

tychoish Nov 25, 2023

scsmithr Nov 25, 2023

tychoish Nov 25, 2023

tychoish Nov 25, 2023

scsmithr Nov 25, 2023

tychoish commented Dec 4, 2023

scsmithr commented Dec 4, 2023

universalmind303 Dec 5, 2023

scsmithr Dec 5, 2023

universalmind303 Dec 5, 2023

scsmithr Dec 5, 2023

scsmithr commented Dec 5, 2023

feat: Data source cache #2164

feat: Data source cache #2164

Conversation

scsmithr commented Nov 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tychoish commented Dec 4, 2023

scsmithr commented Dec 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scsmithr commented Dec 5, 2023

scsmithr commented Nov 24, 2023 •

edited

Loading