Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flatten file signals are still working in filters #212

Closed
Tracked by #33
dmpetrov opened this issue Jul 31, 2024 · 6 comments
Closed
Tracked by #33

Flatten file signals are still working in filters #212

dmpetrov opened this issue Jul 31, 2024 · 6 comments
Labels
blocked Blocked by other tasks priority-p2

Comments

@dmpetrov
Copy link
Member

dmpetrov commented Jul 31, 2024 •
edited
Loading 

from datachain import C, DataChain

(
    DataChain.from_storage("gs://dvcx-datacomp-small/metadata", anon=True)
    .filter(C("name").glob("002*.npz"))
    .show(5)
)

OUTPUT:

Processed: 52 rows [00:00, 6608.21 rows/s]
                       file      file                                  file        file              file  \
                     source    parent                                  name        size           version
0  gs://dvcx-datacomp-small  metadata  0020f0cbd157d470aa56bea08e304b90.npz  3106844080  1705333045264343
1  gs://dvcx-datacomp-small  metadata  002372587416601d1a339f7589e00276.npz  3097824688  1705332995256821

               file      file                             file     file  file
               etag is_latest                    last_modified location vtype
0  CNf/i9Pc34MDEAE=         1 2024-01-15 15:37:25.267000+00:00     None
1  CPXjn7vc34MDEAE=         1 2024-01-15 15:36:35.262000+00:00     None

How it's possible? THere are not name signals.
@dmpetrov dmpetrov mentioned this issue Jul 31, 2024
Closed
@mattseddon
Copy link
Member

This happens because DataChain.from_storage returns

        return (
            cls(path, session=session, recursive=recursive, update=update, **kwargs)
            .map(**{object_name: func})
            .select(object_name) 
        )

which leaves the original names available in the chain until save is called. I.e.

    (
        DataChain.from_storage("gs://dvcx-datacomp-small/metadata", anon=True)
        .save()
        .filter(C("name").glob("002*.npz"))
        .show()
    )

will fail with sqlite3.OperationalError: no such column: name

@mattseddon
Copy link
Member

The full generated SQL is below (the part we are interested in for this issue is INSERT INTO "udf_RPWfGg"):

SQL 
begin
INSERT INTO buckets (uri, status, error_message, error_stack) VALUES ('gs://dvcx-datacomp-small', 1, '', '') ON CONFLICT DO NOTHING
SELECT buckets.id, buckets.uri, buckets.timestamp, buckets.expires, buckets.started_inserting_at, buckets.last_inserted_at, buckets.status, buckets.error_message, buckets.error_stack 
FROM buckets 
WHERE buckets.uri = 'gs://dvcx-datacomp-small'
UPDATE buckets SET timestamp=NULL, expires=NULL, started_inserting_at='2024-08-01 01:50:25.156611', last_inserted_at=NULL, status=2 WHERE buckets.uri = 'gs://dvcx-datacomp-small'
COMMIT
INSERT INTO id_generator (uri, last_id) VALUES ('partials:gs://dvcx-datacomp-small', 0) ON CONFLICT DO NOTHING
begin
INSERT INTO id_generator (uri, last_id) VALUES ('partials:gs://dvcx-datacomp-small', 1) ON CONFLICT (uri) DO UPDATE SET last_id = (id_generator.last_id + 1)
SELECT id_generator.last_id 
FROM id_generator 
WHERE id_generator.uri = 'partials:gs://dvcx-datacomp-small'
COMMIT

CREATE TABLE IF NOT EXISTS prt__ba3a3313db27 (
        path_str TEXT NOT NULL, 
        partial_id INTEGER NOT NULL, 
        timestamp DATETIME, 
        expires DATETIME
)


CREATE INDEX IF NOT EXISTS ix_prt__ba3a3313db27_partial_id ON prt__ba3a3313db27 (partial_id)

CREATE TABLE IF NOT EXISTS buckets (
        id INTEGER NOT NULL, 
        uri TEXT NOT NULL, 
        timestamp DATETIME, 
        expires DATETIME, 
        started_inserting_at DATETIME, 
        last_inserted_at DATETIME, 
        status INTEGER NOT NULL, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (uri)
)



CREATE TABLE IF NOT EXISTS datasets (
        id INTEGER NOT NULL, 
        name TEXT NOT NULL, 
        description TEXT, 
        labels JSON, 
        shadow BOOLEAN NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        PRIMARY KEY (id), 
        UNIQUE (name)
)



CREATE TABLE IF NOT EXISTS datasets_versions (
        id INTEGER NOT NULL, 
        dataset_id INTEGER NOT NULL, 
        version INTEGER NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        num_objects BIGINT, 
        size BIGINT, 
        preview JSON, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        job_id TEXT, 
        is_job_result BOOLEAN NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (dataset_id, version), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id) ON DELETE CASCADE
)



CREATE TABLE IF NOT EXISTS datasets_dependencies (
        id INTEGER NOT NULL, 
        source_dataset_id INTEGER NOT NULL, 
        source_dataset_version_id INTEGER, 
        dataset_id INTEGER, 
        dataset_version_id INTEGER, 
        bucket_id INTEGER, 
        bucket_version TEXT, 
        PRIMARY KEY (id), 
        FOREIGN KEY(source_dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(source_dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(bucket_id) REFERENCES buckets (id)
)



CREATE TABLE IF NOT EXISTS jobs (
        id TEXT NOT NULL, 
        name TEXT NOT NULL, 
        status INTEGER NOT NULL, 
        created_at DATETIME NOT NULL, 
        finished_at DATETIME, 
        "query" TEXT NOT NULL, 
        query_type INTEGER NOT NULL, 
        workers INTEGER NOT NULL, 
        python_version TEXT, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        params JSON NOT NULL, 
        metrics JSON NOT NULL, 
        PRIMARY KEY (id)
)


SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
INSERT INTO datasets (name, shadow, status, feature_schema, created_at, error_message, error_stack, script_output, sources, query_script, schema) VALUES ('gs://dvcx-datacomp-small/metadata/', 0, 1, '{}', '2024-08-01 01:50:25.162689', '', '', '', '', '', '{"vtype": {"type": "String"}, "dir_type": {"type": "Int"}, "parent": {"type": "String"}, "name": {"type": "String"}, "etag": {"type": "String"}, "version": {"type": "String"}, "is_latest": {"type": "Boolean"}, "last_modified": {"type": "DateTime"}, "size": {"type": "Int64"}, "owner_name": {"type": "String"}, "owner_id": {"type": "String"}, "location": {"type": "JSON"}, "source": {"type": "String"}}') ON CONFLICT (name) DO NOTHING
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
INSERT INTO datasets_versions (dataset_id, version, status, feature_schema, created_at, finished_at, error_message, error_stack, script_output, num_objects, size, preview, sources, query_script, schema, job_id, is_job_result) VALUES (1, 1, 2, '{}', '2024-08-01 01:50:25.163717', NULL, '', '', '', NULL, NULL, '[]', '', '', '{"vtype": {"type": "String"}, "dir_type": {"type": "Int"}, "parent": {"type": "String"}, "name": {"type": "String"}, "etag": {"type": "String"}, "version": {"type": "String"}, "is_latest": {"type": "Boolean"}, "last_modified": {"type": "DateTime"}, "size": {"type": "Int64"}, "owner_name": {"type": "String"}, "owner_id": {"type": "String"}, "location": {"type": "JSON"}, "source": {"type": "String"}}', NULL, 0) ON CONFLICT (dataset_id, version) DO NOTHING
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'

CREATE TABLE IF NOT EXISTS "src_gs://dvcx-datacomp-small/metadata/_1" (
        sys__id INTEGER NOT NULL, 
        sys__rand INTEGER DEFAULT (abs(random())) NOT NULL, 
        vtype VARCHAR, 
        dir_type INTEGER, 
        parent VARCHAR, 
        name VARCHAR, 
        etag VARCHAR, 
        version VARCHAR, 
        is_latest BOOLEAN, 
        last_modified DATETIME, 
        size INTEGER, 
        owner_name VARCHAR, 
        owner_id VARCHAR, 
        location JSON, 
        source VARCHAR, 
        PRIMARY KEY (sys__id)
)


SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'src_gs://dvcx-datacomp-small/metadata/_1') AS anon_1
SELECT count("src_gs://dvcx-datacomp-small/metadata/_1".sys__id) AS count_1, sum("src_gs://dvcx-datacomp-small/metadata/_1".size) AS sum_1 
FROM "src_gs://dvcx-datacomp-small/metadata/_1"
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
SELECT "src_gs://dvcx-datacomp-small/metadata/_1".sys__id, "src_gs://dvcx-datacomp-small/metadata/_1".sys__rand, "src_gs://dvcx-datacomp-small/metadata/_1".vtype, "src_gs://dvcx-datacomp-small/metadata/_1".dir_type, "src_gs://dvcx-datacomp-small/metadata/_1".parent, "src_gs://dvcx-datacomp-small/metadata/_1".name, "src_gs://dvcx-datacomp-small/metadata/_1".etag, "src_gs://dvcx-datacomp-small/metadata/_1".version, "src_gs://dvcx-datacomp-small/metadata/_1".is_latest, "src_gs://dvcx-datacomp-small/metadata/_1".last_modified, "src_gs://dvcx-datacomp-small/metadata/_1".size, "src_gs://dvcx-datacomp-small/metadata/_1".owner_name, "src_gs://dvcx-datacomp-small/metadata/_1".owner_id, "src_gs://dvcx-datacomp-small/metadata/_1".location, "src_gs://dvcx-datacomp-small/metadata/_1".source 
FROM "src_gs://dvcx-datacomp-small/metadata/_1"
 LIMIT 20 OFFSET 0
UPDATE datasets_versions SET num_objects=0, preview='[]' WHERE datasets_versions.id = 1

CREATE TABLE IF NOT EXISTS buckets (
        id INTEGER NOT NULL, 
        uri TEXT NOT NULL, 
        timestamp DATETIME, 
        expires DATETIME, 
        started_inserting_at DATETIME, 
        last_inserted_at DATETIME, 
        status INTEGER NOT NULL, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (uri)
)



CREATE TABLE IF NOT EXISTS datasets (
        id INTEGER NOT NULL, 
        name TEXT NOT NULL, 
        description TEXT, 
        labels JSON, 
        shadow BOOLEAN NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        PRIMARY KEY (id), 
        UNIQUE (name)
)



CREATE TABLE IF NOT EXISTS datasets_versions (
        id INTEGER NOT NULL, 
        dataset_id INTEGER NOT NULL, 
        version INTEGER NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        num_objects BIGINT, 
        size BIGINT, 
        preview JSON, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        job_id TEXT, 
        is_job_result BOOLEAN NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (dataset_id, version), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id) ON DELETE CASCADE
)



CREATE TABLE IF NOT EXISTS datasets_dependencies (
        id INTEGER NOT NULL, 
        source_dataset_id INTEGER NOT NULL, 
        source_dataset_version_id INTEGER, 
        dataset_id INTEGER, 
        dataset_version_id INTEGER, 
        bucket_id INTEGER, 
        bucket_version TEXT, 
        PRIMARY KEY (id), 
        FOREIGN KEY(source_dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(source_dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(bucket_id) REFERENCES buckets (id)
)



CREATE TABLE IF NOT EXISTS jobs (
        id TEXT NOT NULL, 
        name TEXT NOT NULL, 
        status INTEGER NOT NULL, 
        created_at DATETIME NOT NULL, 
        finished_at DATETIME, 
        "query" TEXT NOT NULL, 
        query_type INTEGER NOT NULL, 
        workers INTEGER NOT NULL, 
        python_version TEXT, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        params JSON NOT NULL, 
        metrics JSON NOT NULL, 
        PRIMARY KEY (id)
)


PRAGMA read_uncommitted
PRAGMA main.table_xinfo("src_gs://dvcx-datacomp-small/metadata/_1")
SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = 'src_gs://dvcx-datacomp-small/metadata/_1' AND type in ('table', 'view')
PRAGMA main.foreign_key_list("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA temp.foreign_key_list("src_gs://dvcx-datacomp-small/metadata/_1")
SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = 'src_gs://dvcx-datacomp-small/metadata/_1' AND type in ('table', 'view')
PRAGMA main.index_list("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA temp.index_list("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA main.table_info("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA main.index_list("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA temp.index_list("src_gs://dvcx-datacomp-small/metadata/_1")
PRAGMA main.table_info("src_gs://dvcx-datacomp-small/metadata/_1")
SELECT sql FROM  (SELECT * FROM sqlite_master UNION ALL   SELECT * FROM sqlite_temp_master) WHERE name = 'src_gs://dvcx-datacomp-small/metadata/_1' AND type in ('table', 'view')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0020f0cbd157d470aa56bea08e304b90.npz', 'CNf/i9Pc34MDEAE=', '1705333045264343', 1, '2024-01-15 15:37:25.267000', 3106844080, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0020f0cbd157d470aa56bea08e304b90.parquet', 'CPXavc/a34MDEAE=', '1705332500819317', 1, '2024-01-15 15:28:20.824000', 120845372, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '002372587416601d1a339f7589e00276.npz', 'CPXjn7vc34MDEAE=', '1705332995256821', 1, '2024-01-15 15:36:35.262000', 3097824688, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '002372587416601d1a339f7589e00276.parquet', 'CJOZvM7a34MDEAE=', '1705332498697363', 1, '2024-01-15 15:28:18.699000', 120743149, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '003497f3b357f591b59fd6f20967174e.npz', 'CMfY08vc34MDEAE=', '1705333029661767', 1, '2024-01-15 15:37:09.667000', 3140224432, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '003497f3b357f591b59fd6f20967174e.parquet', 'CMeV+s3a34MDEAE=', '1705332497615559', 1, '2024-01-15 15:28:17.619000', 122357389, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '005e1941475026c5167278c9b564d204.npz', 'CI3/osDc34MDEAE=', '1705333005795213', 1, '2024-01-15 15:36:45.800000', 3112410544, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '005e1941475026c5167278c9b564d204.parquet', 'CPaTvs7a34MDEAE=', '1705332498729462', 1, '2024-01-15 15:28:18.731000', 121353140, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '006731584dd46fed36eafe8956742f7f.npz', 'CN6zncLc34MDEAE=', '1705333009897950', 1, '2024-01-15 15:36:49.904000', 3130086832, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '006731584dd46fed36eafe8956742f7f.parquet', 'CIK1/M7a34MDEAE=', '1705332499749506', 1, '2024-01-15 15:28:19.752000', 122071128, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '006f4f04359ef9da4fc0c76a589ef5d0.npz', 'CPuTncHc34MDEAE=', '1705333007796731', 1, '2024-01-15 15:36:47.802000', 3132906928, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '006f4f04359ef9da4fc0c76a589ef5d0.parquet', 'CNn3ttDa34MDEAE=', '1705332502805465', 1, '2024-01-15 15:28:22.808000', 122093895, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '008187fcbb0910e2f03d30fd0d497176.npz', 'CMrUi8bc34MDEAE=', '1705333017995850', 1, '2024-01-15 15:36:58', 3126431152, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '008187fcbb0910e2f03d30fd0d497176.parquet', 'CJOovc/a34MDEAE=', '1705332500812819', 1, '2024-01-15 15:28:20.816000', 121693996, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '008d6058981efa6c89a18ee29d53f2fc.npz', 'COOcytfc34MDEAE=', '1705333054672483', 1, '2024-01-15 15:37:34.677000', 3120219568, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '008d6058981efa6c89a18ee29d53f2fc.parquet', 'COjius3a34MDEAE=', '1705332496576872', 1, '2024-01-15 15:28:16.578000', 121674406, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00a735dd2dd539edc0f4368bd13a0c37.npz', 'CLiE5MTc34MDEAE=', '1705333015249464', 1, '2024-01-15 15:36:55.255000', 3120262576, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00a735dd2dd539edc0f4368bd13a0c37.parquet', 'CJyUgM7a34MDEAE=', '1705332497713692', 1, '2024-01-15 15:28:17.716000', 121662307, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00d88deb6cc5ec92105ba44e7441e22f.npz', 'CLShwsTc34MDEAE=', '1705333014696116', 1, '2024-01-15 15:36:54.699000', 3132550576, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00d88deb6cc5ec92105ba44e7441e22f.parquet', 'CMidw87a34MDEAE=', '1705332498812616', 1, '2024-01-15 15:28:18.816000', 121983445, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00d9c454e77e6af9a03e0fdd78f2809a.npz', 'CLms/Lvc34MDEAE=', '1705332996773433', 1, '2024-01-15 15:36:36.778000', 3117491632, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '00d9c454e77e6af9a03e0fdd78f2809a.parquet', 'COK9+s3a34MDEAE=', '1705332497620706', 1, '2024-01-15 15:28:17.623000', 121418799, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01381e342c2964e0ef8c79898f2938c6.npz', 'CKqb7cDc34MDEAE=', '1705333007011242', 1, '2024-01-15 15:36:47.017000', 3134731696, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01381e342c2964e0ef8c79898f2938c6.parquet', 'CJLhs9Ha34MDEAE=', '1705332504850578', 1, '2024-01-15 15:28:24.853000', 122126284, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01abd55893ee680bcdb2bc7322f3629e.npz', 'CMDAssDc34MDEAE=', '1705333006049344', 1, '2024-01-15 15:36:46.052000', 3122609584, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01abd55893ee680bcdb2bc7322f3629e.parquet', 'CLbdqs7a34MDEAE=', '1705332498411190', 1, '2024-01-15 15:28:18.414000', 121739730, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01d8289bd0868520f5d94431ae6ccfed.npz', 'COn5zb3c34MDEAE=', '1705333000207593', 1, '2024-01-15 15:36:40.212000', 3106033072, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '01d8289bd0868520f5d94431ae6ccfed.parquet', 'CMaVp8/a34MDEAE=', '1705332500449990', 1, '2024-01-15 15:28:20.453000', 120940243, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0239539d00c512673efa3efb7d9f4957.npz', 'CO2fl8rc34MDEAE=', '1705333026574317', 1, '2024-01-15 15:37:06.577000', 3108822448, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0239539d00c512673efa3efb7d9f4957.parquet', 'CK+Axc3a34MDEAE=', '1705332496744495', 1, '2024-01-15 15:28:16.747000', 121060510, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '027b4ef011d33804135ca19cf7cf12f6.npz', 'COu8jLzc34MDEAE=', '1705332997037675', 1, '2024-01-15 15:36:37.042000', 3130670512, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '027b4ef011d33804135ca19cf7cf12f6.parquet', 'CPTK1tPa34MDEAE=', '1705332509615476', 1, '2024-01-15 15:28:29.617000', 121856148, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02a0fc6738dfb212ef9e2a0ac43e2df2.npz', 'CPK/j83c34MDEAE=', '1705333032738802', 1, '2024-01-15 15:37:12.744000', 3106469296, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02a0fc6738dfb212ef9e2a0ac43e2df2.parquet', 'CJnq883a34MDEAE=', '1705332497511705', 1, '2024-01-15 15:28:17.517000', 121093891, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02a82399895757e6f8eff9ecd7d6d610.npz', 'CIHJu8fc34MDEAE=', '1705333020877953', 1, '2024-01-15 15:37:00.883000', 3122990512, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02a82399895757e6f8eff9ecd7d6d610.parquet', 'CNG6p8/a34MDEAE=', '1705332500454737', 1, '2024-01-15 15:28:20.457000', 121693428, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02ad5a8f481c6a0a71fec68556d0d18a.npz', 'CIiJvMDc34MDEAE=', '1705333006206088', 1, '2024-01-15 15:36:46.211000', 3103194544, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02ad5a8f481c6a0a71fec68556d0d18a.parquet', 'CNOk7s7a34MDEAE=', '1705332499518035', 1, '2024-01-15 15:28:19.520000', 120885156, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02e00078ce738fdc5393f73a9d89fa70.npz', 'CMnB0r/c34MDEAE=', '1705333004476617', 1, '2024-01-15 15:36:44.481000', 3123525040, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02e00078ce738fdc5393f73a9d89fa70.parquet', 'CNv41c/a34MDEAE=', '1705332501216347', 1, '2024-01-15 15:28:21.219000', 121834190, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02e27cb3e4bcb4c4553e8a4fb94bc873.npz', 'CKzl6d/c34MDEAE=', '1705333071966892', 1, '2024-01-15 15:37:51.971000', 3123482032, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02e27cb3e4bcb4c4553e8a4fb94bc873.parquet', 'CLOzwc7a34MDEAE=', '1705332498782643', 1, '2024-01-15 15:28:18.785000', 121580016, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02f4d9e757155f708289bf259e108ea5.npz', 'CN/Bs73c34MDEAE=', '1705332999774431', 1, '2024-01-15 15:36:39.779000', 3108687280, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02f4d9e757155f708289bf259e108ea5.parquet', 'CJWGq8/a34MDEAE=', '1705332500513557', 1, '2024-01-15 15:28:20.516000', 121020542, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02fff3c8c49056affd52e2c59a338ee8.npz', 'CKrnhcTc34MDEAE=', '1705333013705642', 1, '2024-01-15 15:36:53.710000', 3105308080, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '02fff3c8c49056affd52e2c59a338ee8.parquet', 'CJHXmM/a34MDEAE=', '1705332500212625', 1, '2024-01-15 15:28:20.216000', 120897714, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '034224cec7a9362b7e8b6b10590f3806.npz', 'CPO8sLnc34MDEAE=', '1705332991336051', 1, '2024-01-15 15:36:31.342000', 3113276848, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '034224cec7a9362b7e8b6b10590f3806.parquet', 'CI7l7c3a34MDEAE=', '1705332497412750', 1, '2024-01-15 15:28:17.416000', 121314384, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0343f539d27054fb89b80cc6cd232aad.npz', 'CJKErr/c34MDEAE=', '1705333003878930', 1, '2024-01-15 15:36:43.883000', 3122419120, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '0343f539d27054fb89b80cc6cd232aad.parquet', 'CN394c3a34MDEAE=', '1705332497219293', 1, '2024-01-15 15:28:17.222000', 121828153, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '036d6b9ae87a00e738f8fc554130b65b.npz', 'CNmUivva34MDEAE=', '1705332592249433', 1, '2024-01-15 15:29:52.253000', 673764696, '', '', NULL, 'gs://dvcx-datacomp-small')
INSERT INTO "src_gs://dvcx-datacomp-small/metadata/_1" (vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) VALUES ('', 0, 'metadata', '036d6b9ae87a00e738f8fc554130b65b.parquet', 'CNG8/sTa34MDEAE=', '1705332478811729', 1, '2024-01-15 15:27:58.814000', 26585624, '', '', NULL, 'gs://dvcx-datacomp-small')
UPDATE buckets SET last_inserted_at='2024-08-01 01:50:25.697941' WHERE buckets.uri = 'gs://dvcx-datacomp-small'
begin
UPDATE buckets SET timestamp='2024-08-01 01:50:25.698381', expires='2024-08-01 05:50:25.698381', last_inserted_at='2024-08-01 01:50:25.698381', status=5, error_message='', error_stack='' WHERE buckets.uri = 'gs://dvcx-datacomp-small'
INSERT INTO prt__ba3a3313db27 (path_str, partial_id, timestamp, expires) VALUES ('metadata/', 1, '2024-08-01 01:50:25.698381', '2024-08-01 05:50:25.698381')
UPDATE datasets SET status=4, finished_at='2024-08-01 01:50:25.699250' WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
UPDATE datasets_versions SET status=4, finished_at='2024-08-01 01:50:25.699250' WHERE datasets_versions.id = 1
COMMIT
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'src_gs://dvcx-datacomp-small/metadata/_1') AS anon_1
SELECT count("src_gs://dvcx-datacomp-small/metadata/_1".sys__id) AS count_1, sum("src_gs://dvcx-datacomp-small/metadata/_1".size) AS sum_1 
FROM "src_gs://dvcx-datacomp-small/metadata/_1"
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
SELECT "src_gs://dvcx-datacomp-small/metadata/_1".sys__id, "src_gs://dvcx-datacomp-small/metadata/_1".sys__rand, "src_gs://dvcx-datacomp-small/metadata/_1".vtype, "src_gs://dvcx-datacomp-small/metadata/_1".dir_type, "src_gs://dvcx-datacomp-small/metadata/_1".parent, "src_gs://dvcx-datacomp-small/metadata/_1".name, "src_gs://dvcx-datacomp-small/metadata/_1".etag, "src_gs://dvcx-datacomp-small/metadata/_1".version, "src_gs://dvcx-datacomp-small/metadata/_1".is_latest, "src_gs://dvcx-datacomp-small/metadata/_1".last_modified, "src_gs://dvcx-datacomp-small/metadata/_1".size, "src_gs://dvcx-datacomp-small/metadata/_1".owner_name, "src_gs://dvcx-datacomp-small/metadata/_1".owner_id, "src_gs://dvcx-datacomp-small/metadata/_1".location, "src_gs://dvcx-datacomp-small/metadata/_1".source 
FROM "src_gs://dvcx-datacomp-small/metadata/_1"
 LIMIT 20 OFFSET 0
UPDATE datasets_versions SET num_objects=52, size=81707590807, preview='[{"sys__id": 1, "sys__rand": 2022854233244241774, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "0020f0cbd157d470aa56bea08e304b90.npz", "etag": "CNf/i9Pc34MDEAE=", "version": "1705333045264343", "is_latest": 1, "last_modified": "2024-01-15T15:37:25.267000+00:00", "size": 3106844080, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 2, "sys__rand": 6683210579250364514, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "0020f0cbd157d470aa56bea08e304b90.parquet", "etag": "CPXavc/a34MDEAE=", "version": "1705332500819317", "is_latest": 1, "last_modified": "2024-01-15T15:28:20.824000+00:00", "size": 120845372, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 3, "sys__rand": 7802124254091268738, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "002372587416601d1a339f7589e00276.npz", "etag": "CPXjn7vc34MDEAE=", "version": "1705332995256821", "is_latest": 1, "last_modified": "2024-01-15T15:36:35.262000+00:00", "size": 3097824688, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 4, "sys__rand": 7112348021656662780, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "002372587416601d1a339f7589e00276.parquet", "etag": "CJOZvM7a34MDEAE=", "version": "1705332498697363", "is_latest": 1, "last_modified": "2024-01-15T15:28:18.699000+00:00", "size": 120743149, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 5, "sys__rand": 3133139630757969517, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "003497f3b357f591b59fd6f20967174e.npz", "etag": "CMfY08vc34MDEAE=", "version": "1705333029661767", "is_latest": 1, "last_modified": "2024-01-15T15:37:09.667000+00:00", "size": 3140224432, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 6, "sys__rand": 8177984447700631371, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "003497f3b357f591b59fd6f20967174e.parquet", "etag": "CMeV+s3a34MDEAE=", "version": "1705332497615559", "is_latest": 1, "last_modified": "2024-01-15T15:28:17.619000+00:00", "size": 122357389, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 7, "sys__rand": 5261197093151063166, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "005e1941475026c5167278c9b564d204.npz", "etag": "CI3/osDc34MDEAE=", "version": "1705333005795213", "is_latest": 1, "last_modified": "2024-01-15T15:36:45.800000+00:00", "size": 3112410544, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 8, "sys__rand": 5501156791341205843, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "005e1941475026c5167278c9b564d204.parquet", "etag": "CPaTvs7a34MDEAE=", "version": "1705332498729462", "is_latest": 1, "last_modified": "2024-01-15T15:28:18.731000+00:00", "size": 121353140, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 9, "sys__rand": 4885584783848280221, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "006731584dd46fed36eafe8956742f7f.npz", "etag": "CN6zncLc34MDEAE=", "version": "1705333009897950", "is_latest": 1, "last_modified": "2024-01-15T15:36:49.904000+00:00", "size": 3130086832, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 10, "sys__rand": 2853087100463196308, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "006731584dd46fed36eafe8956742f7f.parquet", "etag": "CIK1/M7a34MDEAE=", "version": "1705332499749506", "is_latest": 1, "last_modified": "2024-01-15T15:28:19.752000+00:00", "size": 122071128, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 11, "sys__rand": 2686033825468278948, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "006f4f04359ef9da4fc0c76a589ef5d0.npz", "etag": "CPuTncHc34MDEAE=", "version": "1705333007796731", "is_latest": 1, "last_modified": "2024-01-15T15:36:47.802000+00:00", "size": 3132906928, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 12, "sys__rand": 862351858625947920, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "006f4f04359ef9da4fc0c76a589ef5d0.parquet", "etag": "CNn3ttDa34MDEAE=", "version": "1705332502805465", "is_latest": 1, "last_modified": "2024-01-15T15:28:22.808000+00:00", "size": 122093895, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 13, "sys__rand": 17178594339198892, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "008187fcbb0910e2f03d30fd0d497176.npz", "etag": "CMrUi8bc34MDEAE=", "version": "1705333017995850", "is_latest": 1, "last_modified": "2024-01-15T15:36:58+00:00", "size": 3126431152, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 14, "sys__rand": 3555427335807659766, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "008187fcbb0910e2f03d30fd0d497176.parquet", "etag": "CJOovc/a34MDEAE=", "version": "1705332500812819", "is_latest": 1, "last_modified": "2024-01-15T15:28:20.816000+00:00", "size": 121693996, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 15, "sys__rand": 8801407393335631314, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "008d6058981efa6c89a18ee29d53f2fc.npz", "etag": "COOcytfc34MDEAE=", "version": "1705333054672483", "is_latest": 1, "last_modified": "2024-01-15T15:37:34.677000+00:00", "size": 3120219568, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 16, "sys__rand": 3104358069122814025, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "008d6058981efa6c89a18ee29d53f2fc.parquet", "etag": "COjius3a34MDEAE=", "version": "1705332496576872", "is_latest": 1, "last_modified": "2024-01-15T15:28:16.578000+00:00", "size": 121674406, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 17, "sys__rand": 7474660517633526425, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "00a735dd2dd539edc0f4368bd13a0c37.npz", "etag": "CLiE5MTc34MDEAE=", "version": "1705333015249464", "is_latest": 1, "last_modified": "2024-01-15T15:36:55.255000+00:00", "size": 3120262576, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 18, "sys__rand": 2369440441225273130, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "00a735dd2dd539edc0f4368bd13a0c37.parquet", "etag": "CJyUgM7a34MDEAE=", "version": "1705332497713692", "is_latest": 1, "last_modified": "2024-01-15T15:28:17.716000+00:00", "size": 121662307, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 19, "sys__rand": 7068462158607143164, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "00d88deb6cc5ec92105ba44e7441e22f.npz", "etag": "CLShwsTc34MDEAE=", "version": "1705333014696116", "is_latest": 1, "last_modified": "2024-01-15T15:36:54.699000+00:00", "size": 3132550576, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}, {"sys__id": 20, "sys__rand": 3473805552252217275, "vtype": "", "dir_type": 0, "parent": "metadata", "name": "00d88deb6cc5ec92105ba44e7441e22f.parquet", "etag": "CMidw87a34MDEAE=", "version": "1705332498812616", "is_latest": 1, "last_modified": "2024-01-15T15:28:18.816000+00:00", "size": 121983445, "owner_name": "", "owner_id": "", "location": null, "source": "gs://dvcx-datacomp-small"}]' WHERE datasets_versions.id = 1
SELECT prt__ba3a3313db27.expires, prt__ba3a3313db27.partial_id, prt__ba3a3313db27.path_str 
FROM prt__ba3a3313db27 
WHERE prt__ba3a3313db27.path_str = substr('metadata/', 1, length(prt__ba3a3313db27.path_str)) ORDER BY prt__ba3a3313db27.expires DESC
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets') AS anon_1
SELECT EXISTS (SELECT 1 
FROM sqlite_master 
WHERE type = 'table' AND name = 'datasets_versions') AS anon_1
SELECT datasets.id, datasets.name, datasets.description, datasets.labels, datasets.shadow, datasets.status, datasets.feature_schema, datasets.created_at, datasets.finished_at, datasets.error_message, datasets.error_stack, datasets.script_output, datasets.sources, datasets.query_script, datasets.schema, datasets_versions.id AS id_1, datasets_versions.dataset_id, datasets_versions.version, datasets_versions.status AS status_1, datasets_versions.feature_schema AS feature_schema_1, datasets_versions.created_at AS created_at_1, datasets_versions.finished_at AS finished_at_1, datasets_versions.error_message AS error_message_1, datasets_versions.error_stack AS error_stack_1, datasets_versions.script_output AS script_output_1, datasets_versions.num_objects, datasets_versions.size, datasets_versions.preview, datasets_versions.sources AS sources_1, datasets_versions.query_script AS query_script_1, datasets_versions.schema AS schema_1, datasets_versions.job_id, datasets_versions.is_job_result 
FROM datasets LEFT OUTER JOIN datasets_versions ON datasets.id = datasets_versions.dataset_id 
WHERE datasets.name = 'gs://dvcx-datacomp-small/metadata/'
SELECT buckets.id, buckets.uri, buckets.timestamp, buckets.expires, buckets.started_inserting_at, buckets.last_inserted_at, buckets.status, buckets.error_message, buckets.error_stack 
FROM buckets 
WHERE buckets.uri = 'gs://dvcx-datacomp-small'

CREATE TABLE IF NOT EXISTS "udf_K8SDkj" (
        sys__id INTEGER NOT NULL, 
        sys__rand INTEGER, 
        vtype VARCHAR, 
        dir_type INTEGER, 
        parent VARCHAR, 
        name VARCHAR, 
        etag VARCHAR, 
        version VARCHAR, 
        is_latest BOOLEAN, 
        last_modified DATETIME, 
        size INTEGER, 
        owner_name VARCHAR, 
        owner_id VARCHAR, 
        location JSON, 
        source VARCHAR, 
        PRIMARY KEY (sys__id)
)


INSERT INTO "udf_K8SDkj" (sys__rand, vtype, dir_type, parent, name, etag, version, is_latest, last_modified, size, owner_name, owner_id, location, source) SELECT "src_gs://dvcx-datacomp-small/metadata/_1".sys__rand, "src_gs://dvcx-datacomp-small/metadata/_1".vtype, "src_gs://dvcx-datacomp-small/metadata/_1".dir_type, "src_gs://dvcx-datacomp-small/metadata/_1".parent, "src_gs://dvcx-datacomp-small/metadata/_1".name, "src_gs://dvcx-datacomp-small/metadata/_1".etag, "src_gs://dvcx-datacomp-small/metadata/_1".version, "src_gs://dvcx-datacomp-small/metadata/_1".is_latest, "src_gs://dvcx-datacomp-small/metadata/_1".last_modified, "src_gs://dvcx-datacomp-small/metadata/_1".size, "src_gs://dvcx-datacomp-small/metadata/_1".owner_name, "src_gs://dvcx-datacomp-small/metadata/_1".owner_id, "src_gs://dvcx-datacomp-small/metadata/_1".location, "src_gs://dvcx-datacomp-small/metadata/_1".source 
FROM "src_gs://dvcx-datacomp-small/metadata/_1" 
WHERE "src_gs://dvcx-datacomp-small/metadata/_1".dir_type IN (0, 5) AND "src_gs://dvcx-datacomp-small/metadata/_1".is_latest = 1 AND ((CASE WHEN ("src_gs://dvcx-datacomp-small/metadata/_1".parent = '') THEN "src_gs://dvcx-datacomp-small/metadata/_1".name ELSE "src_gs://dvcx-datacomp-small/metadata/_1".parent || '/' || "src_gs://dvcx-datacomp-small/metadata/_1".name END GLOB 'metadata') OR (CASE WHEN ("src_gs://dvcx-datacomp-small/metadata/_1".parent = '') THEN "src_gs://dvcx-datacomp-small/metadata/_1".name ELSE "src_gs://dvcx-datacomp-small/metadata/_1".parent || '/' || "src_gs://dvcx-datacomp-small/metadata/_1".name END GLOB 'metadata/*'))

CREATE TABLE IF NOT EXISTS "udf_RPWfGg" (
        sys__id INTEGER NOT NULL, 
        file__source VARCHAR, 
        file__parent VARCHAR, 
        file__name VARCHAR, 
        file__size INTEGER, 
        file__version VARCHAR, 
        file__etag VARCHAR, 
        file__is_latest BOOLEAN, 
        file__last_modified DATETIME, 
        file__location JSON, 
        file__vtype VARCHAR, 
        PRIMARY KEY (sys__id)
)


SELECT "udf_K8SDkj".sys__id, "udf_K8SDkj".sys__rand, "udf_K8SDkj".vtype, "udf_K8SDkj".dir_type, "udf_K8SDkj".parent, "udf_K8SDkj".name, "udf_K8SDkj".etag, "udf_K8SDkj".version, "udf_K8SDkj".is_latest, "udf_K8SDkj".last_modified, "udf_K8SDkj".size, "udf_K8SDkj".owner_name, "udf_K8SDkj".owner_id, "udf_K8SDkj".location, "udf_K8SDkj".source 
FROM "udf_K8SDkj" ORDER BY "udf_K8SDkj".sys__id
 LIMIT 100000 OFFSET 0
SELECT "udf_K8SDkj".sys__id, "udf_K8SDkj".sys__rand, "udf_K8SDkj".vtype, "udf_K8SDkj".dir_type, "udf_K8SDkj".parent, "udf_K8SDkj".name, "udf_K8SDkj".etag, "udf_K8SDkj".version, "udf_K8SDkj".is_latest, "udf_K8SDkj".last_modified, "udf_K8SDkj".size, "udf_K8SDkj".owner_name, "udf_K8SDkj".owner_id, "udf_K8SDkj".location, "udf_K8SDkj".source 
FROM "udf_K8SDkj" ORDER BY "udf_K8SDkj".sys__id
 LIMIT 100000 OFFSET 100000
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (1, 'gs://dvcx-datacomp-small', 'metadata', '0020f0cbd157d470aa56bea08e304b90.npz', 3106844080, '1705333045264343', 'CNf/i9Pc34MDEAE=', 1, '2024-01-15 15:37:25.267000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (2, 'gs://dvcx-datacomp-small', 'metadata', '0020f0cbd157d470aa56bea08e304b90.parquet', 120845372, '1705332500819317', 'CPXavc/a34MDEAE=', 1, '2024-01-15 15:28:20.824000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (3, 'gs://dvcx-datacomp-small', 'metadata', '002372587416601d1a339f7589e00276.npz', 3097824688, '1705332995256821', 'CPXjn7vc34MDEAE=', 1, '2024-01-15 15:36:35.262000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (4, 'gs://dvcx-datacomp-small', 'metadata', '002372587416601d1a339f7589e00276.parquet', 120743149, '1705332498697363', 'CJOZvM7a34MDEAE=', 1, '2024-01-15 15:28:18.699000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (5, 'gs://dvcx-datacomp-small', 'metadata', '003497f3b357f591b59fd6f20967174e.npz', 3140224432, '1705333029661767', 'CMfY08vc34MDEAE=', 1, '2024-01-15 15:37:09.667000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (6, 'gs://dvcx-datacomp-small', 'metadata', '003497f3b357f591b59fd6f20967174e.parquet', 122357389, '1705332497615559', 'CMeV+s3a34MDEAE=', 1, '2024-01-15 15:28:17.619000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (7, 'gs://dvcx-datacomp-small', 'metadata', '005e1941475026c5167278c9b564d204.npz', 3112410544, '1705333005795213', 'CI3/osDc34MDEAE=', 1, '2024-01-15 15:36:45.800000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (8, 'gs://dvcx-datacomp-small', 'metadata', '005e1941475026c5167278c9b564d204.parquet', 121353140, '1705332498729462', 'CPaTvs7a34MDEAE=', 1, '2024-01-15 15:28:18.731000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (9, 'gs://dvcx-datacomp-small', 'metadata', '006731584dd46fed36eafe8956742f7f.npz', 3130086832, '1705333009897950', 'CN6zncLc34MDEAE=', 1, '2024-01-15 15:36:49.904000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (10, 'gs://dvcx-datacomp-small', 'metadata', '006731584dd46fed36eafe8956742f7f.parquet', 122071128, '1705332499749506', 'CIK1/M7a34MDEAE=', 1, '2024-01-15 15:28:19.752000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (11, 'gs://dvcx-datacomp-small', 'metadata', '006f4f04359ef9da4fc0c76a589ef5d0.npz', 3132906928, '1705333007796731', 'CPuTncHc34MDEAE=', 1, '2024-01-15 15:36:47.802000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (12, 'gs://dvcx-datacomp-small', 'metadata', '006f4f04359ef9da4fc0c76a589ef5d0.parquet', 122093895, '1705332502805465', 'CNn3ttDa34MDEAE=', 1, '2024-01-15 15:28:22.808000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (13, 'gs://dvcx-datacomp-small', 'metadata', '008187fcbb0910e2f03d30fd0d497176.npz', 3126431152, '1705333017995850', 'CMrUi8bc34MDEAE=', 1, '2024-01-15 15:36:58', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (14, 'gs://dvcx-datacomp-small', 'metadata', '008187fcbb0910e2f03d30fd0d497176.parquet', 121693996, '1705332500812819', 'CJOovc/a34MDEAE=', 1, '2024-01-15 15:28:20.816000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (15, 'gs://dvcx-datacomp-small', 'metadata', '008d6058981efa6c89a18ee29d53f2fc.npz', 3120219568, '1705333054672483', 'COOcytfc34MDEAE=', 1, '2024-01-15 15:37:34.677000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (16, 'gs://dvcx-datacomp-small', 'metadata', '008d6058981efa6c89a18ee29d53f2fc.parquet', 121674406, '1705332496576872', 'COjius3a34MDEAE=', 1, '2024-01-15 15:28:16.578000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (17, 'gs://dvcx-datacomp-small', 'metadata', '00a735dd2dd539edc0f4368bd13a0c37.npz', 3120262576, '1705333015249464', 'CLiE5MTc34MDEAE=', 1, '2024-01-15 15:36:55.255000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (18, 'gs://dvcx-datacomp-small', 'metadata', '00a735dd2dd539edc0f4368bd13a0c37.parquet', 121662307, '1705332497713692', 'CJyUgM7a34MDEAE=', 1, '2024-01-15 15:28:17.716000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (19, 'gs://dvcx-datacomp-small', 'metadata', '00d88deb6cc5ec92105ba44e7441e22f.npz', 3132550576, '1705333014696116', 'CLShwsTc34MDEAE=', 1, '2024-01-15 15:36:54.699000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (20, 'gs://dvcx-datacomp-small', 'metadata', '00d88deb6cc5ec92105ba44e7441e22f.parquet', 121983445, '1705332498812616', 'CMidw87a34MDEAE=', 1, '2024-01-15 15:28:18.816000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (21, 'gs://dvcx-datacomp-small', 'metadata', '00d9c454e77e6af9a03e0fdd78f2809a.npz', 3117491632, '1705332996773433', 'CLms/Lvc34MDEAE=', 1, '2024-01-15 15:36:36.778000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (22, 'gs://dvcx-datacomp-small', 'metadata', '00d9c454e77e6af9a03e0fdd78f2809a.parquet', 121418799, '1705332497620706', 'COK9+s3a34MDEAE=', 1, '2024-01-15 15:28:17.623000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (23, 'gs://dvcx-datacomp-small', 'metadata', '01381e342c2964e0ef8c79898f2938c6.npz', 3134731696, '1705333007011242', 'CKqb7cDc34MDEAE=', 1, '2024-01-15 15:36:47.017000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (24, 'gs://dvcx-datacomp-small', 'metadata', '01381e342c2964e0ef8c79898f2938c6.parquet', 122126284, '1705332504850578', 'CJLhs9Ha34MDEAE=', 1, '2024-01-15 15:28:24.853000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (25, 'gs://dvcx-datacomp-small', 'metadata', '01abd55893ee680bcdb2bc7322f3629e.npz', 3122609584, '1705333006049344', 'CMDAssDc34MDEAE=', 1, '2024-01-15 15:36:46.052000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (26, 'gs://dvcx-datacomp-small', 'metadata', '01abd55893ee680bcdb2bc7322f3629e.parquet', 121739730, '1705332498411190', 'CLbdqs7a34MDEAE=', 1, '2024-01-15 15:28:18.414000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (27, 'gs://dvcx-datacomp-small', 'metadata', '01d8289bd0868520f5d94431ae6ccfed.npz', 3106033072, '1705333000207593', 'COn5zb3c34MDEAE=', 1, '2024-01-15 15:36:40.212000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (28, 'gs://dvcx-datacomp-small', 'metadata', '01d8289bd0868520f5d94431ae6ccfed.parquet', 120940243, '1705332500449990', 'CMaVp8/a34MDEAE=', 1, '2024-01-15 15:28:20.453000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (29, 'gs://dvcx-datacomp-small', 'metadata', '0239539d00c512673efa3efb7d9f4957.npz', 3108822448, '1705333026574317', 'CO2fl8rc34MDEAE=', 1, '2024-01-15 15:37:06.577000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (30, 'gs://dvcx-datacomp-small', 'metadata', '0239539d00c512673efa3efb7d9f4957.parquet', 121060510, '1705332496744495', 'CK+Axc3a34MDEAE=', 1, '2024-01-15 15:28:16.747000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (31, 'gs://dvcx-datacomp-small', 'metadata', '027b4ef011d33804135ca19cf7cf12f6.npz', 3130670512, '1705332997037675', 'COu8jLzc34MDEAE=', 1, '2024-01-15 15:36:37.042000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (32, 'gs://dvcx-datacomp-small', 'metadata', '027b4ef011d33804135ca19cf7cf12f6.parquet', 121856148, '1705332509615476', 'CPTK1tPa34MDEAE=', 1, '2024-01-15 15:28:29.617000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (33, 'gs://dvcx-datacomp-small', 'metadata', '02a0fc6738dfb212ef9e2a0ac43e2df2.npz', 3106469296, '1705333032738802', 'CPK/j83c34MDEAE=', 1, '2024-01-15 15:37:12.744000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (34, 'gs://dvcx-datacomp-small', 'metadata', '02a0fc6738dfb212ef9e2a0ac43e2df2.parquet', 121093891, '1705332497511705', 'CJnq883a34MDEAE=', 1, '2024-01-15 15:28:17.517000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (35, 'gs://dvcx-datacomp-small', 'metadata', '02a82399895757e6f8eff9ecd7d6d610.npz', 3122990512, '1705333020877953', 'CIHJu8fc34MDEAE=', 1, '2024-01-15 15:37:00.883000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (36, 'gs://dvcx-datacomp-small', 'metadata', '02a82399895757e6f8eff9ecd7d6d610.parquet', 121693428, '1705332500454737', 'CNG6p8/a34MDEAE=', 1, '2024-01-15 15:28:20.457000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (37, 'gs://dvcx-datacomp-small', 'metadata', '02ad5a8f481c6a0a71fec68556d0d18a.npz', 3103194544, '1705333006206088', 'CIiJvMDc34MDEAE=', 1, '2024-01-15 15:36:46.211000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (38, 'gs://dvcx-datacomp-small', 'metadata', '02ad5a8f481c6a0a71fec68556d0d18a.parquet', 120885156, '1705332499518035', 'CNOk7s7a34MDEAE=', 1, '2024-01-15 15:28:19.520000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (39, 'gs://dvcx-datacomp-small', 'metadata', '02e00078ce738fdc5393f73a9d89fa70.npz', 3123525040, '1705333004476617', 'CMnB0r/c34MDEAE=', 1, '2024-01-15 15:36:44.481000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (40, 'gs://dvcx-datacomp-small', 'metadata', '02e00078ce738fdc5393f73a9d89fa70.parquet', 121834190, '1705332501216347', 'CNv41c/a34MDEAE=', 1, '2024-01-15 15:28:21.219000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (41, 'gs://dvcx-datacomp-small', 'metadata', '02e27cb3e4bcb4c4553e8a4fb94bc873.npz', 3123482032, '1705333071966892', 'CKzl6d/c34MDEAE=', 1, '2024-01-15 15:37:51.971000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (42, 'gs://dvcx-datacomp-small', 'metadata', '02e27cb3e4bcb4c4553e8a4fb94bc873.parquet', 121580016, '1705332498782643', 'CLOzwc7a34MDEAE=', 1, '2024-01-15 15:28:18.785000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (43, 'gs://dvcx-datacomp-small', 'metadata', '02f4d9e757155f708289bf259e108ea5.npz', 3108687280, '1705332999774431', 'CN/Bs73c34MDEAE=', 1, '2024-01-15 15:36:39.779000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (44, 'gs://dvcx-datacomp-small', 'metadata', '02f4d9e757155f708289bf259e108ea5.parquet', 121020542, '1705332500513557', 'CJWGq8/a34MDEAE=', 1, '2024-01-15 15:28:20.516000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (45, 'gs://dvcx-datacomp-small', 'metadata', '02fff3c8c49056affd52e2c59a338ee8.npz', 3105308080, '1705333013705642', 'CKrnhcTc34MDEAE=', 1, '2024-01-15 15:36:53.710000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (46, 'gs://dvcx-datacomp-small', 'metadata', '02fff3c8c49056affd52e2c59a338ee8.parquet', 120897714, '1705332500212625', 'CJHXmM/a34MDEAE=', 1, '2024-01-15 15:28:20.216000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (47, 'gs://dvcx-datacomp-small', 'metadata', '034224cec7a9362b7e8b6b10590f3806.npz', 3113276848, '1705332991336051', 'CPO8sLnc34MDEAE=', 1, '2024-01-15 15:36:31.342000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (48, 'gs://dvcx-datacomp-small', 'metadata', '034224cec7a9362b7e8b6b10590f3806.parquet', 121314384, '1705332497412750', 'CI7l7c3a34MDEAE=', 1, '2024-01-15 15:28:17.416000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (49, 'gs://dvcx-datacomp-small', 'metadata', '0343f539d27054fb89b80cc6cd232aad.npz', 3122419120, '1705333003878930', 'CJKErr/c34MDEAE=', 1, '2024-01-15 15:36:43.883000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (50, 'gs://dvcx-datacomp-small', 'metadata', '0343f539d27054fb89b80cc6cd232aad.parquet', 121828153, '1705332497219293', 'CN394c3a34MDEAE=', 1, '2024-01-15 15:28:17.222000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (51, 'gs://dvcx-datacomp-small', 'metadata', '036d6b9ae87a00e738f8fc554130b65b.npz', 673764696, '1705332592249433', 'CNmUivva34MDEAE=', 1, '2024-01-15 15:29:52.253000', NULL, '')
INSERT INTO "udf_RPWfGg" (sys__id, file__source, file__parent, file__name, file__size, file__version, file__etag, file__is_latest, file__last_modified, file__location, file__vtype) VALUES (52, 'gs://dvcx-datacomp-small', 'metadata', '036d6b9ae87a00e738f8fc554130b65b.parquet', 26585624, '1705332478811729', 'CNG8/sTa34MDEAE=', 1, '2024-01-15 15:27:58.814000', NULL, '')
SELECT anon_1.file__source, anon_1.file__parent, anon_1.file__name, anon_1.file__size, anon_1.file__version, anon_1.file__etag, anon_1.file__is_latest, anon_1.file__last_modified, anon_1.file__location, anon_1.file__vtype 
FROM (SELECT anon_2.sys__id AS sys__id, anon_2.sys__rand AS sys__rand, anon_2.file__source AS file__source, anon_2.file__parent AS file__parent, anon_2.file__name AS file__name, anon_2.file__size AS file__size, anon_2.file__version AS file__version, anon_2.file__etag AS file__etag, anon_2.file__is_latest AS file__is_latest, anon_2.file__last_modified AS file__last_modified, anon_2.file__location AS file__location, anon_2.file__vtype AS file__vtype 
FROM (SELECT anon_3.sys__id AS sys__id, anon_3.sys__rand AS sys__rand, anon_3.vtype AS vtype, anon_3.dir_type AS dir_type, anon_3.parent AS parent, anon_3.name AS name, anon_3.etag AS etag, anon_3.version AS version, anon_3.is_latest AS is_latest, anon_3.last_modified AS last_modified, anon_3.size AS size, anon_3.owner_name AS owner_name, anon_3.owner_id AS owner_id, anon_3.location AS location, anon_3.source AS source, "udf_RPWfGg".file__source AS file__source, "udf_RPWfGg".file__parent AS file__parent, "udf_RPWfGg".file__name AS file__name, "udf_RPWfGg".file__size AS file__size, "udf_RPWfGg".file__version AS file__version, "udf_RPWfGg".file__etag AS file__etag, "udf_RPWfGg".file__is_latest AS file__is_latest, "udf_RPWfGg".file__last_modified AS file__last_modified, "udf_RPWfGg".file__location AS file__location, "udf_RPWfGg".file__vtype AS file__vtype 
FROM (SELECT "udf_K8SDkj".sys__id AS sys__id, "udf_K8SDkj".sys__rand AS sys__rand, "udf_K8SDkj".vtype AS vtype, "udf_K8SDkj".dir_type AS dir_type, "udf_K8SDkj".parent AS parent, "udf_K8SDkj".name AS name, "udf_K8SDkj".etag AS etag, "udf_K8SDkj".version AS version, "udf_K8SDkj".is_latest AS is_latest, "udf_K8SDkj".last_modified AS last_modified, "udf_K8SDkj".size AS size, "udf_K8SDkj".owner_name AS owner_name, "udf_K8SDkj".owner_id AS owner_id, "udf_K8SDkj".location AS location, "udf_K8SDkj".source AS source 
FROM "udf_K8SDkj") AS anon_3 LEFT OUTER JOIN "udf_RPWfGg" ON "udf_RPWfGg".sys__id = anon_3.sys__id) AS anon_2 
WHERE name GLOB '002*.npz'
 LIMIT 20 OFFSET 0) AS anon_1


CREATE TABLE IF NOT EXISTS buckets (
        id INTEGER NOT NULL, 
        uri TEXT NOT NULL, 
        timestamp DATETIME, 
        expires DATETIME, 
        started_inserting_at DATETIME, 
        last_inserted_at DATETIME, 
        status INTEGER NOT NULL, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (uri)
)



CREATE TABLE IF NOT EXISTS datasets (
        id INTEGER NOT NULL, 
        name TEXT NOT NULL, 
        description TEXT, 
        labels JSON, 
        shadow BOOLEAN NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        PRIMARY KEY (id), 
        UNIQUE (name)
)



CREATE TABLE IF NOT EXISTS datasets_versions (
        id INTEGER NOT NULL, 
        dataset_id INTEGER NOT NULL, 
        version INTEGER NOT NULL, 
        status INTEGER NOT NULL, 
        feature_schema JSON, 
        created_at DATETIME, 
        finished_at DATETIME, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        script_output TEXT NOT NULL, 
        num_objects BIGINT, 
        size BIGINT, 
        preview JSON, 
        sources TEXT NOT NULL, 
        query_script TEXT NOT NULL, 
        schema JSON, 
        job_id TEXT, 
        is_job_result BOOLEAN NOT NULL, 
        PRIMARY KEY (id), 
        UNIQUE (dataset_id, version), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id) ON DELETE CASCADE
)



CREATE TABLE IF NOT EXISTS datasets_dependencies (
        id INTEGER NOT NULL, 
        source_dataset_id INTEGER NOT NULL, 
        source_dataset_version_id INTEGER, 
        dataset_id INTEGER, 
        dataset_version_id INTEGER, 
        bucket_id INTEGER, 
        bucket_version TEXT, 
        PRIMARY KEY (id), 
        FOREIGN KEY(source_dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(source_dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(dataset_id) REFERENCES datasets (id), 
        FOREIGN KEY(dataset_version_id) REFERENCES datasets_versions (id), 
        FOREIGN KEY(bucket_id) REFERENCES buckets (id)
)


CREATE TABLE IF NOT EXISTS jobs (
        id TEXT NOT NULL, 
        name TEXT NOT NULL, 
        status INTEGER NOT NULL, 
        created_at DATETIME NOT NULL, 
        finished_at DATETIME, 
        "query" TEXT NOT NULL, 
        query_type INTEGER NOT NULL, 
        workers INTEGER NOT NULL, 
        python_version TEXT, 
        error_message TEXT NOT NULL, 
        error_stack TEXT NOT NULL, 
        params JSON NOT NULL, 
        metrics JSON NOT NULL, 
        PRIMARY KEY (id)
)


DELETE FROM id_generator WHERE id_generator.uri IN ('udf_K8SDkj', 'udf_RPWfGg')

DROP TABLE IF EXISTS "udf_K8SDkj"

DROP TABLE IF EXISTS "udf_RPWfGg"

Is there a way to rename these columns instead of essentially creating the same table twice?

@mattseddon
Copy link
Member

Maybe we can switch .map to .mutate after #198 lands

@EdwardLi-coder EdwardLi-coder mentioned this issue Aug 6, 2024
Closed
@mattseddon mattseddon self-assigned this Aug 6, 2024
@mattseddon
Copy link
Member

@dmpetrov do we still plan on exposing the concept of indexing a bucket to users or will the only entry point be DataChain.from_storage?

@shcheklein shcheklein changed the title Flatten file signals are still working ni filters Flatten file signals are still working in filters Aug 7, 2024
@mattseddon mattseddon mentioned this issue Aug 7, 2024
Merged
@mattseddon mattseddon removed their assignment Aug 8, 2024
@dmpetrov
Copy link
Member Author

@mattseddon yes, all file commands needs to be moved to app level.
I reduced the priority to P2 since the issue seems depends on #33 and #39

@skshetry skshetry added the blocked Blocked by other tasks label Aug 21, 2024
@mattseddon
Copy link
Member

mattseddon commented Sep 17, 2024 •
edited
Loading 

from datachain import C, DataChain

(
    DataChain.from_storage("gs://dvcx-datacomp-small/metadata", anon=True)
    .filter(C("name").glob("002*.npz"))
    .show(5)
)

fails now. Closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocked Blocked by other tasks priority-p2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dmpetrov @skshetry @mattseddon