Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Progress bar missing when reading 46 GB of GDB data #454

Open
marklit opened this issue Nov 14, 2024 · 0 comments
Open

Progress bar missing when reading 46 GB of GDB data #454

marklit opened this issue Nov 14, 2024 · 0 comments

Comments

@marklit
Copy link

marklit commented Nov 14, 2024

The ZIP is 8 GB and decompresses into ~50 files totalling 46 GB uncompressed. There is no progress bar at all. Even when trying to just count records.

$ wget -c https://nationaladdressdata.s3.amazonaws.com/NAD_r17.zip
$ unzip NAD_r17.zip
$ ~/duckdb # v1.1.3 19864453f7
COPY (
    SELECT   * EXCLUDE(AddrPoint,
                       DateUpdate,
                       Longitude,
                       Latitude,
                       Shape),
             DateUpdate::TIMESTAMP         AS DateUpdate,
             ST_POINT(Longitude, Latitude) AS geom
    FROM     ST_READ('NAD_r17.gdb/a00000009.gdbtable')
    ORDER BY HILBERT_ENCODE([Longitude,
                             Latitude]::DOUBLE[2])
) TO 'NAD_r17.pq' (
  FORMAT            'PARQUET',
  CODEC             'ZSTD',
  COMPRESSION_LEVEL 22,
  ROW_GROUP_SIZE    15000);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant