Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog #1

phillipleblanc · 2025-01-27T14:04:24Z

Implements basic support for connecting to the AWS Glue Iceberg REST catalog by supporting the parameters rest.sigv4-enabled, rest.signing-name and rest.signing-region. See the corresponding parameters in the PyIceberg docs: https://py.iceberg.apache.org/configuration/#rest-catalog

I've added a new feature to the iceberg-catalog-rest called sigv4 which when enabled, will recognize the above configuration parameters. I've left this disabled by default.

I added the crate request_middleware to allow running middleware on requests, and if the rest.sigv4-enabled configuration is enabled, the middleware to sign the request and add the correct Authorization header is enabled.

I've tested this implementation in my project, and it works great for connecting to the AWS Glue REST catalog, i.e. https://glue.<region>.amazonaws.com/iceberg

Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.28.3 to 1.28.4. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.28.3...v1.28.4) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Xuanwo <github@xuanwo.io>

…#845)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.216 to 1.0.217. - [Release notes](https://github.com/serde-rs/serde/releases) - [Commits](serde-rs/serde@v1.0.216...v1.0.217) --- updated-dependencies: - dependency-name: serde dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.10 to 0.12.11. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.12.10...v0.12.11) --- updated-dependencies: - dependency-name: reqwest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [aws-sdk-s3tables](https://github.com/awslabs/aws-sdk-rust) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/awslabs/aws-sdk-rust/releases) - [Commits](https://github.com/awslabs/aws-sdk-rust/commits) --- updated-dependencies: - dependency-name: aws-sdk-s3tables dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: xxchan <xxchan22f@gmail.com>

) Co-authored-by: ZENOTME <st810918843@gmail.com>

Co-authored-by: ZENOTME <st810918843@gmail.com>

Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.28.4 to 1.29.4. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.28.4...v1.29.4) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [aws-sdk-s3tables](https://github.com/awslabs/aws-sdk-rust) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/awslabs/aws-sdk-rust/releases) - [Commits](https://github.com/awslabs/aws-sdk-rust/commits) --- updated-dependencies: - dependency-name: aws-sdk-s3tables dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.11 to 0.12.12. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.12.11...v0.12.12) --- updated-dependencies: - dependency-name: reqwest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add Puffin FileMetadata * Fix comment locations * Put Ok(()) branch first * Use map_err * Inline err_out_of_bounds function * Use ok_or_else * Remove #[rustfmt::skip] * Rename input_fields to fields * Simplify flag parsing * Remove unnecessary reference * Make BlobMetadata.length a u64 (instead of usize) * Replace from with as

Co-authored-by: ZENOTME <st810918843@gmail.com>

Resolves apache#811 Depends on apache#820

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

apache#905) Co-authored-by: ZENOTME <st810918843@gmail.com>

This pr supports scanning nested data type(struct, list, map), but deeply nested data types are still not supported. --------- Co-authored-by: ZENOTME <st810918843@gmail.com>

…pache#909)

…<real>)` in some tests (apache#910) Replace `assert!(<actual> == <real>)` by `assert_eq!(<actual>, <real>)` in some tests (apache#910)

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog

Original change: #1 Upstream PR: apache#917

dependabot bot and others added 30 commits December 23, 2024 10:30

update download link (apache#836)

0e5a3c3

refactor: Remove spawn and channel inside arrow reader (apache#806)

f33628e

Signed-off-by: Xuanwo <github@xuanwo.io>

chore: improve and fix the rest example (apache#842)

5400bbe

feat: Bump opendal to 0.51 (apache#839)

d33d221

Signed-off-by: Xuanwo <github@xuanwo.io>

fix: support both gs and gcs schemes for google cloud storage (apache…

d51e818

…#845)

feat: Expose disable_config_load opendal GCS option (apache#847)

21d9568

fix: project_bacth to project_batch (apache#848)

11bc5b7

build: check in Cargo.lock (apache#851)

8883b5e

ci: use officail rustsec/audit-check action (apache#843)

dbfb45e

feat: add s3tables catalog (apache#807)

9f286a0

fix(catalog): fix sql catalog drop table (apache#853)

e5fffe4

ci: add rust-cache action (apache#844)

91980aa

chore: Fix cargo.lock not updated (apache#855)

9b5b2b4

fix(catalog): delete metadata file when droping table in MemoryCatalog

f839b0a

Add orbstack guide (apache#856)

044750f

feat: support metadata table "snapshots" (apache#822)

328e18e

feat: Support metadata table "Manifests" (apache#861)

2fb9808

Co-authored-by: xxchan <xxchan22f@gmail.com>

feat: support serialize/deserialize DataFile into avro bytes (apache#797

09fa1fa

) Co-authored-by: ZENOTME <st810918843@gmail.com>

remove registry mirror recommendations (apache#866)

63fa0c8

fix validate_identifier_ids (apache#864)

e0bd206

Co-authored-by: ZENOTME <st810918843@gmail.com>

chore: datafusion 44 upgrade (apache#867)

974d9de

fix: parse var len of decimal for parquet statistic (apache#837)

1c632b8

Co-authored-by: ZENOTME <st810918843@gmail.com>

ci: use taiki-e/install-action to install tools from binary (apache#852)

e5bdbfc

fqaiser94 and others added 23 commits January 14, 2025 10:14

ci: check MSRV correctly (apache#849)

ae04c8a

fix: spark version in integration_tests (apache#894)

425efc8

refine: refine interface of ManifestWriter (apache#738)

55cca03

Co-authored-by: ZENOTME <st810918843@gmail.com>

Datafusion: Support cast operations (apache#821)

cc73766

Resolves apache#811 Depends on apache#820

chore(deps): Bump opendal from 0.51.0 to 0.51.1 (apache#898)

6d31642

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(deps): Bump async-trait from 0.1.84 to 0.1.85 (apache#897)

3592a91

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(deps): Bump arrow-schema from 53.3.0 to 53.4.0 (apache#900)

07f7451

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

chore(deps): Bump aws-sdk-s3tables from 1.3.0 to 1.4.0 (apache#899)

1028df8

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix timesmtap_ns serde name and add test case for primitive type serde (

985c8d9

apache#905) Co-authored-by: ZENOTME <st810918843@gmail.com>

add python version support range to pyproject.toml (apache#903)

fb46f5a

feat: support scan nested type(struct, map, list) (apache#882)

efca9f0

This pr supports scanning nested data type(struct, list, map), but deeply nested data types are still not supported. --------- Co-authored-by: ZENOTME <st810918843@gmail.com>

fix: Sort Order ID in TableMetadataBuilder changes should be updated (a…

2688e50

…pache#909)

Add pyspark DataFusion integration test (apache#850)

ce068db

test: replace assert!(<actual> == <real>) by `assert_eq!(<actual>, …

31dded6

…<real>)` in some tests (apache#910) Replace `assert!(<actual> == <real>)` by `assert_eq!(<actual>, <real>)` in some tests (apache#910)

refactor: fix a type in manifest_entries field name (apache#911)

ee4e063

chore(deps): Bump aws-sdk-s3tables from 1.4.0 to 1.6.0 (apache#912)

a9e0684

chore(deps): Bump uuid from 1.12.0 to 1.12.1 (apache#913)

430498d

Implement AWS Sigv4 support for Iceberg REST catalogs on Glue

94e22f7

loosen trait requirement

6369459

loosen arrow req

d166e8a

loosen aws req

241aac2

drop df dep

d0ac376

phillipleblanc merged commit 07940c3 into spiceai-0.4.0 Jan 27, 2025
3 checks passed

phillipleblanc deleted the phillip/250127-rest-sigv4 branch January 27, 2025 14:04

phillipleblanc added a commit that referenced this pull request Apr 8, 2025

Merge pull request #1 from spiceai/phillip/250127-rest-sigv4

562ca05

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog

sgrebnov added the needs_upstream label Sep 25, 2025

sgrebnov pushed a commit that referenced this pull request Sep 25, 2025

Implement AWS Sigv4 support for Iceberg REST catalogs on Glue

d144694

Original change: #1 Upstream PR: apache#917

sgrebnov assigned phillipleblanc Sep 25, 2025

peasee pushed a commit that referenced this pull request Oct 27, 2025

Implement AWS Sigv4 support for Iceberg REST catalogs on Glue

115b764

Original change: #1 Upstream PR: apache#917

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog #1

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog #1

Uh oh!

phillipleblanc commented Jan 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog #1

Implement basic support for rest.sigv4_enabled for the Iceberg REST catalog #1

Uh oh!

Conversation

phillipleblanc commented Jan 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants