any plan for Iceberg Table on S3?

[Lindayangyy]

New to Apache Iceberg, We are looking for Iceberg Table or warehouse (catalog) implementation upon S3, if without any reference to Hive and HDFS (hadoop) is possible? The current implementation seems tightly coupled with Hive and hadoop.

[RussellSpitzer]

You can use it with S3 with Hadoop client libraries only, you don't actually need a Hadoop cluster or HDFS.

[HeartSaVioR]

Supporting S3 requires Hive, because of S3's characteristic, eventual consistency. I see OSP version of Delta Lake solved it in different way, but pretty much limited. (It assumes concurrent writes for S3 only happen in "a" Spark driver. https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/storage/S3SingleDriverLogStore.scala)

[aokolnychyi]

Iceberg works reliably with s3 even if the same table is accessed via multiple clusters and query engines. Using Iceberg requires a catalog that can swap a pointer to the metadata file atomically. This can be done using a compare and swap or lock/unlock API. Iceberg contains a built-in implementation that uses Hive metastore to work with s3 reliably (lock/unlock). Anyone could easily build an integration for any catalog. For example, one may have a Cassandra-based catalog and use compare and swap to commit new table versions. That will be enough to work with s3 reliably.

[jacques-n]

We've been working on a non-Hive way to provide this functionality and plan on contributing it to the project within the next two weeks.

[Lindayangyy]

That will be awesome, can't wait to see it. Thank you - jacques-n!

[Lindayangyy]

Thanks for all the responses as alternatives. All answers are great!

[HeartSaVioR]

That sounds great! Assuming it still needs to do CAS with external storage (I'd be really curious if it doesn't rely on the external storage) which is that? Is it one of AWS services? If then even better, as there's no external dependency outside of AWS. Given we assume to use S3, which is already locked-in.

[jacques-n]

We're doing something pluggable but the default implementation is on top of DynamoDB.

[ismailsimsek]

is it possible to write JDBC based catalog? that could unlock many catalog option

[kbendick]

We're doing something pluggable but the default implementation is on top of DynamoDB.

That's a good idea. I know that AWS Glue is backed by DynamoDB, so if you can make a catalog using Dynamo, then possibly the AWS team can implement the atomic swap in Glue. If I'm not mistaken, you'd need to use either read / write consistency or possibly a DynamoDB versioned object.

Looking forward to seeing the DynamoDB catalog as I assume many companies looking to write to S3 are also likely using DynamoDB. I know that my company uses DynamoDB a ton so this would be a great work around until there is Glue Catalog support (which I've been giving some thought to myself).

[jackye1995]

Hi @jacques-n this is Jack from AWS. We are planning to introduce a new iceberg-aws module, and we do have plan to offer a Glue + DynamoDB implementation for Catalog and TableOperations. Since you say you already have something working, let's have a sync after you have a PR and see what is the best way to have this shipped all together 😃

[jacques-n]

Hey guys, we just posted more information on the new stuff we've been building for Iceberg + DynamoDB. You can check it out here: https://projectnessie.org/

We'll have a PR up against Iceberg shortly to contribute the Iceberg integrations:
https://github.com/projectnessie/nessie/tree/main/clients/iceberg

[RussellSpitzer]

Very cool!

…

On Thu, Oct 1, 2020 at 4:34 PM Jacques Nadeau ***@***.***> wrote: Hey guys, we just posted more information on the new stuff we've been building for Iceberg + DynamoDB. You can check it out here: https://projectnessie.org/ We'll have a PR up against Iceberg shortly to contribute the Iceberg integrations: https://github.com/projectnessie/nessie/tree/main/clients/iceberg — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1468 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADE2YKA6G5T55NR2OUSRVLSITYWXANCNFSM4RPIUBNQ> .

[jackye1995]

I just sent out a PR for AWS Glue support. With this update you can use HiveCatalog without the need to set up any Hive infrastructure and build your data lake on top of S3. #1608

[jackye1995]

For anyone new to this issue, I think we have summarized all information in https://iceberg.apache.org/aws/, and we can close this issue. @Lindayangyy