Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: table_type missing from table parameters while only loading iceberg table from Hive metaStore #1331

Closed
gitzwz opened this issue Nov 18, 2024 · 0 comments · Fixed by #1332
Closed

Error: table_type missing from table parameters while only loading iceberg table from Hive metaStore #1331

gitzwz opened this issue Nov 18, 2024 · 0 comments · Fixed by #1332
Milestone
PyIceberg 0.8.1 r...

Comments

@gitzwz
Copy link
Contributor

gitzwz commented Nov 18, 2024

Apache Iceberg version

main (development)

Please describe the bug 🐞

New problem related to #1145, if there are tables with no "table_type" in table.parameters, cli could not list tables properly but runs into an exception. Or should every hive table have "table_type" default value in table.parameters?

iceberg-python/pyiceberg/catalog/hive.py

Lines 647 to 655 in b2f0a9e

database_name = self.identifier_to_database(namespace, NoSuchNamespaceError)
with self._client as open_client:
return [
(database_name, table.tableName)
for table in open_client.get_table_objects_by_name(
dbname=database_name, tbl_names=open_client.get_all_tables(db_name=database_name)
)
if table.parameters[TABLE_TYPE].lower() == ICEBERG
]

gitzwz pushed a commit to gitzwz/iceberg-python that referenced this issue Nov 18, 2024
Ignore tables without table_type parameters while loading all iceberg…
273e4ba 
… table from Glue and Hive catalog (apache#1331)
gitzwz pushed a commit to gitzwz/iceberg-python that referenced this issue Nov 18, 2024
Ignore tables without table_type parameters while loading all iceberg…
d17de79 
… table from Glue and Hive catalog (apache#1331)
gitzwz pushed a commit to gitzwz/iceberg-python that referenced this issue Nov 18, 2024
Ignore tables without table_type parameters while loading all iceberg…
8d40312 
… table from Glue and Hive catalog (apache#1331)
@Fokko Fokko mentioned this issue Nov 19, 2024
Merged
Fokko pushed a commit that referenced this issue Nov 19, 2024
@gitzwz
Ignore tables without table_type from Glue and Hive
a66ddc0 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
@kevinjqliu kevinjqliu added this to the PyIceberg 0.8.1 release milestone Nov 21, 2024
kevinjqliu pushed a commit to kevinjqliu/iceberg-python that referenced this issue Nov 25, 2024
@gitzwz @kevinjqliu
Ignore tables without table_type from Glue and Hive
2673d29 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (apache#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
kevinjqliu added a commit that referenced this issue Nov 27, 2024
@kevinjqliu @sumanth-manchala
@gitzwz @vincenzon @bigluck @binayakd @Fokko
[release] Pyiceberg 0.8.1 (#1369)
d24e0c9 
* use the non-deprecated func (#1326)

* 0.8.0 post release steps (#1334)

* add

* fix mkdoc

* Drop upper bounds for fsspec and it's implementations (#1341)

* Drop upper bounds for fsspec and it's implementations

* Run poetry lock

* Ignore tables without `table_type` from Glue and Hive

* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>

* Replace reference of `Table.identifier` with `Table.name` (#1346)

* fix Table.name

* replace Table.identifier with Table.name

* add warning filter

* Allow leading underscore in column name used in row filter (#1358)

* Update parser.py

Allow leading underscore in column name used in row filter.

* Update test_parser.py

* Update test_parser.py

* Update test_parser.py

* Remove Python 3.13 upper bound restriction (#1355)

* Remove Python 3.13 upper bound restriction

* Fix missing poetry.lock file

* Upgrading numpy on the poetry.lock file from v1.26.0 to v1.26.4

* Improve documentation for "how to release" (#1359)

* initial update

* edits

* add gpg instructions

* verify artifacts

* add twine not

* grammar

* edits

* remove old artifacts

* update doc workflow action

* and name

* add docs on patch vs major/minor release

* fix `KeyError` raised by `add_files` when parquet file doe not have column stats (#1354)

* fix KeyError, by switching del to pop

* added unit test

* update test

* fix python 3.9 compatibility, and refactor test

* update test

* bump to 0.8.1

* Add instruction for patch release (#1373)

* add instruction for patch release

* create branch from tag

* Write `null` when there is no parent-snapshot-id (#1383)

---------

Co-authored-by: Sumanth <33193748+sumanth-manchala@users.noreply.github.com>
Co-authored-by: gitzwz <72312233+gitzwz@users.noreply.github.com>
Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
Co-authored-by: vincenzon <mvcalder@gmail.com>
Co-authored-by: Luca Bigon <luca.bigon@bauplanlabs.com>
Co-authored-by: Binayak Dasgupta <binayakd86@gmail.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
kevinjqliu pushed a commit to kevinjqliu/iceberg-python that referenced this issue Nov 27, 2024
@gitzwz @kevinjqliu
Ignore tables without table_type from Glue and Hive
1adb83c 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (apache#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
kevinjqliu pushed a commit that referenced this issue Nov 27, 2024
@gitzwz @kevinjqliu
Ignore tables without table_type from Glue and Hive
8643938 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
sungwy pushed a commit to sungwy/iceberg-python that referenced this issue Dec 7, 2024
@gitzwz @sungwy
Ignore tables without table_type from Glue and Hive
e503fe1 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (apache#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
sungwy pushed a commit to sungwy/iceberg-python that referenced this issue Dec 7, 2024
@gitzwz @sungwy
Ignore tables without table_type from Glue and Hive
6d738a2 
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (apache#1331)

* Use TABLE_TYPE

---------

Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
PyIceberg 0.8.1 release
2 participants
@kevinjqliu @gitzwz