PostgresHook is ignoring custom adapters registered for dicts and lists #50976
Description
Apache Airflow Provider(s)
postgres
Versions of Apache Airflow Providers
apache-airflow-providers-postgres==6.1.3
Apache Airflow version
2.11.0
Operating System
Debian GNU/Linux 12 (bookworm)
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
The changes introduced by #49120 made PostgresHook ignore custom adapters registered via psycopg2.extensions.register_adapter for the dict and list types.
Code that relied on custom adapters for dicts and lists is now broken because PostgresHook._serialize_cell is changing their type from dict or list to Json, making psycopg2 ignore the custom adapters registered for dicts and lists.
Automatically using Json to adapt list is a specially bad case because it prevents users from adapting python's list[int] or list[str] to postgres' int[] or text[].
The default Json adapter is also often times insufficient, because it doesn't support uuid.UUIDs, decimal.Decimals and datetime.datetimes, for example.
What you think should happen instead
I think PostgresHook should not introduce a special case when serialing dict and list values and it should respect the custom adapters registered using the register_adapter mechanism provided by psycopg2.
#49120 should be reverted and code that relied on it should use the code below to replicate the desired behaviour.
import psycopg2
from psycopg2.extras import Json
psycopg2.extensions.register_adapter(list, Json)
psycopg2.extensions.register_adapter(dict, Json)How to reproduce
Using apache-airflow-providers-postgres==6.1.3, register a custom adapter for dict or list using psycopg2.extensions.register_adapter and try to run a query with PostgresHook.insert_rows which adapts a dict or list. The adapter will never be called.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct