Skip to content

[Bug]: Python JDBCIO failed to handle DATE fields #35244

@shunping

Description

@shunping

What happened?

Steps to reproduce:

  1. Create a table in a database (e.g. Postgres) and insert rows
 CREATE TABLE my_test (ITEM VARCHAR(25), LAST_TS DATE);
 INSERT INTO my_test (ITEM, LAST_TS) VALUES ('item1', '2025-06-15');
 INSERT INTO my_test (ITEM, LAST_TS) VALUES ('item2', '2025-06-16');
 INSERT INTO my_test (ITEM, LAST_TS) VALUES ('item3', NULL);
  1. Run the following python code
import logging
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io.jdbc import ReadFromJdbc

logging.basicConfig(level=logging.INFO)

options = PipelineOptions()

driver = 'org.postgresql.Driver'
jdbc_url = 'jdbc:postgresql://localhost:5432/<your_db>'
username = 'postgres'
password = ''


with beam.Pipeline(options=options) as p:
  _ = (
      p
      | 'Read from jdbc' >> ReadFromJdbc(
          table_name="my_test",
          driver_class_name=driver,
          jdbc_url=jdbc_url,
          username=username,
          password=password)
      )
      | beam.Map(print))

The following error is raised when running the pipeline.

  File "../sdks/python/apache_beam/typehints/schemas.py", line 926, in to_language_type
    return Timestamp(seconds=int(value.seconds), micros=int(value.micros))
                             ^^^^^^^^^^^^^^^^^^
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

There is a workaround to put the following lines before the "with" statement.

from apache_beam.coders.row_coder import LogicalType
from apache_beam.typehints.schemas import MillisInstant
LogicalType.register_logical_type(MillisInstant)

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions