Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: pq: date/time field value out of range: "13/01/2019 00:00:00 #31

Closed
brfk opened this issue Feb 23, 2020 · 2 comments
Closed

panic: pq: date/time field value out of range: "13/01/2019 00:00:00 #31

brfk opened this issue Feb 23, 2020 · 2 comments

Comments

@brfk
Copy link

brfk commented Feb 23, 2020

System

  • TimescaleDB 1.6
  • PostgreSQL 12.2
  • PostGIS 3
  • Windows 10 1909

I have a database named "aisdk".

I add the and postgis and timescale extensions:

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
CREATE EXTENSION IF NOT EXISTS postgis;

Then I create the following table:

CREATE TABLE aisdk ( ID SERIAL NOT NULL, Time TIMESTAMPTZ NOT NULL, MMSI INT NOT NULL, Navstatus SMALLINT NULL, Shiptype SMALLINT NULL, ROT SMALLINT NULL, SOG SMALLINT NULL, COG SMALLINT NULL, GEOM GEOGRAPHY NOT NULL, PRIMARY KEY(ID, time) );

Then I convert the table to a hypertable:
SELECT create_hypertable('aisdk', 'time');

I have 2 CSV Files (can be downloaded below)
The first one contrains dates that are less than 13/01/2019
The second one has dates larger than 13/01/2019

The files look like this:
PP_aisdk_20190101.csv

time mmsi navstatus shiptype rot sog cog geom
1 01/01/2019 00:00:00 266434000 15 25 0 SRID=4326;POINT(10.583544999999999 57.715105)
2 01/01/2019 00:00:00 636015279 0 25 0 1390 1240 SRID=4326;POINT(11.0845 54.619167000000004)
3 01/01/2019 00:00:00 273383610 0 25 0 780 2393 SRID=4326;POINT(15.74572 55.819488)

PP_aisdk_20190113.csv

time mmsi navstatus shiptype rot sog cog geom
0 13/01/2019 00:00:00 269021000 0 25 -145 1390 550 SRID=4326;POINT(8.399167 57.341667)
1 13/01/2019 00:00:00 219019804 15 25 0 1640 SRID=4326;POINT(9.954 57.5921)
2 13/01/2019 00:00:00 219005901 0 25 0 0 2865 SRID=4326;POINT(8.121438000000001 56.368356999999996)

I use the following flags with timescaledb-parallel-copy.exe

timescaledb-parallel-copy --db-name=aisdk --table=aisdk --connection postgresql://postgres:postgres@localhost:5432/aisdk?sslmode=disable --skip-header --reporting-period 1s --file PP_aisdk_20190113.csv --workers 16

The first table uploads perfectly with dates ranging from 01/01/2019 00:00:00 - 02/01/2019 23:59:59, (about 180 million rows), then I get the following error

panic: pq: date/time field value out of range: "13/01/2019 00:00:00"

I have checked my database datestyle, it is already DMY

If i use psql to upload the data like this it works:
\COPY aisdk FROM 'S://temp/csv/done/PP_aisdk_20190113.csv' DELIMITER ',' CSV HEADER;

I guess it has something do with the pq driver...it somehow thinks the datestyle is MDY and panics.

My current soloution is to change my date fromat in the csv files YYYY/MM/DD and somehow it works!

@brfk
Copy link
Author

brfk commented Feb 23, 2020

The data
PP_aisdk_201901.zip

jchampio added a commit to jchampio/timescaledb-parallel-copy that referenced this issue Jun 9, 2022
As mentioned in timescale#31, lib/pq appears to ignore the DateStyle setting and
formats timestamps with MDY order. This has been serendipitously fixed
by switching to the pgx driver; add a test to make sure it doesn't
regress again.
@jchampio
Copy link
Contributor

Should be fixed by #63; please reopen and tag me if you find otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants