sql: refactor dates to be fully PG compatible #36938

maddyblue · 2019-04-18T20:13:47Z

Previously dates were an int64 number of days from 1970-01-01. Functions
and operators needing to act on a date could manipulate the value of
the date without any overflow checks, causing multiple bugs:

The postgres binary wire protocol uses an int32 number of days since
2000-01-01, thus it was possible to have dates in cockroach outside
of the expressible range over the wire. We were blindly converting the
int64 to an int32, possibly discarding data.
Adding dates to ints was done by normal addition. However this could
overflow. Our normal int + int path uses overflow-checked addition,
but dates were not subject to such checks.
Converting dates to a text representation (for sending over pgwire or
our distsql serialization) converted the number of days to a time.Time
using the standard library time.Date method, which (more-or-less) stores
time as an int64 number of nanos. A very high year could overflow the
nanosecond part causing it to silently by incorrect.
The infinity dates were not treated as infinity, but converted to an
actual date.

This change adds a new pgdate.Date type. It internally represented as
an int32 number of days since 2000-01-01, because that's exactly what
is needed by the postgres wire protocol. It doesn't export any internal
properties, and instead forces users of it to use its API, which is
guaranteed to be safe. The API does all overflow and bounds checking,
and correctly handles the infinity dates. This fixes the above bugs
since it's now not possible to mistakenly misuse Dates.

When reading from any existing on-disk data, dates outside of the postgres
bounds are converted to +/- infinity. This presents a problem since
datums need to roundtrip to the same on-disk encoding so that old index
entries can be deleted. In order to round trip correctly, the original
on-disk integer is saved and used during serialization if it exists.

Fixes #36557

Release note (sql change): Dates are now fully Postgres-compatible,
including support for sentinel values (+/- infinity) and the Postgres
date range (4714-11-24 BC to 5874897-12-31). Existing dates outside of
this range will be converted to the +/- infinity dates.

cockroach-teamcity · 2019-04-18T20:13:55Z

This change is

yuzefovich

This is great stuff! The API is very clean!

And I also like the compatibility approach that you and Bram came up with. Just wondering: do we plan to ever remove orig fields to reduce the space usage? I understand that it'll break the compatibility with already existing indices, but maybe like in 2 or 3 major releases?

Reviewed 29 of 29 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @knz, and @mjibson)

pkg/sql/logictest/testdata/logic_test/datetime, line 143 at r1 (raw file):


query TTTT
SELECT 'infinity'::date + 1, 'infinity'::date - 1, '-infinity'::date + 1, '-infinity'::date - 1

[nit]: this query overlaps with the one a few above, possibly combine them.

pkg/sql/logictest/testdata/logic_test/datetime, line 1281 at r1 (raw file):

subtest regression_36146

statement error out of range

Why does it have a different error message?

pkg/sql/logictest/testdata/logic_test/pg_catalog, line 853 at r1 (raw file):

1042  bpchar        1307062959    NULL      -1      false     b
1043  varchar       1307062959    NULL      -1      false     b
1082  date          1307062959    NULL      16      true      b

[question]: just to confirm my understanding, days take up 4 bytes, orig another 8, and hasOrig yet another 4 bytes, correct?

pkg/util/timeutil/pgdate/pgdate.go, line 18 at r1 (raw file):

// and times in a manner that is compatible with PostgreSQL.
//
// The implementation here is inspired by the following

[nit]: possibly this comment needs adjusting.

pkg/util/timeutil/pgdate/pgdate.go, line 69 at r1 (raw file):

	// instead of days when encoding for on-disk. This is required
	// so that we can roundtrip to the same on-disk value, which is
	// necessary deleting old index entries. This value should never

[nit]: necessary for deleting.

pkg/util/timeutil/pgdate/pgdate.go, line 138 at r1 (raw file):

}

func makeDateFromYMD(year, month, day int) Date {

I can't seem to find where this method is used. Also, is panicking here unavoidable? Maybe return an error instead?

pkg/util/timeutil/pgdate/pgdate.go, line 164 at r1 (raw file):

	secondsPerHour  = 60 * 60
	secondsPerDay   = 24 * secondsPerHour
	daysPer400Years = 365*400 + 97

Also I can't seem to find where these daysPer... are used.

pkg/util/timeutil/pgdate/pgdate.go, line 176 at r1 (raw file):

	case math.MaxInt32:
		buf.WriteString("infinity")
	default:

Do we need a special case for epoch here?

pkg/util/timeutil/pgdate/pgdate.go, line 262 at r1 (raw file):

	if !ok {
		return Date{}, pgerror.NewErrorf(pgerror.CodeDatetimeFieldOverflowError,
			"%s + %d is out of range", d, days)

[nit]: %s - %d.

maddyblue

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @knz, and @yuzefovich)

pkg/sql/logictest/testdata/logic_test/datetime, line 143 at r1 (raw file):