Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling of empty string to datetime conversion #2891

Merged
merged 15 commits into from
Jan 22, 2019

Conversation

srfrog
Copy link
Contributor

@srfrog srfrog commented Jan 12, 2019

This PR fixes an issue when converting empty string to datetime value.

  • When altering a schema from string to datetime that has empty strings, Dgraph fails because it doesn't know how to convert empty string values to datetime.
  • Int and float zero values are converted to zero-time value, and zero when converted back.
  • Conversions from string to datetime use zero time value.
  • New tests added for Convert cases.

New behavior

int: convert 0 to datetime -> zero time
float: convert 0 to datetime -> zero time
string: convert "" to datetime -> zero time
binary: convert '' to datetime -> zero time

datetime: convert zero time to int -> 0
datetime: convert zero time to float -> 0
datetime: convert zero-time to string -> ""
datetime: convert zero-time to binary -> ''

Closes #2310


This change is Reviewable

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int: convert 0 to datetime -> epoch
float: convert 0 to datetime -> epoch
string: convert "" to datetime -> zero time

I'd argue that they should all be zero time. Zero int, Zero float and empty string are default values for each of these types, and they should default to empty time{} struct, which is Zero time.

Looks mostly good. Except this change.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @srfrog and @manishrjain)


types/conversion.go, line 223 at r1 (raw file):

			case BinaryID:
				if vc {
					*res = []byte{1}

How about:

*res = []byte{0}
if vc { *res = []byte{1} }

Avoids the break statements. Here and below.


types/conversion_test.go, line 526 at r1 (raw file):

		// 0
		{in: Val{Tid: IntID, Value: []byte{0, 0, 0, 0, 0, 0, 0, 0}},
			out: Val{Tid: DateTimeID, Value: time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC)}},

This should be Zero time.


types/conversion_test.go, line 548 at r1 (raw file):

		// 0
		{in: Val{Tid: FloatID, Value: []byte{0, 0, 0, 0, 0, 0, 0, 0}},
			out: Val{Tid: DateTimeID, Value: time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC)}},

Should be Zero time.

Copy link
Contributor Author

@srfrog srfrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain)


types/conversion.go, line 223 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

How about:

*res = []byte{0}
if vc { *res = []byte{1} }

Avoids the break statements. Here and below.

Done.


types/conversion_test.go, line 526 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This should be Zero time.

Done.


types/conversion_test.go, line 548 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Should be Zero time.

Done.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes tests a lot harder to read. I'm not sure why was there a need to change them?

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain and @srfrog)


types/conversion_test.go, line 526 at r2 (raw file):

	}{
		// time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC)
		{in: Val{Tid: DateTimeID, Value: []byte{1, 0, 0, 0, 14, 194, 139, 231, 112, 0, 0, 0, 0, 255, 255}},

The tests before were a lot more readable. Byte arrays are impossible to understand. Also, moving time.Date to comment isn't great.

Copy link
Contributor Author

@srfrog srfrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add some spaces. Most of these tests are new or rewritten, the file was mostly outdated and commented out.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain and @srfrog)

Copy link
Contributor Author

@srfrog srfrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain)


types/conversion_test.go, line 526 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

The tests before were a lot more readable. Byte arrays are impossible to understand. Also, moving time.Date to comment isn't great.

Done.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: One comment. See if you'd like to address it before submitting.

Reviewed 1 of 2 files at r2, 1 of 1 files at r3.
Reviewable status: 2 of 5 files reviewed, 3 unresolved discussions (waiting on @srfrog)


types/conversion.go, line 75 at r3 (raw file):

			case DateTimeID:
				var t time.Time
				if !bytes.Equal(data, []byte("")) {

if len(data) > 0?


types/conversion.go, line 285 at r3 (raw file):

				} else {
					secs := t.Unix()
					if secs > math.MaxInt64 || secs < math.MinInt64 {

Not your code, but would this condition ever be true? Think never.


types/conversion_test.go, line 398 at r3 (raw file):

		{
			in:  float64(3.0),
			out: true,

I think you should be able to line in a single line, instead of splitting these over 4 lines each.

Would improve the readability considerably. Here and elsewhere.

Copy link
Contributor Author

@srfrog srfrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain)


types/conversion.go, line 75 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

if len(data) > 0?

No, data is a slice so nil would be the same as empty slice.

Ex: https://play.golang.camp/p/gQKQuP4fjIw


types/conversion.go, line 285 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Not your code, but would this condition ever be true? Think never.

Yes, the first test will fail on "2147483647-12-31 13:00:01.147483647 +0000 UTC" but then we would be dust and it wont matter. The other case will fail if we go back in time before epoch and adjust our clock :)


types/conversion_test.go, line 398 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

I think you should be able to line in a single line, instead of splitting these over 4 lines each.

Would improve the readability considerably. Here and elsewhere.

Done.

Copy link
Contributor Author

@srfrog srfrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 5 files reviewed, 3 unresolved discussions (waiting on @manishrjain)


types/conversion.go, line 285 at r3 (raw file):

Previously, srfrog (Gus) wrote…

Yes, the first test will fail on "2147483647-12-31 13:00:01.147483647 +0000 UTC" but then we would be dust and it wont matter. The other case will fail if we go back in time before epoch and adjust our clock :)

Done.

@srfrog srfrog merged commit f6a2c04 into master Jan 22, 2019
@srfrog srfrog deleted the srfrog/issue-2310_empty_datetime_parsing branch January 22, 2019 05:59
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 19, 2019
* commented unused func indexCellsForCap (linter).

* let conversion of empty string to datetime work for zero time value.
linter fixes.

* added Converter tests

* linter fixes

* removed unused func createDate and other linter fixes

* convert zero-time value to empty/zero values of other types.
code formatting changes.

* added tests for 0 int/float to convert to datatime zero-time value.
added more tests and fixed format.

* refactored tests for better legibility and support

* removed unlikely test on time value

* reformatted tests

* minor cleanups

* added binary, bool and datetime tests

* added bool to string test

* minor cleanups
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Empty datetime parsing issue
2 participants