ARROW-835: [Format][C++][Java] Create a new Duration type #3644

emkornfield · 2019-02-14T11:30:52Z

Create a new DurationInterval type in format.
Implement all interval types in C++ and Java including integration test.

Once this is checked in, I think https://issues.apache.org/jira/browse/ARROW-352 can be resolved as a won't fix?

codecov-io · 2019-02-15T08:55:44Z

Codecov Report

Merging #3644 into master will decrease coverage by <.01%.
The diff coverage is 84.24%.

@@            Coverage Diff             @@
##           master    #3644      +/-   ##
==========================================
- Coverage   88.07%   88.06%   -0.01%     
==========================================
  Files         773      774       +1     
  Lines       96887    97162     +275     
  Branches     1251     1251              
==========================================
+ Hits        85329    85566     +237     
- Misses      11322    11360      +38     
  Partials      236      236

Impacted Files	Coverage Δ
cpp/src/arrow/ipc/read-write-test.cc	`99.81% <ø> (ø)`	⬆️
cpp/src/arrow/ipc/json-test.cc	`98.46% <ø> (ø)`	⬆️
cpp/src/arrow/visitor.h	`50% <ø> (ø)`	⬆️
cpp/src/arrow/scalar.h	`88.23% <ø> (ø)`	⬆️
cpp/src/arrow/array/builder_primitive.h	`95.86% <ø> (ø)`	⬆️
cpp/src/arrow/type_traits.h	`85.71% <0%> (-7.25%)`	⬇️
cpp/src/arrow/array/builder_dict.cc	`86.06% <0%> (-1.44%)`	⬇️
cpp/src/arrow/pretty_print.cc	`83.48% <0%> (-1.6%)`	⬇️
cpp/src/arrow/visitor.cc	`0% <0%> (ø)`	⬆️
cpp/src/arrow/type-test.cc	`99.21% <100%> (+0.05%)`	⬆️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ca6fe8...973d80f. Read the comment docs.

format/Schema.fbs

java/vector/src/main/java/org/apache/arrow/vector/IntervalEpochMilliVector.java

emkornfield · 2019-02-23T06:21:40Z

@wesm could you look at the format changes to see if you are ok with them?
@pravindra Can you look at the impact on the java side (I added all the necessary java classes I think) to confirm this is acceptable for the Java?

emkornfield · 2019-02-23T07:42:22Z

Failure was NodeJs

pravindra · 2019-02-25T06:20:15Z

since this is a format change (backward compatible though), I'll wait for @wesm to approve too before merging.

emkornfield · 2019-02-25T06:33:54Z

@pravindra thanks for the review

emkornfield · 2019-02-25T06:50:03Z

format/Schema.fbs

+/// millisecond, but can be any of the other supported TimeUnit values as
+/// with Timestamp and Time types.  This type is always represented as
+/// an 8-byte integer.
+table DurationInterval {


Looking over the old e-mail threads it might be that TimeDelta or TimestampDelta is a better name for this type?

Are you happy with "DurationInterval"? It could also be called simply "Duration" or "TimeDuration"

emkornfield · 2019-03-11T03:18:01Z

@wesm based on comments on the other JIRAs CLs it seems like it would be desirable to add an implementation for C++, I can add that to this CL but would you mind taking a look to see if you are ok with naming/modeling (I would prefer not to have to refactor just due to naming issues if possible).

emkornfield · 2019-04-30T23:59:32Z

I believe the Python on OS/X error is spurious (it was green on previous build).

wesm · 2019-05-03T18:02:57Z

Will work on reviewing the C++ side of this

wesm

The C++ side of this, and the changes to the Flatbuffers files, all looks good to me. Thanks @emkornfield for slogging through this and implementing C++, Java, and integration tests in one go!

One question I have is whether we want to commit to the name "DurationInterval" rather than alternatives such as "Duration" or "TimeDelta" or "TimeDuration". I don't think the names are sacred

Can someone (@siddharthteotia ?) carve out a little time to review the Java so we can try to get this merged soon?

wesm · 2019-03-27T12:45:05Z

java/vector/src/main/codegen/data/ValueVectorTypes.tdd

+        { class: "DurationIntervalMicro", javaType: "long", friendlyType: "Duration",
+          arrowType: "org.apache.arrow.vector.types.pojo.ArrowType.DurationInterval"}
+        { class: "DurationIntervalNano", javaType: "long", friendlyType: "Duration",
+          arrowType: "org.apache.arrow.vector.types.pojo.ArrowType.Duration"}


Do you think it's possible to collapse this into a single class? There was an effort to do this for Timestamp (with a TimeUnit class) but it failed only because Dremio had taken on dependency with the TimestampUNIT classes

Potentially, I followed the pattern already established in the codebase, but I'm open to trying if it is desirable. @pravindra @siddharthteotia do you think this is desirable/possible?

Sounds okay to me. Let's see what @siddharthteotia and @pravindra say.

IMO, having a single class is definitely desirable. Single this one is starting on a fresh slate, it's worth a try.

wesm · 2019-05-03T18:04:07Z

cpp/src/arrow/array.h

@@ -650,6 +650,36 @@ class ARROW_EXPORT FixedSizeBinaryArray : public PrimitiveArray {
  int32_t byte_width_;
 };

+/// DayTimeArray
+/// ---------------------
+/// \brief Array of Day and Millisecond values.


FWIW, I'm a bit disappointed that we aren't deprecating this type =/

I didn't want to get dragged into what it means deprecate this, should be easy enough to do if we want it. As noted below I tried to say it was optional which I seem to recall seeing this was ok on a previous ML thread.

wesm · 2019-05-03T18:05:25Z

cpp/src/arrow/array/builder_time.h

+using Time32Builder = NumericBuilder<Time32Type>;
+using Time64Builder = NumericBuilder<Time64Type>;
+using Date32Builder = NumericBuilder<Date32Type>;
+using Date64Builder = NumericBuilder<Date64Type>;


Is this redundant with type_fwd.h?

Yes, it appears so, I'll push a commit removing them. It looks like there might be more redundancy in builder_primitive.h

wesm · 2019-05-03T18:07:31Z

cpp/src/arrow/builder.cc

@@ -111,6 +112,18 @@ Status MakeBuilder(MemoryPool* pool, const std::shared_ptr<DataType>& type,
      DictionaryBuilderCase visitor = {pool, dict_type, out};
      return VisitTypeInline(*dict_type.dictionary()->type(), &visitor);
    }
+    case Type::INTERVAL: {
+      const auto& interval_type = dynamic_cast<const IntervalType&>(*type);


I think static_cast (or checked_cast) is sufficient here?

wesm · 2019-05-03T18:41:46Z

cpp/src/arrow/type.h

+std::shared_ptr<DataType> ARROW_EXPORT day_time_interval();
+
+/// \brief Return an MonthIntervalType instance
+std::shared_ptr<DataType> ARROW_EXPORT month_interval();


nit: as far as naming I could see an argument for using

interval_duration

interval_day_time

interval_month

but I'm OK with the way things are here

I could see this to, but I'm happier without the reverse notation.

note "duration" conflicts with some of our time classes so I renamed to "duration_type", hopefully that is ok

"some of our time classes"?

Need to look back I think it was in the vendored time library we were using. I can look deeper if you don't think there should be a conflict

looking again they shouldn't conflict, I will try to fix this today.

Awesome, thanks.

renamed. had to touch some of the vendored code.

wesm · 2019-05-03T18:44:57Z

format/Schema.fbs

+//   4-byte integers. 
+// DAY_TIME - Indicates the number of elapsed days and milliseconds,
+//   stored as 2 contiguous 32-bit integers (8-bytes in total).  Support
+//   of this IntervalUnit is not required for full arrow compatibility.


Got it. If it isn't being deprecated then it's reasonable to implement it in C++ and have integration tests

wesm · 2019-05-03T18:45:49Z

format/Schema.fbs

+/// millisecond, but can be any of the other supported TimeUnit values as
+/// with Timestamp and Time types.  This type is always represented as
+/// an 8-byte integer.
+table DurationInterval {


Are you happy with "DurationInterval"? It could also be called simply "Duration" or "TimeDuration"

emkornfield · 2019-05-03T19:52:48Z

@wesm Thanks for the review I'll try to address comments in the next few days. In terms of naming, I don't know the etiquette for changing the name once it gets approved on the ML. If was going to change it, I think Duration would be my choice.

@pravindra already reviewed most of the java code, the main changes that happened where plumbing through support for the integration test.

kou · 2019-05-05T23:59:10Z

cpp/src/arrow/array/builder_dict.cc

@@ -349,6 +355,9 @@ template class DictionaryBuilder<Date64Type>;
 template class DictionaryBuilder<Time32Type>;
 template class DictionaryBuilder<Time64Type>;
 template class DictionaryBuilder<TimestampType>;
+template class DictionaryBuilder<DurationIntervalType>;
+template class DictionaryBuilder<DayTimeIntervalType>;
+template class DictionaryBuilder<MonthIntervalType>;


Just for your information.

These changes will be conflicted with #4255.
If #4255 is merged, we will be able to just remove these changes.

Thanks, they were removed.

emkornfield · 2019-05-14T00:13:00Z

@wesm I think I addressed all your feedback. The OS/X build seems to in an infinite CMake loop with something like

  the following targets are still affected by RPATH settings:
   arrow_python_shared
   arrow_shared
   arrow_testing_shared
   parquet_shared
   plasma_shared

I'm not sure why my change would have caused this. Any thoughts?

@pravindra I was able to consolidate all the java vectors into one, could you take a look?

wesm · 2019-05-15T19:30:12Z

Baffling

wesm · 2019-05-15T19:31:29Z

It must have something to do with this being an older PR. Is Appveyor enabled on your fork? If not I can push this to my fork to get an Appveyor build going

emkornfield · 2019-05-15T19:33:15Z

No appveyor isn't enabled on my fork. let me try squashing everything, see if that helps.

and mark DayTimeInterval as deprecated. Provides implementation for Duration, DayTimeInterval and YearMonth interval in C++ and Java (for duration). Adds integration test.

emkornfield · 2019-05-15T19:35:53Z

Ahh, squash seems to have worked, lets hope things stay green.

wesm · 2019-05-15T23:26:24Z

@emkornfield it looks you made some changes to vendored code which I think we should try to avoid doing in general, can we open a JIRA to restore to the vendored version and address the name conflict another way? I don't want this issue to block the patch since I need this to go in and then rebase #4316 on top

wesm · 2019-05-15T23:29:31Z

Appveyor appears to be about ~6 hours backed up so I'm going to push a branch on my fork to try to get a quicker thumbs up to merge this

wesm · 2019-05-15T23:31:02Z

Here's an Appveyor build starting now on my fork which will take a while to run https://ci.appveyor.com/project/wesm/arrow/builds/24577591

emkornfield · 2019-05-15T23:33:44Z

Thanks. https://issues.apache.org/jira/browse/ARROW-5346 opened to revert the changes. Right now the only solution seems like reordering includes in some cases or fixing upstream. I will have to think about this.

wesm · 2019-05-16T00:25:55Z

Appveyor build looks OK, I'm going to wait a little while yet to make sure the MinGW build works before merging

emkornfield · 2019-05-16T02:14:48Z

@wesm appveyor on your branch looks all green.

wesm · 2019-05-16T02:39:27Z

Awesome. Thanks @emkornfield!!

- Create a new DurationInterval type in format. - Implement all interval types in C++ and Java including integration test. Once this is checked in, I think https://issues.apache.org/jira/browse/ARROW-352 can be resolved as a won't fix? Author: Micah Kornfield <emkornfield@gmail.com> Closes apache#3644 from emkornfield/new_type and squashes the following commits: cc6d39c <Micah Kornfield> remove system clock daf8b4f <Micah Kornfield> duration_type->duration b064816 <Micah Kornfield> Introduce a new Duration type can that represent time deltas, and mark DayTimeInterval as deprecated.

emkornfield mentioned this pull request Feb 14, 2019

ARROW-352: [Format] Add time resolution to Interval, schema documentation #920

Closed

emkornfield changed the title ~~ARROW-352: [WIP][Format] Add time resolution to Interval by creating a new type~~ ARROW-352:[Format] Add time resolution to Interval by creating a new type Feb 15, 2019

pravindra reviewed Feb 18, 2019

View reviewed changes

format/Schema.fbs Outdated Show resolved Hide resolved

java/vector/src/main/java/org/apache/arrow/vector/IntervalEpochMilliVector.java Outdated Show resolved Hide resolved

wesm force-pushed the master branch from 3088183 to 0c6b2d2 Compare February 18, 2019 19:34

emkornfield changed the title ~~ARROW-352:[Format] Add time resolution to Interval by creating a new type~~ ARROW-352:[Format][Java] Add time resolution to Interval by creating a new type Feb 23, 2019

kou changed the title ~~ARROW-352:[Format][Java] Add time resolution to Interval by creating a new type~~ ARROW-352: [Format][Java] Add time resolution to Interval by creating a new type Feb 24, 2019

pravindra approved these changes Feb 25, 2019

View reviewed changes

emkornfield commented Feb 25, 2019

View reviewed changes

emkornfield mentioned this pull request Mar 11, 2019

ARROW-4810: [Format][C++] Add "LargeList" type with 64-bit offsets #3848

Closed

emkornfield force-pushed the new_type branch from d6174bc to e50aea2 Compare April 11, 2019 08:31

emkornfield changed the title ~~ARROW-352: [Format][Java] Add time resolution to Interval by creating a new type~~ ARROW-352: [WIP][Format][C++] Add time resolution to Interval by creating a new type Apr 12, 2019

emkornfield force-pushed the new_type branch from f957c42 to 55470e9 Compare April 29, 2019 17:33

emkornfield changed the title ~~ARROW-352: [WIP][Format][C++] Add time resolution to Interval by creating a new type~~ ARROW-835: [WIP][Format][C++] Add time resolution to Interval by creating a new type Apr 29, 2019

emkornfield changed the title ~~ARROW-835: [WIP][Format][C++] Add time resolution to Interval by creating a new type~~ ARROW-835: [WIP][Format][C++][Java] Create a new DurationInterval type Apr 29, 2019

emkornfield force-pushed the new_type branch from 266bf28 to a3fbb1a Compare April 30, 2019 20:27

emkornfield changed the title ~~ARROW-835: [WIP][Format][C++][Java] Create a new DurationInterval type~~ ARROW-835: [Format][C++][Java] Create a new DurationInterval type Apr 30, 2019

wesm reviewed May 3, 2019

View reviewed changes

kou reviewed May 5, 2019

View reviewed changes

emkornfield force-pushed the new_type branch from d4e999c to b07369c Compare May 13, 2019 20:22

emkornfield changed the title ~~ARROW-835: [Format][C++][Java] Create a new DurationInterval type~~ ARROW-835: [Format][C++][Java] Create a new Duration type May 13, 2019

emkornfield force-pushed the new_type branch 3 times, most recently from 2913b09 to 773a2b0 Compare May 15, 2019 19:28

emkornfield force-pushed the new_type branch from 773a2b0 to 5a8cd38 Compare May 15, 2019 19:30

Introduce a new Duration type can that represent time deltas,

b064816

and mark DayTimeInterval as deprecated. Provides implementation for Duration, DayTimeInterval and YearMonth interval in C++ and Java (for duration). Adds integration test.

emkornfield force-pushed the new_type branch from 5a8cd38 to b064816 Compare May 15, 2019 19:35

emkornfield added 2 commits May 15, 2019 13:33

duration_type->duration

daf8b4f

remove system clock

cc6d39c

wesm mentioned this pull request May 15, 2019

ARROW-3144: [C++/Python] Move "dictionary" member from DictionaryType to ArrayData to allow for variable dictionaries #4316

Closed

wesm closed this in 6f80ea4 May 16, 2019

emkornfield deleted the new_type branch May 16, 2019 03:19

emkornfield mentioned this pull request Jan 25, 2021

PARQUET-675: Specify Interval LogicalType apache/parquet-format#165

Closed

ARROW-835: [Format][C++][Java] Create a new Duration type #3644

ARROW-835: [Format][C++][Java] Create a new Duration type #3644

Uh oh!

Conversation

emkornfield commented Feb 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Feb 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

emkornfield commented Feb 23, 2019

Uh oh!

emkornfield commented Feb 23, 2019

Uh oh!

pravindra commented Feb 25, 2019

Uh oh!

emkornfield commented Feb 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Mar 11, 2019

Uh oh!

emkornfield commented Apr 30, 2019

Uh oh!

wesm commented May 3, 2019

Uh oh!

wesm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented May 3, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Feb 14, 2019 •

edited

Loading

codecov-io commented Feb 15, 2019 •

edited

Loading

wesm commented May 15, 2019 •

edited

Loading

emkornfield commented May 16, 2019 •

edited

Loading