Added New York city's taxi and limousine commission trip yellow taxi support #151

otegami · 2022-08-31T08:54:28Z

Related Issue

Add support for New York city Taxi and Limousine Commission trip record data red-datasets-parquet#8

What I did

added red-parquet to runtime dependencies
added New York city's taxi and limousine commission trip yellow taxi support
added simple unit test

Details

Yellow Trips Data Dictionary
- ref: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page

What I Didn't

added following datasets related with NYC Taxi and Limousine Commission
- Green Trips Data
- FHV Trips Data
- High Volume FHV Trips Data

What I checked

ran tlc-trip-yellow-taxi example

% ruby example/tlc-trip-yellow-taxi.rb 
[:creative_mobile_technologies, 2022-01-01 09:35:40 +0900, 2022-01-01 09:53:29 +0900, 2.0, 3.8, :standard_rate, false, 142, 236, :credit_card, 14.5, 3.0, 0.5, 3.65, 0.0, 0.3, 21.95, 2.5, 0.0]
[:creative_mobile_technologies, 2022-01-01 09:33:43 +0900, 2022-01-01 09:42:07 +0900, 1.0, 2.1, :standard_rate, false, 236, 42, :credit_card, 8.0, 0.5, 0.5, 4.0, 0.0, 0.3, 13.3, 0.0, 0.0]

Notes for reviewers

Would you give some advice about Class Name
- TLCTripYellowTaxi or TLCTrip::YellowTaxi or Should I add NYC as a prefix
If you don't mind, I want to try to add the other following datasets too
- Green Trips Data
- FHV Trips Data
- High Volume FHV Trips Data

…support

kou

Thanks!

lib/datasets/tlc-trip-yellow-taxi.rb

kou · 2022-09-01T07:37:23Z

lib/datasets/tlc-trip-yellow-taxi.rb

+      @metadata.url = "https://d37ci6vzurychx.cloudfront.net/trip-data"
+      @metadata.licenses = [
+        {
+          name: "Public",


Suggested change

name: "Public",

spdx_id: "CC0-1.0",

fix: 5816dee Fixed metadata information
Although I was thinking I should use array style to express it like @metadata.licenses = ["CC0-1.0"], I couldn't find the license CC0-1.0 in NYC Open Data site so I used hash style.

I rechecked this. The dataset site doesn't mention CC0-1.0 or public domain as you said. Sorry. We should not it.

I found https://opendata.cityofnewyork.us/overview/#termsofuse instead.

How about the following?

@metadata.licenses = [ { name: "NYC Open Data Terms of Use", url: "https://opendata.cityofnewyork.us/overview/#termsofuse", } ]

fix: ca35199 Thank you so much. It sounds perfect for me.🔆

lib/datasets/tlc-trip-yellow-taxi.rb

…y integer.

otegami · 2022-09-04T02:33:55Z

Thank you for reviewing and giving me some advice . I dealt with all of them you pointed so far🙏

Would you help me to figure out why CI was failed on Ubuntu OS and what I should do in next step?
I'm thinking It happed because we failed to install libarrow-dev by using apt...

installing 'libarrow-dev' native package... failed
Failed to run '/usr/bin/sudo -p [sudo]\ password\ for\ %u\ to\ install
<libarrow-dev>:\ apt-get install -V -y libarrow-dev'.

ref: https://github.com/red-data-tools/red-datasets/runs/8173499574?check_suite_focus=true

kou · 2022-09-04T12:35:40Z

test/tlc/test-yellow-taxi-trip.rb

@@ -0,0 +1,59 @@
+module TLC


We don't need this module because we don't share our test and YellowTaxiTripTest will not be conflicted.

If we want to add TLC information to distinct from other test cases, class TLCYellowTaxiTripTest to reduce indent.

We don't need this module because we don't share our test and YellowTaxiTripTest will not be conflicted.

Thank you. I understood it. I will rename it to TLCYellowTaxiTripTest🙏

fix: c75357d

kou · 2022-09-04T12:37:52Z

We need to prepare the Apache Arrow's APT repository for Ubuntu like https://github.com/red-data-tools/red-datasets-arrow/blob/master/.github/workflows/test.yml#L56-L64

Because Apache Arrow package isn't included in the official Ubuntu repository yet.

We don't have to share our test and name of this test case won't be conflicted.

Apache Arrow package isn't included in the official Ubuntu repository yet. ref: red-data-tools#151 (comment)

otegami · 2022-09-05T12:34:15Z

Thank you for reviewing this PR. I dealt with all of them your comments so far🙏

lib/datasets/tlc/yellow-taxi-trip.rb

kou · 2022-09-05T20:58:21Z

Thanks!

TODO: We should consider red-parquet dependency before we release a new version.

… yellow taxi trip records (red-data-tools#151)" We decided not to use Apache Parquet data. Instead of it, we can use it by Red Datasets Parquet. The reason is why we don't want to add red-parquet dependency to red-datasets for easy to install. red-parquet is an extension library. So it's difficult to install rather than a pure Ruby library.

… yellow taxi trip records" (#154) ## What I did - Reverted #151 ### Reasons We decided not to use Apache Parquet data in Red Datasets because we don't want to add red-parquet dependency to red-datasets for easy to install. In details, red-parquet is an extension library. So it's difficult to install rather than a pure Ruby library. Instead of it, we created [Red Datasets Parquet](https://github.com/red-data-tools/red-datasets-parquet) to use Apache Parquet data.

otegami added 2 commits August 31, 2022 17:53

Added New York city's taxi and limousine commission trip yellow taxi …

645e36a

…support

Added a example for using Datasets::TLCTripYellowTaxi class

e4f63ff

otegami marked this pull request as ready for review August 31, 2022 09:08

kou reviewed Sep 1, 2022

View reviewed changes

otegami added 3 commits September 4, 2022 10:47

Renamed TLCTripYellowTaxi Class to TLC::YellowTaxiTrip Class

7806239

Fixed id, name, url and licenses about metadata

5816dee

Formatted year and month when they are used because they are generall…

fc525f8

…y integer.

kou reviewed Sep 4, 2022

View reviewed changes

otegami added 3 commits September 5, 2022 07:41

Removed unnecessary tlc namespace

c75357d

We don't have to share our test and name of this test case won't be conflicted.

Prepared the Apache Arrow's APT repository for Ubuntu

f58ce62

Apache Arrow package isn't included in the official Ubuntu repository yet. ref: red-data-tools#151 (comment)

NYC Open Data doesn't mention CC0-1.0 license or public domain.

ca35199

kou reviewed Sep 5, 2022

View reviewed changes

lib/datasets/tlc/yellow-taxi-trip.rb Outdated Show resolved Hide resolved

Fix a typo

622c0d8

kou merged commit 520694e into red-data-tools:master Sep 5, 2022

kou mentioned this pull request Sep 5, 2022

How to support Apache Parquet data? #152

Closed

otegami deleted the new-york-city-taxi-n-limousine-commission-trip branch September 5, 2022 21:15

otegami mentioned this pull request Sep 22, 2022

Revert "Add support for New York city's Taxi and Limousine Commission yellow taxi trip records" #154

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added New York city's taxi and limousine commission trip yellow taxi support #151

Added New York city's taxi and limousine commission trip yellow taxi support #151

otegami commented Aug 31, 2022 •

edited

Loading

kou left a comment

kou Sep 1, 2022

otegami Sep 4, 2022 •

edited

Loading

kou Sep 4, 2022

otegami Sep 5, 2022

otegami commented Sep 4, 2022 •

edited

Loading

kou Sep 4, 2022

otegami Sep 4, 2022

otegami Sep 4, 2022

kou commented Sep 4, 2022

otegami commented Sep 5, 2022

kou commented Sep 5, 2022

Added New York city's taxi and limousine commission trip yellow taxi support #151

Added New York city's taxi and limousine commission trip yellow taxi support #151

Conversation

otegami commented Aug 31, 2022 • edited Loading

Related Issue

What I did

Details

What I Didn't

What I checked

Notes for reviewers

kou left a comment

Choose a reason for hiding this comment

kou Sep 1, 2022

Choose a reason for hiding this comment

otegami Sep 4, 2022 • edited Loading

Choose a reason for hiding this comment

kou Sep 4, 2022

Choose a reason for hiding this comment

otegami Sep 5, 2022

Choose a reason for hiding this comment

otegami commented Sep 4, 2022 • edited Loading

kou Sep 4, 2022

Choose a reason for hiding this comment

otegami Sep 4, 2022

Choose a reason for hiding this comment

otegami Sep 4, 2022

Choose a reason for hiding this comment

kou commented Sep 4, 2022

otegami commented Sep 5, 2022

kou commented Sep 5, 2022

otegami commented Aug 31, 2022 •

edited

Loading

otegami Sep 4, 2022 •

edited

Loading

otegami commented Sep 4, 2022 •

edited

Loading