feat: add save dynamic csv #5130
Conversation
klDen
force-pushed
the
feat/add-dynamic-csv-rw
branch
from
390f8cc
to
caba916
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #5130 +/- ##
==========================================
+ Coverage 63.34% 63.49% +0.14%
==========================================
Files 291 293 +2
Lines 10840 10844 +4
Branches 753 752 -1
==========================================
+ Hits 6867 6885 +18
+ Misses 3973 3959 -14 ☔ View full report in Codecov by Sentry. |
klDen
force-pushed
the
feat/add-dynamic-csv-rw
branch
from
caba916
to
3e448a5
Compare
| import com.spotify.scio.values.SCollection | ||
| import kantan.csv._ | ||
| import kantan.codecs.compat._ |
There was a problem hiding this comment.
IntelliJ probably tells this import is unused for 2.13, but it is required for +compile with 2.12
There was a problem hiding this comment.
Sounds good. Re-adding it
| * }}} | ||
| */ | ||
| package object csv extends AllSyntax { | ||
| final private[scio] class CsvSink[T: HeaderEncoder](csvConfig: CsvConfiguration) |
There was a problem hiding this comment.
Defining classes in package object has been discouraged. This creates the following warning:
it is not recommended to define classes/objects inside of package objects.
If possible, define class CsvSink in package csv instead.
It would be nicer to move this to another compile unit inside the csv package.
There was a problem hiding this comment.
Got it. I'll move CsvSink in a dedicated file under scio.extra.csv
Is there a plan to migrate other classes that are defined in package objects? e.g. https://github.com/klDen/scio/blob/d97f0387393315d89fd0bbd6684306bd5f37a376/scio-parquet/src/main/scala/com/spotify/scio/parquet/package.scala
There was a problem hiding this comment.
Yes, this is a work in progress: #5127
klDen
force-pushed
the
feat/add-dynamic-csv-rw
branch
2 times, most recently
from
ed7573f
to
c027145
Compare
klDen
force-pushed
the
feat/add-dynamic-csv-rw
branch
2 times, most recently
from
308aaac
to
91bb15c
Compare
klDen
force-pushed
the
feat/add-dynamic-csv-rw
branch
from
91bb15c
to
437ce44
Compare
scio-extra/src/test/scala/com/spotify/scio/extra/csv/dynamic/CsvDynamicTest.scala
Outdated
Show resolved
Hide resolved
Description
Add save dynamic CSV files.