Feature proposal: Add Parquet as a Batch encoding option

[jamielxcarter]

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

Currently the only Batch encoding option is JSONL. This feature will add support for writing batch files as parquet.

[aaronsteers]

Hi, @jamielxcarter - thanks for logging this.

Just as a heads up, we currently have pyarrow as a dev dependency in the SDK, but this likely would propose adding as an extra or as a core dependency. We previously added special conditions for pyarrow inclusion because it did not yet support Python 3.11 - although it appears that is now resolved:

• pyarrow wheel for python 3.11 apache/arrow#14572

It would be great if there were other smaller/lighter dependencies for writing Parquet that are also very stable+fast, but wanted to note this 👆 as an FYI if going the pyarrow route.

[jamielxcarter]

Thanks @aaronsteers. If it weren't already a dependency I think I would have used fastparquet, since it has a significantly smaller footprint: fastparquet 3.9M vs pyarrow 87M. I can add pyarrow as an extra just to be safe if that's what you prefer.

[aaronsteers]

@jamielxcarter - fastparquet actually sounds like a better choice, IMHO. I haven't used it myself but I've heard quite a lot about it and the smaller footprint is very desirable.

The pyarrow dependent can be removed or refactored. To clarify my point above, as of now, pyarrow is only used for one of the samples (tap-parquet, as a dev/test sample for contributors) and it's not actually leveraged as of now any of the main codebase. As such, it isn't installed by default in taps and targets that use the SDK.

It sounds like fastparquet may be small enough to include it by default in taps and targets, which is the ideal behavior, so that all taps and targets can support reading from and writing to parquet files, without additional work from the end user consuming the connector.

[jamielxcarter]

@jamielxcarter - fastparquet actually sounds like a better choice, IMHO. I haven't used it myself but I've heard quite a lot about it and the smaller footprint is very desirable.

The pyarrow dependent can be removed or refactored. To clarify my point above, as of now, pyarrow is only used for one of the samples (tap-parquet, as a dev/test sample for contributors) and it's not actually leveraged as of now any of the main codebase. As such, it isn't installed by default in taps and targets that use the SDK.

It sounds like fastparquet may be small enough to include it by default in taps and targets, which is the ideal behavior, so that all taps and targets can support reading from and writing to parquet files, without additional work from the end user consuming the connector.

Hey @aaronsteers, I think the only issue with fastparquet is that it requires that we write using a pandas dataframe, where as pyarrow has a method specifically for writing from a list of python dictionaries. I still lean toward fastparquet, but it does mean importing pandas.DataFrame. Thoughts?

[VMois]

What do you think about adding support for arbitrary batch file format? Not only JSONL or Parquet, etc.

The use case, for example, is when I only want to copy files from tap to some staging area (CSV/Parquet files from an external S3 bucket to mine). This is currently not possible. My tap fails if I specify a CSV file as encoding in the config.

sdk/singer_sdk/helpers/_batch.py

Line 214 in ad53dfa

self.encoding = BaseBatchFileEncoding.from_dict(self.encoding)

[aaronsteers]

What do you think about adding support for arbitrary batch file format? Not only JSONL or Parquet, etc.

The use case, for example, is when I only want to copy files from tap to some staging area (CSV/Parquet files from an external S3 bucket to mine). This is currently not possible. My tap fails if I specify a CSV file as encoding in the config.

sdk/singer_sdk/helpers/_batch.py

Line 214 in ad53dfa

self.encoding = BaseBatchFileEncoding.from_dict(self.encoding)

Did you have a specific implementation in mind? We don't have any specific preference to limit what formats should be supported, but for the dev effort, each file format generally requires a non-trivial amount of effort to generate and/or configure.

For example, Parquet is easy to configure but non-trivial to implement. CSV is trivial to implement but non-trivial to configure.

[VMois]

Did you have a specific implementation in mind?

Nothing in mind. I think I have misunderstood how SDK exactly works. It does a lot of job for you to load/unpack files from S3/Local. For now, I have modified a non-SDK tap/target to work with a batch format I prefer (CSV, not compressed).

The idea for SDK can be the ability to define/extend formats and provide your methods to load/unpack data.

---

After looking into the source code, I think overwriting the process_batch_file function will allow me to control how to extract data:

sdk/singer_sdk/sinks/core.py

Lines 465 to 470 in 405ee68

	def process_batch_files(
	self,
	encoding: BaseBatchFileEncoding,
	files: Sequence[str],
	) -> None:
	"""Process a batch file with the given batch context.

I can modify get_batches for taps, but the SDK config fails if a non-supported format and storage are provided here:

sdk/singer_sdk/helpers/_batch.py

Line 59 in 405ee68

encoding_cls = cls.registered_encodings[encoding_format]

File "/Users/vvvv/.local/pipx/venvs/tap-batch/lib/python3.9/site-packages/singer_sdk/helpers/_batch.py", line 59, in from_dict
    encoding_cls = cls.registered_encodings[encoding_format]
KeyError: 'csv'

So possible solution might be to relax those config settings and let people use any value.

[aaronsteers]

@VMois - So happy to hear you've been able to build an implementation here! If you have time, I've love to hear your thoughts on our (very!) new spec proposal for CSV-backed batch message operations:

• Add support for CSV encoding for BATCH messaging #1584

And specifically this latest comment with a draft proposal:

• Add support for CSV encoding for BATCH messaging #1584 (comment)

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant

[BTheunissen]

I see a PR got opened for this item, once this gets released I'd like to add batch json/parquet support to my Target for Clickhouse :)