Redshift dialect with multi-row insert

[ahmeroxa]

This PR/Spike introduces a new Redshift Dialect and support for multi-row inserts.

Ideally we can roll in Redshift specific optimizations into the dialect. In particular the default data mapping of Kafka Records string to Postgres TEXT results in fields being defined as VARCHAR(256) within Redshift. You can read about the Redshift data type mapping here:
https://docs.aws.amazon.com/redshift/latest/dg/r_Character_types.html#r_Character_types-storage-and-ranges

For the sake of testing I have arbitrarily mapped string to VARCHAR(5000) but ideally a more deliberate/intelligent mapping would be implemented.

The multi-row insert is aimed at improving performance of a Sink connector writing to Redshift. During testing multi-row inserts are magnitudes faster than the existing single inserts that the insert mode uses. You can read about AWS's recommendation of multi-row insert here:
https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-multi-row-inserts.html

[willyborankin]

@ahmeroxa Hi, thank you for PR. It looks great. After discussion with my colleges we would kindly ask you to add some changes:

• Since Redshift is derived PostgreSQL database and data types are same except TEXT it is better to extend PostgreSqlDatabaseDialect, override the method getSqlType and use VARCHAR instead of TEXT.
• In case of jdbc connect VARCHAR usually has a possible maximum size, so for Redshift it is better to set it not 5000 but 65535
• Support of multi-insert is in the DatabaseDialect class, it should be implemented for other databases, as well. I think it would better to add implementation in the GenericDatabaseDialect class, so most of the dialects get it by default, except SAP Hana, Sybase and Derby they don't support multi-insert and Oracle has its own syntax (https://oracle-base.com/articles/9i/multitable-inserts#conditional-insert-all)
• All of new features should be covered by tests

[ahmeroxa]

@willyborankin Ok great, those suggestions make sense to me. I can work on those changes.

Regarding VARCHAR(65535) though, the general guidance is to use the smallest possible column size (for you particular usage) due to the fact that VARCHAR columns are left uncompressed in certain cases (e.g. complex queries).

You can find more information here: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-smallest-column-size.html

Perhaps it would be best to make that configurable and provide a sane default (either some arbitrary large number or maybe set the default to MAX and allow users to override it to something smaller).

What do you think?

[ivanyu]

@ahmeroxa yeah, I see no better way apart from configuration.

BTW, I made a separate issue for supporting multi-row insert independently from Redshift: #40. Would you like to work on it?
If so, I think we can close this PR.

[ahmeroxa]

@ivanyu Sure, I can take #40

I do think it's worth having a Redshift specific dialect at some point though (to handle some of the quirks outlined above). We can close this PR and tackle that separately.

[ivanyu]

@ahmeroxa

I do think it's worth having a Redshift specific dialect at some point though (to handle some of the quirks outlined above).

Of course! I mean, when we have the support for multi-row insert in general (in existing dialects), adding Redshift would be a small change.