(@aws-cdk.aws-glue-alpha): Glue table separator + skip header

[aram-eskandari]

Describe the issue

I have created a glue-table using DataFormat.CSV with version aws-cdk.aws-glue-alpha==2.51.1a0 and python 3.7.13. The code looks similar to:

Now I'm trying to adjust the parameters skip.header.line.count and separatorChar. Specifically I want to skip the 1st row and use a semicolon separator.

Using L1's, the code would look similar to:

Looked through the L2-documentation but couldn't find anything regarding this, any tips?

On a separate note, there is a repeating sentence in the documentation:

Links

https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_glue_alpha/README.html#table

[peterwoodworth]

On a quick glance it doesn't look like this is supported on out L2, and will need to be done either with the L1 like you've done, or using escape hatches

Source code:

aws-cdk/packages/@aws-cdk/aws-glue/lib/table.ts

Line 302 in 3d30cdb

const tableResource = new CfnTable(this, 'Table', {

As for the docs, this should be easy to fix by removing the repeated sentence from the README https://github.com/aws/aws-cdk/blob/v2.51.1/packages/%40aws-cdk/aws-glue/README.md

[aram-eskandari]

i've never used escape hatches will have to read up on it, thanks for your response and the pointer @peterwoodworth!

[WtfJoke]

@markusl The skip.header.line.count is a table properties and therefore can be set like:

const cfnTable = table.node.defaultChild as CfnTable;
cfnTable.addPropertyOverride("TableInput.Parameters", {
    "skip.header.line.count": "1",
});

[markusl]

@WtfJoke Thanks! Edited my comment for clarity. Overriding TableInput.Parameters seems to work for setting skip.header.line.count.

[WtfJoke]

@markusl yeah we tested this successfully. Are you sure the properties are visible as table properties?
As far as I remember with your snippet it was in the serde properties instead of the table properties.

We have also serde properties overwritten:

  /*
      Set additional serializer properties (since its not possible with the current L2 construct):
      - Switch serializer from OpenCSV to LazySimpleSerDe (OpenCSV supports only string).
      - Specify ',' as field separator (default is tab)
   */
  cfnTable.addPropertyOverride("TableInput.StorageDescriptor.SerdeInfo", {
      SerializationLibrary: "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
      Parameters: {
          "field.delim": ",",
      },
  });

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.