Export Hooks: Glue Catalog exporter #6653

Isan-Rivkin · 2023-09-26T20:17:22Z

TODO - will create additional PR and merge into current:

Add system test with Glue / AWS
Add documentation

Usage

Pre-requisite

Write some table data to lakeFS either parquet or CSV format, snippet of parquet added:

data =[("James","","Smith","36636","M",3000),
              ("Michael","Rose","","40288","M",4000),
              ("Robert","","Williams","42114","M",4000),
              ("Maria","Anne","Jones","39192","F",4000),
              ("Jen","Mary","Brown","","F",-1)]

columns=["firstname","middlename","lastname","dob","gender","salary"]

df=spark.createDataFrame(data,columns)

Define table in `_lakefs_tables/employees.yaml`

name: employees
type: hive
path: <path/to/parquet/files>
partition_columns: ['gender', 'salary']
schema:
  type: struct
  fields:
    - name: gender
      type: string
      nullable: false
      metadata: {}
    - name: salary
      type: integer
      nullable: false
      metadata: {}
    - name: firstname
      type: string
      nullable: false
      metadata: {}
    - name: middlename
      type: string
      nullable: false
      metadata: {}
    - name: lastname
      type: string
      nullable: false
      metadata: {}
    - name: dob
      type: string
      nullable: false
      metadata: {}

Create `_lakefs_action/employee_exporter.yaml`

name: glue catalog exporter
on:
  post-create-branch:
    branches: ["main*"]
  post-commit:
    branches: ["main*"]
hooks:
  # symlink_exporter placegolder - can be separated to create symlink file
  - id: symlink_exporter
  - id: glue_exporter
    type: lua
    properties:
      script_path: scripts/glue_export.lua
      args:
        # Export configuration
        aws:
          aws_access_key_id: ''
          aws_secret_access_key: ''
          aws_region: ''
        table_source: '_lakefs_tables/employees.yaml'
        catalog:
          db_name: "<glue-database-name>"
          # csv input example, can be passed from here or from script file
          # table_input:
          #   StorageDescriptor: 
          #     InputFormat: "org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat"
          #     # InputFormat: "org.apache.hadoop.mapred.TextInputFormat"
          #     OutputFormat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
          #     SerdeInfo:
          #         SerializationLibrary: "org.apache.hadoop.hive.serde2.OpenCSVSerde"
          #         Parameters:
          #             separatorChar: ","
          #   Parameters: 
          #     classification: "csv"

Create export hook in `scripts/glue_export.lua`

local aws = require("aws")
local exporter = require("lakefs/catalogexport/glue_exporter")

-- config 
local table_path = args.table_source
local db = args.catalog.db_name
local options = {
    debug=true,
}

-- init glue client 
local glue = aws.glue_client(args.aws.aws_access_key_id, args.aws.aws_secret_access_key, args.aws.aws_region)

-- configure parquet
local table_input = {
    StorageDescriptor={
        InputFormat= "org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat",
        OutputFormat= "org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat",
        SerdeInfo={
            SerializationLibrary= "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
        },
    },
    Parameters={
        classification= "parquet",
        "parquet.compression"= "SNAPPY",
    }
}

-- export glue table 
local result = exporter.export_glue(glue, db, table_path, table_input, action, options)
print("success exporting table to glue location", result.create_table_input.StorageDescriptor.Location)

Isan-Rivkin · 2023-10-01T16:19:24Z

@nopcoder please notice that PR description added usage (TODO's will be done after this PR code approved)

nopcoder · 2023-10-02T08:14:25Z

pkg/actions/lua/storage/aws/client.go

name the file aws.go as we do for other packages - name the main library as the package.

nopcoder · 2023-10-02T10:13:01Z

pkg/actions/lua/storage/aws/glue.go

+		config.WithRegion(c.Region),
+		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(c.AccessKeyID, c.SecretAccessKey, "")),


The credentioals are required, but the region can be empty - verify that LoadDefaultConfig with empty region works. I think you should not pass it to the options in order to use the default configured.

Verified, works :)

nopcoder · 2023-10-02T10:24:11Z

pkg/actions/lua/lakefs/catalogexport/hive.lua

                table.remove(page, 1)
+            else
+                -- remove hidden files from the entry set
+                table.remove(page, 1)


Looks like we are doing the same - so you can just move table.remove out of the both if scope.

nopcoder · 2023-10-02T10:44:12Z

pkg/actions/lua/lakefs/catalogexport/glue_exporter.lua

+-- create a standard AWS Glue table input (i.e not Apache Iceberg), add input values to base input and configure the rest
+local function build_glue_create_table_input(base_input, descriptor, symlink_location, columns, partitions, action_info,


do you see any real usage for this one in the near future? if not inline it

Yes, I see usage incase user want's to use opts.override_create_table_input and change certain things in build_glue_create_table_input output.
Actually I added now export for this function as well outside of the module.

nopcoder · 2023-10-02T10:45:32Z

pkg/actions/lua/lakefs/catalogexport/glue_exporter.lua

+    local repo_id = action_info.repository_id
+    local commit_id = action_info.commit_id


not using repo and commit id

nopcoder · 2023-10-02T10:46:50Z

pkg/actions/lua/lakefs/catalogexport/glue_exporter.lua

+    -- create table
+    local json_input = json.marshal(table_input)
+    if opts.debug then
+        print("Creating Glue Table - input: ", json_input)


unless you concat the strings

Suggested change

print("Creating Glue Table - input: ", json_input)

print("Creating Glue Table - input:", json_input)

nopcoder · 2023-10-02T10:47:56Z

pkg/actions/lua/lakefs/catalogexport/glue_exporter.lua

+    return {
+        create_table_input=table_input
+    }


why not returning just table_input?

To leave room for other variables and make it a "result" without breaking function signature.

For example, in a more complex scenario where export will do version updates to the same table we will want the new latest version things like that.

changed to table_input

nopcoder · 2023-10-02T10:50:48Z

pkg/actions/lua/lakefs/catalogexport/internal.lua

@@ -1,6 +1,18 @@
 local url = require("net/url")
 local DEFAULT_SHORT_DIGEST_LEN=6

+local function clone_table(original_table)


Suggest use the deepcopy (and call it deepcopy as it does deep copy for any type) from the formal lua wiki:

function deepcopy(orig) local orig_type = type(orig) local copy if orig_type == 'table' then copy = {} for orig_key, orig_value in next, orig, nil do copy[deepcopy(orig_key)] = deepcopy(orig_value) end setmetatable(copy, deepcopy(getmetatable(orig))) else -- number, string, boolean, etc copy = orig end return copy end

woha took me a while to compile that in my mind, but better for sure - updated thanks.

nopcoder · 2023-10-02T10:55:24Z

pkg/actions/lua/storage/aws/glue.go

+		// parse table input JSON
+		var tableInput types.TableInput
+		err := json.Unmarshal([]byte(tableInputJSON), &tableInput)
+		if err != nil {
+			lua.Errorf(l, err.Error())
+			panic("unreachable")
+		}


move parse after the you finish the params extract - I think it will be readable and better structure of the function implementation.

nopcoder · 2023-10-02T11:01:19Z

pkg/actions/lua/storage/aws/glue.go

+		retArgs := util.DeepPush(l, itemMap)
+		l.PushBoolean(true)
+		return retArgs + 1


Strange that DeepPush needs to return a number - it expected to already return 1.
Same for this function getTable return - we must return 2 as we like the boolean to be on position 2.

Good catch, changed 👍

Isan-Rivkin · 2023-10-02T11:41:39Z

@nopcoder Thanks! Please re-review 🙏

nopcoder

LGTM!

prefix array

1cf8162

Isan-Rivkin added the include-changelog PR description should be included in next release changelog label Sep 26, 2023

Isan-Rivkin self-assigned this Sep 26, 2023

Isan-Rivkin changed the title ~~Export Hooks: Glue Catalog exporter~~ WIP: Export Hooks: Glue Catalog exporter Sep 26, 2023

Isan-Rivkin mentioned this pull request Sep 26, 2023

Bug: Lua runtime - marshaling arrays #6655

Merged

Isan-Rivkin added 6 commits September 27, 2023 00:03

working create table

6a92708

update table

9643f0f

Merge branch 'master' into 6575-glue-exporter

e13b1a0

update

3451f33

working glue lib

3d75553

remove todos

e06ceb4

Isan-Rivkin changed the title ~~WIP: Export Hooks: Glue Catalog exporter~~ Export Hooks: Glue Catalog exporter Oct 1, 2023

fix linter

cc9e4d2

Isan-Rivkin requested a review from nopcoder October 1, 2023 16:18

Isan-Rivkin added 2 commits October 2, 2023 10:38

Merge branch 'master' into 6575-glue-exporter

2f55597

linter fix

af6d707

nopcoder suggested changes Oct 2, 2023

View reviewed changes

Isan-Rivkin added 3 commits October 2, 2023 14:33

review comments

33724d2

review comments

fba4e03

rename aws file

3823378

Isan-Rivkin requested a review from nopcoder October 2, 2023 11:41

minor

1222bf1

nopcoder approved these changes Oct 2, 2023

View reviewed changes

Isan-Rivkin added 2 commits October 2, 2023 18:28

add comments final review fixes

e49f891

Merge branch 'master' into 6575-glue-exporter

679f2b2

Isan-Rivkin merged commit 7855a18 into master Oct 3, 2023

Isan-Rivkin deleted the 6575-glue-exporter branch October 3, 2023 06:23

Isan-Rivkin mentioned this pull request Oct 3, 2023

[Docs]: Glue Exporter #6685

Closed

Isan-Rivkin mentioned this pull request Oct 3, 2023

Test: Hooks Glue Exporter #6686

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export Hooks: Glue Catalog exporter #6653

Export Hooks: Glue Catalog exporter #6653

Isan-Rivkin commented Sep 26, 2023 •

edited

Loading

Isan-Rivkin commented Oct 1, 2023

nopcoder Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

nopcoder Oct 2, 2023

Isan-Rivkin Oct 2, 2023

Isan-Rivkin commented Oct 2, 2023 •

edited

Loading

nopcoder left a comment

		config.WithRegion(c.Region),
		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(c.AccessKeyID, c.SecretAccessKey, "")),

		-- create a standard AWS Glue table input (i.e not Apache Iceberg), add input values to base input and configure the rest
		local function build_glue_create_table_input(base_input, descriptor, symlink_location, columns, partitions, action_info,

		local repo_id = action_info.repository_id
		local commit_id = action_info.commit_id

	print("Creating Glue Table - input: ", json_input)
	print("Creating Glue Table - input:", json_input)

Export Hooks: Glue Catalog exporter #6653

Export Hooks: Glue Catalog exporter #6653

Conversation

Isan-Rivkin commented Sep 26, 2023 • edited Loading

TODO - will create additional PR and merge into current:

Usage

Pre-requisite

Define table in _lakefs_tables/employees.yaml

Create _lakefs_action/employee_exporter.yaml

Create export hook in scripts/glue_export.lua

Isan-Rivkin commented Oct 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Isan-Rivkin commented Oct 2, 2023 • edited Loading

nopcoder left a comment

Choose a reason for hiding this comment

Isan-Rivkin commented Sep 26, 2023 •

edited

Loading

Define table in `_lakefs_tables/employees.yaml`

Create `_lakefs_action/employee_exporter.yaml`

Create export hook in `scripts/glue_export.lua`

Isan-Rivkin commented Oct 2, 2023 •

edited

Loading