EEL DSL for a CLI shell

[hannesmiller]

EEL DSL for a CLI shell

• A Scala DSL for EEL commands.
• The Scala REPL to be used as an interactive shell
• Scala variables, loops and conditional statements can be combined with DSL commands to easily script tasks
• A bootstrap shell script called eel-shell similar to spark-shell - will automatically import the eel DSL packages
• Also automatically import OS packages to allow for OS commands to be run and combined with the DSL

Import

Import data with options from an EEL source to a sink

import from jdbc
with driver=blah,url=blah,sql=blah
to hive
with db=blah,table=blah

Import from jdbc
with driver=blah,url=blah,sql=blah
to Parquet
with path=blah

Export

Export data with options to an EEL sink from a source

export to hive
with db=blah,table=blah
from jdbc
with driver=blah,url=blah,sql=blah

export to hive
with db=blah,table=blah
from Parquet
with path=blah

• Note allow a transform sub-command at the appropriate place to do custom transformations on the underlying rows in the frame.

More commands to follow....

[hannesmiller]

DDL

Display the DDL from create table command for JDBC query:

ddl from JDBC
with driver=blah,
url=jdbc:blah,
sql=blah,
dialect=Parquet,
location=blah,
sql=blah
partitions=p1:string,p2:int

• partitions are specified using the notation name:type

[hannesmiller]

File Compaction

Reduce several small files into a single file for a folder or hive table.

Note for Parquet or other similar types the new file's schema should be the union of all schemas.

In addition you may have to add columns to the HiveMetasore.

• compact Parquet path
• compact Orc path
• compact Hive dbName table