Skip to content

Conversation

@jinshen-cn
Copy link

@jinshen-cn jinshen-cn commented Oct 5, 2017

Support create table from a parquet file schema

As described in jira:

CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE 'PARQUET' '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
'/user/test/def/'; 

this is a very ugly fix and I would like someone to help to review and refine.
and it only supports create hive table with schema from parquet file.

Tested by test case and tested about build the runnable distribution

test("create table like parquet") {

    val f = getClass.getClassLoader.
      getResource("test-data/dec-in-fixed-len.parquet").getPath
    val v1 =
      """
        |create table if not exists db1.table1 like 'parquet'
      """.stripMargin.concat("'" + f + "'").concat(
      """
        |stored as sequencefile
        |location '/tmp/table1'
      """.stripMargin
      )

    val (desc, allowExisting) = extractTableDesc(v1)

    assert(allowExisting)
    assert(desc.identifier.database == Some("db1"))
    assert(desc.identifier.table == "table1")
    assert(desc.tableType == CatalogTableType.EXTERNAL)
    assert(desc.schema == new StructType()
      .add("fixed_len_dec", "decimal(10,2)"))
    assert(desc.bucketSpec.isEmpty)
    assert(desc.viewText.isEmpty)
    assert(desc.viewDefaultDatabase.isEmpty)
    assert(desc.viewQueryColumnNames.isEmpty)
    assert(desc.storage.locationUri == Some(new URI("/tmp/table1")))
    assert(desc.storage.inputFormat == Some("org.apache.hadoop.mapred.SequenceFileInputFormat"))
    assert(desc.storage.outputFormat == Some("org.apache.hadoop.mapred.SequenceFileOutputFormat"))
    assert(desc.storage.serde == Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
  }

@maropu
Copy link
Member

maropu commented Jul 23, 2018

@crazyjacky Can you close this for now cuz it's not active for a long time?

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@asfgit asfgit closed this in a3ba3a8 Nov 11, 2018
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
Closes apache#21766
Closes apache#21679
Closes apache#21161
Closes apache#20846
Closes apache#19434
Closes apache#18080
Closes apache#17648
Closes apache#17169

Add:
Closes apache#22813
Closes apache#21994
Closes apache#22005
Closes apache#22463

Add:
Closes apache#15899

Add:
Closes apache#22539
Closes apache#21868
Closes apache#21514
Closes apache#21402
Closes apache#21322
Closes apache#21257
Closes apache#20163
Closes apache#19691
Closes apache#18697
Closes apache#18636
Closes apache#17176

Closes apache#23001 from wangyum/CloseStalePRs.

Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants