Can Parquet and Orc sources support path expressions on the schema projection?

[hannesmiller]

• I can successfully write out rows with structs like so with Parquet:

    val personDetailsStruct = Field.createStructField("PERSON_DETAILS",
      Seq(
        Field("NAME", StringType),
        Field("AGE", IntType.Signed),
        Field("SALARY", DecimalType(Precision(38), Scale(5))),
        Field("CREATION_TIME", TimestampMillisType)
      )
    )
    val schema = StructType(personDetailsStruct)

    val rows = Vector(
      Vector(Vector("Fred", 50, BigDecimal("50000.99000"), new Timestamp(System.currentTimeMillis()))),
      Vector(Vector("Gary", 50, BigDecimal("20000.34000"), new Timestamp(System.currentTimeMillis()))),
      Vector(Vector("Alice", 50, BigDecimal("99999.98000"), new Timestamp(System.currentTimeMillis())))
    )
    Frame.fromValues(schema, rows)
      .to(ParquetSink(parquetFilePath))

• But fails to read it back using the path expression PERSON_DETAILS.NAME:

    ParquetSource(parquetFilePath)
      .withProjection("PERSON_DETAILS.NAME")
      .toFrame()
      .collect()
      .foreach(row => println(row))

• Note path expressions work with withPredicate

    ParquetSource(parquetFilePath)
      .withPredicate(Predicate.or(Predicate.equals("PERSON_DETAILS.NAME", "Alice"), Predicate.equals("PERSON_DETAILS.NAME", "Gary")))
      .toFrame()
      .collect()
      .foreach(row => println(row))

• Note a Hive table pointing at this file works using the following HiveQL syntax:

hive> select person_details.name, person_details.age
    > from struct_person
    > where person_details.name in ('Alice', 'Gary' );
OK
Gary    50
Alice   50
Time taken: 0.067 seconds, Fetched: 2 row(s)
hive>