Move analyze skipping index rules to config #288
Conversation
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
checked partitioning api again, it returns the physical partitioning of this table. We used schema to get list of columns and partitioning to get partition columns. |
rupal-bq
requested review from
dai-chen,
vamsimanohar,
penghuo,
anirudha,
kaituo and
YANG-DB
as code owners
Could you clarify what's the physical partitioning? I was concerned if this API return static list of partitioned column or all partitions. Because I see you also use |
| rules("PARTITION")._1, | ||
| rules("PARTITION")._2) | ||
| } else if (rules.contains(field.dataType.toString)) { | ||
| rules.getString("recommendation.data_type_rules.PARTITION.skipping_type"), |
There was a problem hiding this comment.
extract util method to avoid appending string in different place?
| @@ -7,27 +7,16 @@ package org.opensearch.flint.spark.skipping.recommendations | |||
|
|
|||
| import scala.collection.mutable.ArrayBuffer | |||
|
|
|||
| import org.opensearch.flint.spark.skipping.FlintSparkSkippingStrategy.SkippingKind.{BLOOM_FILTER, MIN_MAX, PARTITION, VALUE_SET} | |||
| import com.typesafe.config.{Config, ConfigFactory} | |||
|
|
|||
| import org.apache.spark.sql.{Row, SparkSession} | |||
| import org.apache.spark.sql.flint.{loadTable, parseTableName} | |||
|
|
|||
| class DataTypeSkippingStrategy extends AnalyzeSkippingStrategy { | |||
There was a problem hiding this comment.
Could you add missing Javadoc on new class, interface and public methods?
|
Thanks for reviewing @dai-chen. Closing this PR for now. Will address all comments in another PR. |
Description
Issues Resolved
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.