Skip to content

Commit

Permalink
Refactor PhysicalTableDefinition and Update BaseTableLoader for table…
Browse files Browse the repository at this point in the history 
… dependency
  • Loading branch information
@garyluoex
garyluoex committed Apr 3, 2017
1 parent 7143e53 commit 4e58f37
Show file tree
Hide file tree
Showing 8 changed files with 511 additions and 195 deletions.
16 changes: 16 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ pull request if there was one.
Current
-------
### Added:

- [CompositePhysicalTable Core Components Refactor](https://github.com/yahoo/fili/pull/179)
* Added `ConcretePhysicalTable` and `ConcreteAvailability` to model table in druid datasource and its availabillity in the new table availability structure
* Added class variable for `DataSourceMetadataService` and `ConfigurationLoader` into `AbstractBinderFactory` for application to access
* Added `loadPhsycialTablesWithDependency` into `BaseTableLoader` to load physical tables with dependencies

- [PermissiveAvailability and PermissiveConcretePhysicalTable](https://github.com/yahoo/fili/pull/190)
* Added `PermissiveConcretePhysicalTable` and `PermissiveAvailability` to model table in druid datasource and its availability in the new table availability structure.
Expand Down Expand Up @@ -59,6 +61,13 @@ Current

### Changed:

- [Refactor Physical Table Definition and Update Table Loader](https://github.com/yahoo/fili/pull/207)
* `PhysicalTableDefinition` is now an abstract class, construct using `ConcretePhysicalTableDefinition` instead
* `PhysicalTableDefinition` now requires a `build` methods to be implemented that builds a physical table
* `BaseTableLoader` now constructs physical tables by calling `build` method on `PhysicalTableDefinition`s in `buildPhysicalTablesWithDependency`
* `buildDimensionSpanningTableGroup` method in `BaseTableLoader` now uses `loadPhysicalTablesWithDependency` instead of removed `loadPhysicalTables`
* `buildDimensionSpanningTableGroup` method in `BaseTableLoader` now does not take druid metric as arguments, instead `PhysicalTableDefinition` does

- [Make `TemplateDruidQuery::getMetricField` get the first field instead of any field](https://github.com/yahoo/fili/pull/210)
* Previously, order was by luck, now it's by the contract of `findFirst`

Expand All @@ -69,6 +78,7 @@ Current
- [CompositePhsyicalTable Core Components Refactor](https://github.com/yahoo/fili/pull/179)
* `TableLoader` now takes an additional constructor argument `DataSourceMetadataService` for creating tables
* `findMissingRequestTimeGrainIntervals` method in `PartialDataHandler` now takes `DataSourceConstraint`
* Renamed `buildTableGroup` method to `buildDimensionSpanningTableGroup`

- [Restored flexibility about columns for query from DruidResponseParser](https://github.com/yahoo/fili/pull/198)
* Immutable schemas prevented custom query types from changing `ResultSetSchema` columns.
Expand Down Expand Up @@ -169,6 +179,12 @@ Current


### Removed:

- [Refactor Physical Table Definition and Update Table Loader](https://github.com/yahoo/fili/pull/207)
* Removed deprecated `PhysicalTableDefinition` constructor that takes an `ZonlessTimeGrain`, use `ZonedTimeGrain` instead
* Removed `loadPhysicalTable` in `BaseTableLoader`, use `loadPhysicalTablesWithDependency` instead
* Removed `buildPhysicalTable` in `BaseTableLoader`, building table logic is pushed into `PhysicalTableDefinition`

- [CompositePhsyicalTable Core Components Refactor](https://github.com/yahoo/fili/pull/179)
* Removed deprecated method `findMissingRequestTimeGrainIntervals` from `PartialDataHandler`
* Removed `permissive_column_availability_enabled` feature flag support and corresponding functionality in `PartialDataHandler`, permissive availability will be a table configuration
Expand Down
204 changes: 117 additions & 87 deletions fili-core/src/main/java/com/yahoo/bard/webservice/data/config/table/BaseTableLoader.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,18 @@
package com.yahoo.bard.webservice.data.config.table;

import com.yahoo.bard.webservice.data.config.ResourceDictionaries;
import com.yahoo.bard.webservice.data.config.dimension.DimensionConfig;
import com.yahoo.bard.webservice.data.config.names.ApiMetricName;
import com.yahoo.bard.webservice.data.config.names.FieldName;
import com.yahoo.bard.webservice.data.config.names.TableName;
import com.yahoo.bard.webservice.data.dimension.Dimension;
import com.yahoo.bard.webservice.data.dimension.DimensionColumn;
import com.yahoo.bard.webservice.data.dimension.DimensionDictionary;
import com.yahoo.bard.webservice.data.metric.MetricColumn;
import com.yahoo.bard.webservice.data.metric.MetricDictionary;
import com.yahoo.bard.webservice.druid.model.query.Granularity;
import com.yahoo.bard.webservice.metadata.DataSourceMetadataService;
import com.yahoo.bard.webservice.table.Column;
import com.yahoo.bard.webservice.table.ConcretePhysicalTable;
import com.yahoo.bard.webservice.table.LogicalTable;
import com.yahoo.bard.webservice.table.LogicalTableDictionary;
import com.yahoo.bard.webservice.table.PhysicalTable;
import com.yahoo.bard.webservice.table.PhysicalTableDictionary;
import com.yahoo.bard.webservice.table.TableGroup;
import com.yahoo.bard.webservice.table.TableIdentifier;

Expand All @@ -28,9 +25,11 @@

import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.Stream;

/**
* Provides commonly-needed methods for loading tables.
Expand Down Expand Up @@ -77,24 +76,21 @@ protected BaseTableLoader(DateTimeZone defaultTimeZone, DataSourceMetadataServic
* Builds and loads the physical tables for the physical table definitions as well.
*
* @param apiMetrics The set of metric names surfaced to the api
* @param druidMetrics Names of druid datasource metric columns
* @param tableDefinitions A list of config objects for physical tables
* @param dictionaries The container for all the data dictionaries
*
* @return A table group binding all the tables for a logical table view together.
*/
public TableGroup buildDimensionSpanningTableGroup(
Set<ApiMetricName> apiMetrics,
Set<FieldName> druidMetrics,
Set<PhysicalTableDefinition> tableDefinitions,
ResourceDictionaries dictionaries
) {
// Load a physical table for each of the table definitions
LinkedHashSet<PhysicalTable> physicalTables = new LinkedHashSet<>();
for (PhysicalTableDefinition def : tableDefinitions) {
PhysicalTable table = loadPhysicalTable(def, druidMetrics, dictionaries);
physicalTables.add(table);
}
LinkedHashSet<PhysicalTable> physicalTables = loadPhysicalTablesWithDependency(
tableDefinitions,
dictionaries
);

//Derive the logical dimensions by taking the union of all the physical dimensions
Set<Dimension> dimensions = physicalTables.stream()
Expand Down Expand Up @@ -158,6 +154,113 @@ public void loadLogicalTableWithGranularities(
}
}

/**
* Load a new physical table into the dictionary and return the loaded physical table.
*
* @param definitions A config object for the physical table
* @param dictionaries The resource dictionaries for reading and storing resource data
*
* @return The physical table created
*/
protected LinkedHashSet<PhysicalTable> loadPhysicalTablesWithDependency(
Set<PhysicalTableDefinition> definitions,
ResourceDictionaries dictionaries
) {
LinkedHashSet<PhysicalTable> loadedTables = new LinkedHashSet<>();

Map<TableName, PhysicalTableDefinition> mutableCopyDefinitions =
buildPhysicalTableDefinitionDictionary(definitions);


definitions.forEach(definition -> buildPhysicalTablesWithDependency(
definition.getName(),
loadedTables,
mutableCopyDefinitions,
dictionaries
));

return loadedTables;
}

/**
* Iterate through given definitions and builds all corresponding physical table with satisfied dependency.
*
* @param currentTableName Iterator for the mutable definition
* @param loadedTables A mutable set of physical tables that are build as a result of this method call
* @param mutableTableDefinitionMap A map of table name to table definition that are awaiting to be built
* @param dictionaries Contains both dimension and physical table dictionary for building and dependency resolution
*/
protected void buildPhysicalTablesWithDependency(
TableName currentTableName,
Set<PhysicalTable> loadedTables,
Map<TableName, PhysicalTableDefinition> mutableTableDefinitionMap,
ResourceDictionaries dictionaries
) {
PhysicalTableDictionary physicalTableDictionary = dictionaries.getPhysicalDictionary();

if (physicalTableDictionary.containsKey(currentTableName.asName())) {
PhysicalTable physicalTable = physicalTableDictionary.get(currentTableName.asName());
loadedTables.add(physicalTable);
return;
}

PhysicalTableDefinition currentTableDefinition = mutableTableDefinitionMap.remove(currentTableName);

// If the dependent table definition is currently being build or missing
if (Objects.isNull(currentTableDefinition)) {
LOG.error("Unable to resolve physical table dependency for physical table: " + currentTableName.asName());
throw new RuntimeException("Unable to resolve physical table dependency for physical table: " +
currentTableName
.asName());
}

// Recurse on all dependent tables
currentTableDefinition.getDependentTableNames()
.forEach(tableName -> buildPhysicalTablesWithDependency(
tableName,
loadedTables,
mutableTableDefinitionMap,
dictionaries
));

Optional<PhysicalTable> optionalPhysicalTable = currentTableDefinition.build(
dictionaries,
getDataSourceMetadataService()
);

if (optionalPhysicalTable.isPresent()) {
PhysicalTable currentPhysicalTable = optionalPhysicalTable.get();
loadedTables.add(currentPhysicalTable);
physicalTableDictionary.put(currentPhysicalTable.getName(), currentPhysicalTable);
} else {
LOG.error("Unable to build physical table: " + currentTableDefinition.getName().asName());
throw new RuntimeException("Unable to build physical table: " + currentTableDefinition.getName().asName());
}
}

/**
* Getter for the data source metadata service use for creating physical tables.
*
* @return the data source metadata service associated with the table loader
*/
protected DataSourceMetadataService getDataSourceMetadataService() {
return metadataService;
}

/**
* Build a map from physical table name to its table definition.
*
* @param physicalTableDefinitions Definitions to build the map from
*
* @return the map of physical table name to its table definition
*/
private Map<TableName, PhysicalTableDefinition> buildPhysicalTableDefinitionDictionary(
Set<PhysicalTableDefinition> physicalTableDefinitions
) {
return physicalTableDefinitions.stream()
.collect(Collectors.toMap(PhysicalTableDefinition::getName, Function.identity()));
}

/**
* Build a logical table, supplying it with a name, grain, table group, and metrics, the default category and
* longName set to the name.
Expand Down Expand Up @@ -256,79 +359,6 @@ public TableGroup buildTableGroup(
Set<PhysicalTableDefinition> tableDefinitions,
ResourceDictionaries dictionaries
) {
return buildDimensionSpanningTableGroup(apiMetrics, druidMetrics, tableDefinitions, dictionaries);
}

/**
* Load a new physical table into the dictionary and return the loaded physical table.
*
* @param definition A config object for the physical table
* @param metricNames The Set of metric names on the table
* @param dictionaries The resource dictionaries for reading and storing resource data
*
* @return The physical table created
*/
protected PhysicalTable loadPhysicalTable(
PhysicalTableDefinition definition,
Set<FieldName> metricNames,
ResourceDictionaries dictionaries
) {
PhysicalTable existingTable = dictionaries.getPhysicalDictionary().get(definition.getName().asName());
if (existingTable == null) {
// Build the physical table
existingTable = buildPhysicalTable(definition, metricNames, dictionaries.getDimensionDictionary());

// Add the table to the dictionary
LOG.debug("Physical table: {} \n\n" + "Cache: {} ",
definition.getName().asName(),
existingTable.getColumns());
dictionaries.getPhysicalDictionary().put(definition.getName().asName(), existingTable);
}

return existingTable;
}

/**
* Build a physical table, populating columns and initializing the cache.
*
* @param definition A config object for the physical table
* @param metricNames Set of Druid metrics on the table
* @param dimensionDictionary The dimension dictionary
*
* @return The physical table, configured per the parameters
*/
protected PhysicalTable buildPhysicalTable(
PhysicalTableDefinition definition,
Set<FieldName> metricNames,
DimensionDictionary dimensionDictionary
) {
LinkedHashSet<Column> columns = Stream.concat(
// Load the dimension columns
definition.getDimensions().stream()
.map(DimensionConfig::getApiName)
.map(dimensionDictionary::findByApiName)
.map(DimensionColumn::new),
// And the metric columns
metricNames.stream()
.map(FieldName::asName)
.map(MetricColumn::new)
).collect(Collectors.toCollection(LinkedHashSet::new));

return new ConcretePhysicalTable(
definition.getName(),
definition.getGrain(),
columns,
definition.getLogicalToPhysicalNames(),
getDataSourceMetadataService()
);
}

/**
* Getter for the data source metadata service use for creating physical tables.
*
* @return the data source metadata service associated with the table loader
*/
protected DataSourceMetadataService getDataSourceMetadataService() {
return metadataService;
return buildDimensionSpanningTableGroup(apiMetrics, tableDefinitions, dictionaries);
}
}
60 changes: 60 additions & 0 deletions ...ain/java/com/yahoo/bard/webservice/data/config/table/ConcretePhysicalTableDefinition.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
// Copyright 2017 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE.md file distributed with this work for terms.
package com.yahoo.bard.webservice.data.config.table;

import com.yahoo.bard.webservice.data.config.ResourceDictionaries;
import com.yahoo.bard.webservice.data.config.dimension.DimensionConfig;
import com.yahoo.bard.webservice.data.config.names.FieldName;
import com.yahoo.bard.webservice.data.config.names.TableName;
import com.yahoo.bard.webservice.data.time.ZonedTimeGrain;
import com.yahoo.bard.webservice.metadata.DataSourceMetadataService;
import com.yahoo.bard.webservice.table.ConcretePhysicalTable;
import com.yahoo.bard.webservice.table.PhysicalTable;

import java.util.Collections;
import java.util.Optional;
import java.util.Set;

/**
* Holds the fields needed to define a Concrete Physical Table.
*/
public class ConcretePhysicalTableDefinition extends PhysicalTableDefinition {

/**
* Define a physical table using a zoned time grain.
*
* @param name The table name
* @param timeGrain The zoned time grain
* @param metricNames The Set of metric names on the table
* @param dimensionConfigs The dimension configurations
*/
public ConcretePhysicalTableDefinition(
TableName name,
ZonedTimeGrain timeGrain,
Set<FieldName> metricNames,
Set<? extends DimensionConfig> dimensionConfigs
) {
super(name, timeGrain, metricNames, dimensionConfigs);
}

@Override
public Set<TableName> getDependentTableNames() {
return Collections.emptySet();
}

@Override
public Optional<PhysicalTable> build(
ResourceDictionaries dictionaries,
DataSourceMetadataService metadataService
) {
return Optional.of(
new ConcretePhysicalTable(
getName(),
getTimeGrain(),
buildColumns(dictionaries.getDimensionDictionary()),
getLogicalToPhysicalNames(),
metadataService
)
);
}
}
Loading

0 comments on commit 4e58f37

Please sign in to comment.