Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare For Partial Data V2 #264

Merged
merged 7 commits into from
May 9, 2017
Merged

Prepare For Partial Data V2 #264

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
726d5d4
Prepare For Partial Data V2
QubitPi May 2, 2017
80325d6
address @cdeszaq's comments
QubitPi May 3, 2017
a5e6c01
address comments
QubitPi May 5, 2017
3135b53
address comments
QubitPi May 5, 2017
1408d64
address comments
QubitPi May 5, 2017
e5de67a
address comments
QubitPi May 5, 2017
e5f0c9e
fix comment error
QubitPi May 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ Current
-------
### Added:

- [Prepare For Partial Data V2](https://github.com/yahoo/fili/pull/264)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list things like, added more error message, added new feature flag, added new queryContext

- [Add MetricUnionCompositeTableDefinition](https://github.com/yahoo/fili/pull/258)

- [Add partition availability and table](https://github.com/yahoo/fili/pull/244)
Expand Down
17 changes: 15 additions & 2 deletions fili-core/src/main/java/com/yahoo/bard/webservice/druid/model/query/QueryContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import static com.yahoo.bard.webservice.druid.model.query.QueryContext.Param.PRIORITY;
import static com.yahoo.bard.webservice.druid.model.query.QueryContext.Param.QUERY_ID;
import static com.yahoo.bard.webservice.druid.model.query.QueryContext.Param.TIMEOUT;
import static com.yahoo.bard.webservice.druid.model.query.QueryContext.Param.UNCOVERED_INTERVALS_LIMIT;
import static com.yahoo.bard.webservice.druid.model.query.QueryContext.Param.USE_CACHE;

import com.fasterxml.jackson.annotation.JsonIgnore;
Expand Down Expand Up @@ -36,7 +37,9 @@ public enum Param {
USE_CACHE("useCache"),
POPULATE_CACHE("populateCache"),
BY_SEGMENT("bySegment"),
FINALIZE("finalize");
FINALIZE("finalize"),
UNCOVERED_INTERVALS_LIMIT("uncoveredIntervalsLimit")
;

private final String jsonName;

Expand All @@ -63,7 +66,8 @@ String getName() {
new SimpleImmutableEntry<>(USE_CACHE, Boolean.class),
new SimpleImmutableEntry<>(POPULATE_CACHE, Boolean.class),
new SimpleImmutableEntry<>(BY_SEGMENT, Boolean.class),
new SimpleImmutableEntry<>(FINALIZE, Boolean.class))
new SimpleImmutableEntry<>(FINALIZE, Boolean.class),
new SimpleImmutableEntry<>(UNCOVERED_INTERVALS_LIMIT, Number.class))
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)));

@JsonIgnore
Expand Down Expand Up @@ -208,6 +212,10 @@ public QueryContext withBySegment(Boolean bySegment) {
public QueryContext withFinalize(Boolean finalize) {
return withValue(FINALIZE, finalize);
}

public QueryContext withUncoveredIntervalsLimit(Integer uncoveredIntervalsLimit) {
return withValue(UNCOVERED_INTERVALS_LIMIT, uncoveredIntervalsLimit);
}
// CHECKSTYLE:ON

@JsonIgnore
Expand Down Expand Up @@ -255,6 +263,11 @@ public Boolean getFinalize() {
return (Boolean) contextMap.get(FINALIZE);
}

@JsonInclude(JsonInclude.Include.NON_NULL)
public Integer getUncoveredIntervalsLimit() {
return (Integer) contextMap.get(UNCOVERED_INTERVALS_LIMIT);
}

@Override
public boolean equals(Object o) {
if (o instanceof QueryContext) {
Expand Down
11 changes: 11 additions & 0 deletions fili-core/src/main/java/com/yahoo/bard/webservice/web/ErrorMessageFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,17 @@ public enum ErrorMessageFormat implements MessageFormatter {

RESULT_MAPPING_FAILURE(
"Error occurred while processing response data: %s"
),

DATA_AVAILABILITY_MISMATCH(
"Data availability expectation does not match with actual query result obtained from druid for the " +
"following intervals %s where druid does not have data"
),

TOO_MUCH_INTERVAL_MISSING(
"More than %s interval missing information received from druid, inspect if query " +
"expects more than %s missing intervals or increase " +
"uncoveredIntervalsLimit configuration value"
)
;

Expand Down
3 changes: 3 additions & 0 deletions fili-core/src/main/resources/moduleConfig.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,9 @@ bard__default_asyncAfter=never
# Flag to turn on case sensitive keys in keyvalue store
bard__case_sensitive_keys_enabled = false

# Sets the upper limit of the number of Druid uncovered interval. Default is -1 meaning no uncovered interval is allowed
bard__druid_uncovered_interval_limit = -1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't 0 be just as good for 'no uncovered interval'?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michael-mclawhorn @cdeszaq @garyluoex I vote for 0 but I'm open to either

Copy link
Collaborator

@garyluoex garyluoex May 5, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets change it to 0, I was thinking something that doesn't work. Change all check for partial data v2 enabled to be strictly greater than 0 also.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garyluoex @michael-mclawhorn I have changed it to 0. #267 has been update as well.


# The implementation of the com.yahoo.bard.webservice.logging.LogFormatter to use to format the RequestLog logging
# blocks. By default, the RequestLog is formatted as JSON.
bard__log_formatter_implementation=com.yahoo.bard.webservice.logging.JsonLogFormatter
2 changes: 0 additions & 2 deletions fili-core/src/test/resources/testApplicationConfig.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ bard__redis_namespace = test
# The channel on which RedisBroadcastChannel can publish/listen to messages
bard__redisbroadcastchannel_name = broadcast


# Don't start Partial data loading during tests
bard__druid_seg_loader_timer_delay = 60000

Expand All @@ -60,7 +59,6 @@ bard__lucene_index_path = ./target/tmp/
# data endpoint does not paginate by default.
bard__default_per_page = 10000


# Intersection reporting enabled or not.
bard__intersection_reporting_enabled = false

Expand Down