Skip to content

Commit

Permalink
Actually use the current time range for querying sections
Browse files Browse the repository at this point in the history
Doh!
Without this, we keep inferring modes for the same sections over and over.

Analysis below.

------------------------------------------------------------------------

Aha! we now have two copies of the analysis results

```
In [19]: edb.get_analysis_timeseries_db().find({"metadata.key": "inference/prediction"},
    ...:  {"data.predicted_mode_map": "1"}).count()
Out[19]: 137

In [20]: edb.get_analysis_timeseries_db().find({"metadata.key": "analysis/inferred_secti
    ...: on"}, {"data.sensed_mode": "1"}).count()
Out[20]: 137

In [21]: edb.get_analysis_timeseries_db().find({"metadata.key": "analysis/inferred_secti
    ...: on"}, {"data.sensed_mode": "1"}).count()
Out[21]: 274

In [22]: edb.get_analysis_timeseries_db().find({"metadata.key": "inference/prediction"},
    ...:  {"data.predicted_mode_map": "1"}).count()
Out[22]: 274
```

Let's figure out why?!

Ok so when we get to the MODE_INFERENCE stage, the `start_ts` is set correctly.

```
2018-02-26 10:03:41,381:INFO:140735691387712:For stage PipelineStages.MODE_INFERENCE, start_ts = 2016-10-08T01:34:35.600000
```

But we still read all the sections again. Ah, the query doesn't appear to
involve the timestamp at all.

```
2018-02-26 10:03:41,383:DEBUG:140735691387712:curr_query = {'user_id': UUID('ea07ac60-0724-4eec-aab2-8240fcbf74ce'), '$or': [{'metadata.key': 'analysis/cleaned_section'}]}, sort_key = metadata.write_ts
```
  • Loading branch information
shankari committed Feb 26, 2018
1 parent 84e9f35 commit 1d30f32
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ def getLastSectionDone(self):
def runPredictionPipeline(self, user_id, timerange):
self.ts = esta.TimeSeries.get_time_series(user_id)
self.toPredictSections = esda.get_entries(esda.CLEANED_SECTION_KEY, user_id,
time_query=None)
time_query=timerange)
if (len(self.toPredictSections) == 0):
logging.debug("len(toPredictSections) == 0, early return")
assert self.last_section_done is None, ("self.last_section_done == %s, expecting None" % \
Expand Down

0 comments on commit 1d30f32

Please sign in to comment.