Fix batch queries missing all fields after the first nil field

[nathanielc]

Fixes #1294 , @hraftery do you have the ability to build and test out this PR to confirm it fixes your issue?

• Rebased/mergable
• Tests pass
• CHANGELOG.md updated

[hraftery]

@nathanielc I do now! I've set up a dev environment and run my tests both with and without the PR. Initially looked good, but ran into problems. Certainly appears to fix the first issue in #1294 :

• SELECT * in influx returns all tags and fields, including time. In kapacitor it only returns the first field by alphabetical order!

I then went back and tried it against the original report. Also looking good.

Observe:

~/gocode/bin/influx -execute "drop database d"
~/gocode/bin/influx -execute "create database d"
~/gocode/bin/influx -execute 'INSERT m a=1 1' -database d
~/gocode/bin/influx -execute 'INSERT m b=2 2' -database d
~/gocode/bin/influx -execute 'INSERT m,t1=3 a=3,b=3 3' -database d
~/gocode/bin/influx -execute 'INSERT m,t2=4 a=4,b=4 4' -database d
~/gocode/bin/influx -execute 'INSERT m,t1=5 a=5,b=5 5' -database d
~/gocode/bin/influx -execute 'INSERT m,t1=6 b=6 6' -database d
~/gocode/bin/influx -execute 'INSERT m,t2=7 b=7 7' -database d
~/gocode/bin/influx -execute 'INSERT m,t1=8 b=8 8' -database d
~/gocode/bin/influx -execute 'INSERT m,t1=9 b=9 9' -database d
~/gocode/bin/influx -execute 'INSERT m b=10 10' -database d
~/gocode/bin/influx -execute 'INSERT m a=11 11' -database d
~/gocode/bin/influx -execute 'SELECT * FROM m' -database d

Which gives:

name: m
time a  b  t1 t2
---- -  -  -- --
1    1        
2       2     
3    3  3  3  
4    4  4     4
5    5  5  5  
6       6  6  
7       7     7
8       8  8  
9       9  9  
10      10    
11   11   

Now:

~/gocode/bin/kapacitor define record_issue -type stream -tick record_issue.tick  -dbrp d.autogen
rid=$(~/gocode/bin/kapacitor record query -query $'SELECT * FROM "d"."autogen"."m"' -type stream)
~/gocode/bin/kapacitor replay -task record_issue -recording $rid -rec-time

Then /var/log/kapacitor/kapacitor.log shows:

[task_master:d5c5f8e5-b4fd-4cfd-832c-1a3ef849ecf0] 2017/04/15 08:29:43 I! opened
[task_master:d5c5f8e5-b4fd-4cfd-832c-1a3ef849ecf0] 2017/04/15 08:29:43 I! Started task: record_issue
[record_issue:log2] 2017/04/15 08:29:43 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"a":1},"Time":"1970-01-01T00:00:00.000000001Z"}
[record_issue:log2] 2017/04/15 08:29:43 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":2},"Time":"1970-01-01T00:00:00.000000002Z"}
[record_issue:log2] 2017/04/15 08:29:43 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"a":3,"b":3,"t1":"3"},"Time":"1970-01-01T00:00:00.000000003Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"a":4,"b":4,"t2":"4"},"Time":"1970-01-01T00:00:00.000000004Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"a":5,"b":5,"t1":"5"},"Time":"1970-01-01T00:00:00.000000005Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":6,"t1":"6"},"Time":"1970-01-01T00:00:00.000000006Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":7,"t2":"7"},"Time":"1970-01-01T00:00:00.000000007Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":8,"t1":"8"},"Time":"1970-01-01T00:00:00.000000008Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":9,"t1":"9"},"Time":"1970-01-01T00:00:00.000000009Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"b":10},"Time":"1970-01-01T00:00:00.00000001Z"}
[record_issue:log2] 2017/04/15 08:29:44 I! RECORD_ISSUE {"Name":"m","Database":"d","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"a":11},"Time":"1970-01-01T00:00:00.000000011Z"}
[task_master:d5c5f8e5-b4fd-4cfd-832c-1a3ef849ecf0] 2017/04/15 08:29:44 I! Stopped task: record_issue
[task_master:d5c5f8e5-b4fd-4cfd-832c-1a3ef849ecf0] 2017/04/15 08:29:44 I! closed

So all data accounted for and all in order, but all tags have become fields. So next I tried to test the other main sub-issue:

• SELECT field1 ... GROUP BY * does return the tags, but ordered by tag values, not by time. This makes replaying useless.

by recording with GROUP BY:

~/gocode/bin/kapacitor define record_issue -type stream -tick record_issue.tick  -dbrp d.autogen
rid=$(~/gocode/bin/kapacitor record query -query $'SELECT * FROM "d"."autogen"."m" GROUP BY *' -type stream)

And the call never returns. kapacitord locks up and needs a kill -9 and influxd starts spitting out [I] 2017-04-14T22:38:51Z Post http://localhost:9092/write?consistency=&db=telegraf&precision=ns&rp=autogen: net/http: request canceled (Client.Timeout exceeded while awaiting headers) service=subscriber.

Alas, it looks like a small step forward and a big step backward. Maybe there was a reason for that break statement!

Further info: it's the ~/gocode/bin/influx -execute 'INSERT m,t1=5 a=5,b=5 5' -database d point that causes the lock up. With the first 4 points, no problems. As soon as I try to put that 5th point in to test the out of order issue, it locks up. In fact, GROUP BY t2 is enough.

[nathanielc]

@hraftery Thanks for the detailed writeup, I'll take a look at the second set of issues.

[nathanielc]

@hraftery I have fixed the second issue. Basically the first issue was masking another hidden bug that caused an infinite loop in the ordering logic for recordings. At this point your test (using GROUP BY *) should work, and the recording should contain all the data in order by time. Can you give it a try when you get a chance?

[hraftery]

Looks great. Quite a process to update and test, but results look good.

[nathanielc]

@hraftery Thanks for going through the effort. I'll get this merged and cut a new beta release of 1.3.0