Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.9.5-nightly-827c513] cannot unmarshal array into Go value #4550

Closed
sergzin opened this issue Oct 23, 2015 · 10 comments
Closed

[0.9.5-nightly-827c513] cannot unmarshal array into Go value #4550

sergzin opened this issue Oct 23, 2015 · 10 comments
Assignees
@dgnorton

Comments

@sergzin
Copy link

sergzin commented Oct 23, 2015

I am running an influxdb cluster with the following version:

InfluxDB starting, version 0.9.5-nightly-827c513, branch master, commit 827c51384c9b77343cbfc6b3f2e5c1e8bf608bee, built '2015-10-23T04:00:42+0000'
Go version go1.5.1, GOMAXPROCS set to 32

three out of 4 nodes return the following output for show measurements command

# /opt/influxdb/influx -database observer
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.9.5-nightly-827c513
InfluxDB shell 0.9.5-nightly-827c513
>
> show measurements
ERR: json: cannot unmarshal array into Go value of type tsdb.MapperOutputJSON
> show tag keys from firewall
ERR: write message type: write tcp 172.31.141.195:33254->172.31.141.196:8088: use of closed network connection
> show tag keys from traffic
ERR: write message type: write tcp 172.31.141.195:33277->172.31.141.196:8088: use of closed network connection

some other queries works fine

> show tag values from firewall with key=name where type='policer'
name: nameTagValues
-------------------
name
AAA-POLICER-MGT-RADIUS
AAA-POLICER-MGT-TACACS
DOMAIN-POLICER-MGT-DOMAIN
> show tag values from firewall with key=type
name: typeTagValues
-------------------
type
counter
policer

I tried to restart all nodes but it didn't help.
@otoolep
Copy link
Contributor

otoolep commented Oct 23, 2015

Any ideas @dgnorton ?

@dgnorton
Copy link
Contributor

@sergzin can you post the influxd log?

@beckettsean
Copy link
Contributor

@sergzin can you also post the results of SHOW SERVERS?

@otoolep otoolep assigned otoolep and dgnorton and unassigned otoolep Oct 24, 2015
@otoolep otoolep mentioned this issue Oct 24, 2015
Closed
@otoolep
Copy link
Contributor

otoolep commented Oct 24, 2015

#4561 is also a report of this against a 0.9.5 nightly.

@priyadarsh
Copy link

Is it safe to assume that the data being written is getting stored properly but only the retrieving part is failing?

@sergzin
Copy link
Author

sergzin commented Oct 26, 2015

The issue started after I upgraded to 0.9.5-nightly.
I wanted to try new aggregation query syntax select time,max(in_bps) ...

Here is my environment:
There were 3 servers in the beginning. I added in September 2 more. Then I have to delete one due to failed disk with DROP SERVER command.

> show servers
id      cluster_addr            raft    raft-leader
1       172.31.141.195:8088     true    true
2       172.31.141.197:8088     true    false
3       172.31.141.196:8088     true    false
4       172.31.141.199:8088     false   false

> show shards
name: observer
--------------
id      start_time              end_time                expiry_time             owners
1       2015-06-23T00:00:00Z    2015-06-24T00:00:00Z    2015-08-23T00:00:00Z    3,1,2
2       2015-06-24T00:00:00Z    2015-06-25T00:00:00Z    2015-08-24T00:00:00Z    2,3,1
3       2015-06-25T00:00:00Z    2015-06-26T00:00:00Z    2015-08-25T00:00:00Z    3,1,2
4       2015-06-26T00:00:00Z    2015-06-27T00:00:00Z    2015-08-26T00:00:00Z    1,2,3
5       2015-06-27T00:00:00Z    2015-06-28T00:00:00Z    2015-08-27T00:00:00Z    2,3,1
6       2015-06-28T00:00:00Z    2015-06-29T00:00:00Z    2015-08-28T00:00:00Z    3,1,2
7       2015-06-29T00:00:00Z    2015-06-30T00:00:00Z    2015-08-29T00:00:00Z    1,2,3
8       2015-06-30T00:00:00Z    2015-07-01T00:00:00Z    2015-08-30T00:00:00Z    2,3,1
9       2015-07-01T00:00:00Z    2015-07-02T00:00:00Z    2015-08-31T00:00:00Z    3,1,2
10      2015-07-02T00:00:00Z    2015-07-03T00:00:00Z    2015-09-01T00:00:00Z    1,2,3
11      2015-07-03T00:00:00Z    2015-07-04T00:00:00Z    2015-09-02T00:00:00Z    2,3,1
12      2015-07-04T00:00:00Z    2015-07-05T00:00:00Z    2015-09-03T00:00:00Z    3,1,2
13      2015-07-05T00:00:00Z    2015-07-06T00:00:00Z    2015-09-04T00:00:00Z    1,2,3
14      2015-07-06T00:00:00Z    2015-07-07T00:00:00Z    2015-09-05T00:00:00Z    2,3,1
15      2015-07-07T00:00:00Z    2015-07-08T00:00:00Z    2015-09-06T00:00:00Z    3,1,2
16      2015-07-08T00:00:00Z    2015-07-09T00:00:00Z    2015-09-07T00:00:00Z    1,2,3
17      2015-07-09T00:00:00Z    2015-07-10T00:00:00Z    2015-09-08T00:00:00Z    2,3,1
18      2015-07-10T00:00:00Z    2015-07-11T00:00:00Z    2015-09-09T00:00:00Z    3,1,2
19      2015-07-11T00:00:00Z    2015-07-12T00:00:00Z    2015-09-10T00:00:00Z    1,2,3
20      2015-07-12T00:00:00Z    2015-07-13T00:00:00Z    2015-09-11T00:00:00Z    2,3,1
21      2015-07-13T00:00:00Z    2015-07-14T00:00:00Z    2015-09-12T00:00:00Z    3,1,2
22      2015-07-14T00:00:00Z    2015-07-15T00:00:00Z    2015-09-13T00:00:00Z    1,2,3
23      2015-07-15T00:00:00Z    2015-07-16T00:00:00Z    2015-09-14T00:00:00Z    2,3,1
24      2015-07-16T00:00:00Z    2015-07-17T00:00:00Z    2015-09-15T00:00:00Z    3,1,2
25      2015-07-17T00:00:00Z    2015-07-18T00:00:00Z    2015-09-16T00:00:00Z    1,2,3
26      2015-07-18T00:00:00Z    2015-07-19T00:00:00Z    2015-09-17T00:00:00Z    2,3,1
27      2015-07-19T00:00:00Z    2015-07-20T00:00:00Z    2015-09-18T00:00:00Z    3,1,2
28      2015-07-20T00:00:00Z    2015-07-21T00:00:00Z    2015-09-19T00:00:00Z    1,2,3
29      2015-07-21T00:00:00Z    2015-07-22T00:00:00Z    2015-09-20T00:00:00Z    2,3,1
30      2015-07-22T00:00:00Z    2015-07-23T00:00:00Z    2015-09-21T00:00:00Z    3,1,2
31      2015-07-23T00:00:00Z    2015-07-24T00:00:00Z    2015-09-22T00:00:00Z    1,2,3
32      2015-07-24T00:00:00Z    2015-07-25T00:00:00Z    2015-09-23T00:00:00Z    2,3,1
33      2015-07-25T00:00:00Z    2015-07-26T00:00:00Z    2015-09-24T00:00:00Z    3,1,2
34      2015-07-26T00:00:00Z    2015-07-27T00:00:00Z    2015-09-25T00:00:00Z    3,1,2
35      2015-07-27T00:00:00Z    2015-07-28T00:00:00Z    2015-09-26T00:00:00Z    1,2,3
36      2015-07-28T00:00:00Z    2015-07-29T00:00:00Z    2015-09-27T00:00:00Z    2,3,1
37      2015-07-29T00:00:00Z    2015-07-30T00:00:00Z    2015-09-28T00:00:00Z    3,1,2
38      2015-07-30T00:00:00Z    2015-07-31T00:00:00Z    2015-09-29T00:00:00Z    1,2,3
39      2015-07-31T00:00:00Z    2015-08-01T00:00:00Z    2015-09-30T00:00:00Z    2,3,1
40      2015-08-01T00:00:00Z    2015-08-02T00:00:00Z    2015-10-01T00:00:00Z    3,1,2
41      2015-08-02T00:00:00Z    2015-08-03T00:00:00Z    2015-10-02T00:00:00Z    1,2,3
42      2015-08-03T00:00:00Z    2015-08-04T00:00:00Z    2015-10-03T00:00:00Z    2,3,1
43      2015-08-04T00:00:00Z    2015-08-05T00:00:00Z    2015-10-04T00:00:00Z    3,1,2
44      2015-08-05T00:00:00Z    2015-08-06T00:00:00Z    2015-10-05T00:00:00Z    1,2,3
45      2015-08-06T00:00:00Z    2015-08-07T00:00:00Z    2015-10-06T00:00:00Z    2,3,1
46      2015-08-07T00:00:00Z    2015-08-08T00:00:00Z    2015-10-07T00:00:00Z    3,1,2
47      2015-08-08T00:00:00Z    2015-08-09T00:00:00Z    2015-10-08T00:00:00Z    1,2,3
48      2015-08-09T00:00:00Z    2015-08-10T00:00:00Z    2015-10-09T00:00:00Z    2,3,1
49      2015-08-10T00:00:00Z    2015-08-11T00:00:00Z    2015-10-10T00:00:00Z    3,1,2
50      2015-08-11T00:00:00Z    2015-08-12T00:00:00Z    2015-10-11T00:00:00Z    1,2,3
51      2015-08-12T00:00:00Z    2015-08-13T00:00:00Z    2015-10-12T00:00:00Z    2,3,1
52      2015-08-13T00:00:00Z    2015-08-14T00:00:00Z    2015-10-13T00:00:00Z    3,1,2
53      2015-08-14T00:00:00Z    2015-08-15T00:00:00Z    2015-10-14T00:00:00Z    1,2,3
54      2015-08-15T00:00:00Z    2015-08-16T00:00:00Z    2015-10-15T00:00:00Z    2,3,1
55      2015-08-16T00:00:00Z    2015-08-17T00:00:00Z    2015-10-16T00:00:00Z    3,1,2
56      2015-08-17T00:00:00Z    2015-08-18T00:00:00Z    2015-10-17T00:00:00Z    1,2,3
57      2015-08-18T00:00:00Z    2015-08-19T00:00:00Z    2015-10-18T00:00:00Z    2,3,1
58      2015-08-19T00:00:00Z    2015-08-20T00:00:00Z    2015-10-19T00:00:00Z    3,1,2
59      2015-08-20T00:00:00Z    2015-08-21T00:00:00Z    2015-10-20T00:00:00Z    1,2,3
60      2015-08-21T00:00:00Z    2015-08-22T00:00:00Z    2015-10-21T00:00:00Z    2,3,1
61      2015-08-22T00:00:00Z    2015-08-23T00:00:00Z    2015-10-22T00:00:00Z    3,1,2
62      2015-08-23T00:00:00Z    2015-08-24T00:00:00Z    2015-10-23T00:00:00Z    1,2,3
63      2015-08-24T00:00:00Z    2015-08-25T00:00:00Z    2015-10-24T00:00:00Z    3,1,2
64      2015-08-25T00:00:00Z    2015-08-26T00:00:00Z    2015-10-25T00:00:00Z    2,3,1
65      2015-08-26T00:00:00Z    2015-08-27T00:00:00Z    2015-10-26T00:00:00Z    1,2,3
66      2015-08-27T00:00:00Z    2015-08-28T00:00:00Z    2015-10-27T00:00:00Z    3,1,2
67      2015-08-28T00:00:00Z    2015-08-29T00:00:00Z    2015-10-28T00:00:00Z    2,3,1
68      2015-08-29T00:00:00Z    2015-08-30T00:00:00Z    2015-10-29T00:00:00Z    1,2,3
69      2015-08-30T00:00:00Z    2015-08-31T00:00:00Z    2015-10-30T00:00:00Z    3,1,2
70      2015-08-31T00:00:00Z    2015-09-01T00:00:00Z    2015-10-31T00:00:00Z    2,3,1
71      2015-09-01T00:00:00Z    2015-09-02T00:00:00Z    2015-11-01T00:00:00Z    1,2,3
72      2015-09-02T00:00:00Z    2015-09-03T00:00:00Z    2015-11-02T00:00:00Z    3,1,2
73      2015-09-03T00:00:00Z    2015-09-04T00:00:00Z    2015-11-03T00:00:00Z    2,3,1
74      2015-09-04T00:00:00Z    2015-09-05T00:00:00Z    2015-11-04T00:00:00Z    1,2,3
75      2015-09-05T00:00:00Z    2015-09-06T00:00:00Z    2015-11-05T00:00:00Z    3,1,2
76      2015-09-06T00:00:00Z    2015-09-07T00:00:00Z    2015-11-06T00:00:00Z    2,3,1
77      2015-09-07T00:00:00Z    2015-09-08T00:00:00Z    2015-11-07T00:00:00Z    1,2,3
78      2015-09-08T00:00:00Z    2015-09-09T00:00:00Z    2015-11-08T00:00:00Z    3,1,2
79      2015-09-09T00:00:00Z    2015-09-10T00:00:00Z    2015-11-09T00:00:00Z    2,3,1
80      2015-09-10T00:00:00Z    2015-09-11T00:00:00Z    2015-11-10T00:00:00Z    1,2,3
81      2015-09-11T00:00:00Z    2015-09-12T00:00:00Z    2015-11-11T00:00:00Z    3,1,2
82      2015-09-12T00:00:00Z    2015-09-13T00:00:00Z    2015-11-12T00:00:00Z    2,3,1
83      2015-09-13T00:00:00Z    2015-09-14T00:00:00Z    2015-11-13T00:00:00Z    1,2,3
84      2015-09-14T00:00:00Z    2015-09-15T00:00:00Z    2015-11-14T00:00:00Z    3,1,2
85      2015-09-15T00:00:00Z    2015-09-16T00:00:00Z    2015-11-15T00:00:00Z    2,3,1
86      2015-09-16T00:00:00Z    2015-09-17T00:00:00Z    2015-11-16T00:00:00Z    1,2,3
87      2015-09-17T00:00:00Z    2015-09-18T00:00:00Z    2015-11-17T00:00:00Z    3,1,2
88      2015-09-18T00:00:00Z    2015-09-19T00:00:00Z    2015-11-18T00:00:00Z    2,3,1
89      2015-09-19T00:00:00Z    2015-09-20T00:00:00Z    2015-11-19T00:00:00Z    1,2,3
90      2015-09-20T00:00:00Z    2015-09-21T00:00:00Z    2015-11-20T00:00:00Z    3,1,2
91      2015-09-21T00:00:00Z    2015-09-22T00:00:00Z    2015-11-21T00:00:00Z    2,3,1
92      2015-09-22T00:00:00Z    2015-09-23T00:00:00Z    2015-11-22T00:00:00Z    1,2,3
93      2015-09-23T00:00:00Z    2015-09-24T00:00:00Z    2015-11-23T00:00:00Z    3,1,2
94      2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-11-24T00:00:00Z    2,3,1
95      2015-09-25T00:00:00Z    2015-09-26T00:00:00Z    2015-11-25T00:00:00Z    1,2,3
99      2015-09-26T00:00:00Z    2015-09-27T00:00:00Z    2015-11-26T00:00:00Z    2,3,1
103     2015-09-27T00:00:00Z    2015-09-28T00:00:00Z    2015-11-27T00:00:00Z    2,3,1
107     2015-09-28T00:00:00Z    2015-09-29T00:00:00Z    2015-11-28T00:00:00Z    2,3,1
111     2015-09-29T00:00:00Z    2015-09-30T00:00:00Z    2015-11-29T00:00:00Z    1,2,3
115     2015-09-30T00:00:00Z    2015-10-01T00:00:00Z    2015-11-30T00:00:00Z    1,2,3
119     2015-10-01T00:00:00Z    2015-10-02T00:00:00Z    2015-12-01T00:00:00Z    2,3,1
125     2015-10-02T00:00:00Z    2015-10-03T00:00:00Z    2015-12-02T00:00:00Z    4,1
134     2015-10-03T00:00:00Z    2015-10-04T00:00:00Z    2015-12-03T00:00:00Z    4,1
143     2015-10-04T00:00:00Z    2015-10-05T00:00:00Z    2015-12-04T00:00:00Z    1,2
152     2015-10-05T00:00:00Z    2015-10-06T00:00:00Z    2015-12-05T00:00:00Z    1,2,3
161     2015-10-06T00:00:00Z    2015-10-07T00:00:00Z    2015-12-06T00:00:00Z    1,2,3
168     2015-10-07T00:00:00Z    2015-10-08T00:00:00Z    2015-12-07T00:00:00Z    1,2,3
175     2015-10-08T00:00:00Z    2015-10-09T00:00:00Z    2015-12-08T00:00:00Z    1,2,3
182     2015-10-09T00:00:00Z    2015-10-10T00:00:00Z    2015-12-09T00:00:00Z    1,2,3
189     2015-10-10T00:00:00Z    2015-10-11T00:00:00Z    2015-12-10T00:00:00Z    1,2,3
196     2015-10-11T00:00:00Z    2015-10-12T00:00:00Z    2015-12-11T00:00:00Z    1,2,3
203     2015-10-12T00:00:00Z    2015-10-13T00:00:00Z    2015-12-12T00:00:00Z    1,2,3
210     2015-10-13T00:00:00Z    2015-10-14T00:00:00Z    2015-12-13T00:00:00Z    1,2,3
217     2015-10-14T00:00:00Z    2015-10-15T00:00:00Z    2015-12-14T00:00:00Z    1,2,3
224     2015-10-15T00:00:00Z    2015-10-16T00:00:00Z    2015-12-15T00:00:00Z    1,2,3
231     2015-10-16T00:00:00Z    2015-10-17T00:00:00Z    2015-12-16T00:00:00Z    1,2,3
238     2015-10-17T00:00:00Z    2015-10-18T00:00:00Z    2015-12-17T00:00:00Z    1,2,3
245     2015-10-18T00:00:00Z    2015-10-19T00:00:00Z    2015-12-18T00:00:00Z    2,3,4
252     2015-10-19T00:00:00Z    2015-10-20T00:00:00Z    2015-12-19T00:00:00Z    3,4
260     2015-10-20T00:00:00Z    2015-10-21T00:00:00Z    2015-12-20T00:00:00Z    1,2,3
267     2015-10-21T00:00:00Z    2015-10-22T00:00:00Z    2015-12-21T00:00:00Z    1,2,3
274     2015-10-22T00:00:00Z    2015-10-23T00:00:00Z    2015-12-22T00:00:00Z    2,3,4
281     2015-10-23T00:00:00Z    2015-10-24T00:00:00Z    2015-12-23T00:00:00Z    2,3,4
288     2015-10-24T00:00:00Z    2015-10-25T00:00:00Z    2015-12-24T00:00:00Z    4,1,2
290     2015-10-25T00:00:00Z    2015-10-26T00:00:00Z    2015-12-25T00:00:00Z    2,3,4
292     2015-10-26T00:00:00Z    2015-10-27T00:00:00Z    2015-12-26T00:00:00Z    2,3,4

# cat /etc/opt/influxdb/influxdb.conf
reporting-disabled = true
[registration]
  enabled = false
[meta]
  dir = "/data/influxdb/meta"
  hostname = "172.31.141.195"
  bind-address = ":8088"
  retention-autocreate = true
  election-timeout = "1s"
  heartbeat-timeout = "1s"
  leader-lease-timeout = "500ms"
  commit-timeout = "50ms"
  peers = ["172.31.141.195:8088", "172.31.141.196:8088", "172.31.141.197:8088"]
[data]
  dir = "/data/influxdb/data"
  max-wal-size = 1048576000 # 1G
  wal-flush-interval = "10m" # Maximum time data can sit in WAL before a flush.
  wal-partition-flush-delay = "2s" # The delay time between each WAL partition being flushed.
  wal-dir = "/data/influxdb/wal"
  wal-enable-logging = true
[cluster]
  shard-writer-timeout = "10s" # The time within which a shard must respond to write.
  write-timeout = "5s" # The time within which a write operation must complete on the cluster.
[retention]
  enabled = true
  check-interval = "30m"
[monitor]
  store-enabled = false # Whether to record statistics internally.
  store-database = "_internal" # The destination database for recorded statistics
  store-interval = "10s" # The interval at which to record statistics
[admin]
  enabled = true
  bind-address = ":8083"
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
[[graphite]]
  enabled = true
  database = "graphite"
  bind-address = ":2003"
  protocol = "tcp"
  consistency-level = "one"
  name-separator = "."
[collectd]
  enabled = false
[opentsdb]
  enabled = false
[[udp]]
  enabled = false
[continuous_queries]
  log-enabled = true
  enabled = true
  recompute-previous-n = 2
  recompute-no-older-than = "10m"
  compute-runs-per-interval = 10
  compute-no-more-than = "2m"
[hinted-handoff]
  enabled = true
  dir = "/data/influxdb/hh"
  max-size = 1073741824
  max-age = "168h"
  retry-rate-limit = 0
  retry-interval = "1s"
  retry-max-interval = "1m"

The following query works fine on all nodes.

/opt/influxdb/influx -database observer -host 172.31.141.199 -execute "select * from traffic where if_id='130' and time > now() - 1h"
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
name: traffic
-------------
time                    ae_if_id        if_id   in_bps                  out_bps                 phy_if_id       router
1445864739000000000     2985902         130     5.65870442107e+09       6.99972102942e+08       0               7
1445865038000000000     2985902         130     5.6335774686e+09        7.33178417017e+08       0               7
1445865345000000000     2985902         130     5.44026585931e+09       7.30231255531e+08       0               7
1445865643000000000     2985902         130     5.78437244838e+09       7.5191753506e+08        0               7
1445865943000000000     2985902         130     5.82072042477e+09       7.79557631013e+08       0               7
1445866242000000000     2985902         130     5.88209280407e+09       7.6136775414e+08        0               7
1445866547000000000     2985902         130     5.73250201136e+09       7.40687404485e+08       0               7
1445866846000000000     2985902         130     5.88932236193e+09       7.88574859104e+08       0               7
1445867140000000000     2985902         130     6.22608571078e+09       7.7124359298e+08        0               7
1445867447000000000     2985902         130     5.8825954277e+09        7.46991306632e+08       0               7
1445867742000000000     2985902         130     6.1448840163e+09        7.57241877803e+08       0               7
1445868045000000000     2985902         130     5.88382954397e+09       7.20051266851e+08       0               7

If I query for tags I get the following output.

Query to server 172.31.141.196:

> show tag keys from traffic
ERR: write message type: write tcp 172.31.141.196:52740->172.31.141.195:8088: use of closed network connection

InfluxDB log on machine 196:

Oct 26 15:29:35 nl-ams02c-tsd02 influxd[20876]: [query] 2015/10/26 15:29:35 SHOW TAG KEYS FROM "observer"."daily".traffic
Oct 26 15:29:35 nl-ams02c-tsd02 influxd[20876]: [http] 2015/10/26 15:29:35 172.31.141.195 - - [26/Oct/2015:15:29:35 +0100] GET /query?db=observer&epoch=ns&q=show+tag+keys+from+traffic HTTP/1.1 200 137 - InfluxDBShell/0.9.5-nightly-827c513 fb3ef3db-7bed-11e5-a188-000000000000 2.820334ms

Log on machine 195:

Oct 26 15:29:35 nl-ams02c-tsd01 influxd[22065]: [tcp] 2015/10/26 15:29:35 accept remote connection from 172.31.141.196:52740
Oct 26 15:29:35 nl-ams02c-tsd01 influxd[22065]: [tcp] 2015/10/26 15:29:35 accept remote connection from 172.31.141.196:52741
Oct 26 15:29:35 nl-ams02c-tsd01 influxd[22065]: [tcp] 2015/10/26 15:29:35 close remote connection from 172.31.141.196:52741
Oct 26 15:29:35 nl-ams02c-tsd01 influxd[22065]: [tcp] 2015/10/26 15:29:35 close remote connection from 172.31.141.196:52740

@sergzin
Copy link
Author

sergzin commented Oct 26, 2015

Another might be interesting example from logs on all servers:

InfluxDB CLI:

> show measurements
ERR: json: cannot unmarshal array into Go value of type tsdb.MapperOutputJSON
> show measurements
ERR: json: cannot unmarshal array into Go value of type tsdb.MapperOutputJSON
> show measurements
ERR: json: cannot unmarshal array into Go value of type tsdb.MapperOutputJSON
> show tag keys from firewall
ERR: write message type: write tcp 172.31.141.195:54264->172.31.141.197:8088: use of closed network connection
>

Query log on server .195:

ct 26 16:36:44 nl-ams02c-tsd01 influxd[22065]: [query] 2015/10/26 16:36:44 SHOW MEASUREMENTS
Oct 26 16:36:44 nl-ams02c-tsd01 influxd[22065]: [http] 2015/10/26 16:36:44 172.31.141.195 - - [26/Oct/2015:16:36:44 +0100] GET /query?db=observer&epoch=ns&q=show+measurements HTTP/1.1 200 115 - InfluxDBShell/0.9.5-nightly-827c513 5ca43515-7bf7-11e5-be50-000000000000 7.789705ms


Oct 26 16:36:58 nl-ams02c-tsd01 influxd[22065]: [query] 2015/10/26 16:36:58 SHOW MEASUREMENTS
Oct 26 16:36:58 nl-ams02c-tsd01 influxd[22065]: [http] 2015/10/26 16:36:58 172.31.141.195 - - [26/Oct/2015:16:36:58 +0100] GET /query?db=observer&epoch=ns&q=show+measurements HTTP/1.1 200 115 - InfluxDBShell/0.9.5-nightly-827c513 64e06343-7bf7-11e5-be6f-000000000000 7.947113ms

Oct 26 16:38:12 nl-ams02c-tsd01 influxd[22065]: [query] 2015/10/26 16:38:12 SHOW MEASUREMENTS
Oct 26 16:38:12 nl-ams02c-tsd01 influxd[22065]: [http] 2015/10/26 16:38:12 172.31.141.195 - - [26/Oct/2015:16:38:12 +0100] GET /query?db=observer&epoch=ns&q=show+measurements HTTP/1.1 200 115 - InfluxDBShell/0.9.5-nightly-827c513 9101279f-7bf7-11e5-800c-000000000000 8.273568ms

InfluxDB log on .196:

Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:45233
Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:45236
Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:45237
Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:45233
Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:45237
Oct 26 16:38:26 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:45236



Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45172: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45172
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45175: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45175
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45171: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45171
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45176: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45176
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45159: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45159
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45217: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45217
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45219: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45219
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45221: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45221
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.196:8088->172.31.141.195:45222: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd02 influxd[20876]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:45222

InfluxDB log on .197:

Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 accept remote connection from 172.31.141.195:54176
Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 accept remote connection from 172.31.141.195:54178
Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 accept remote connection from 172.31.141.195:54180
Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 close remote connection from 172.31.141.195:54176
Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 close remote connection from 172.31.141.195:54178
Oct 26 16:36:34 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:34 close remote connection from 172.31.141.195:54180
Oct 26 16:36:44 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:44 accept remote connection from 172.31.141.195:54189
Oct 26 16:36:44 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:36:44 accept remote connection from 172.31.141.195:54192




Oct 26 16:38:12 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:12 accept remote connection from 172.31.141.195:54250
Oct 26 16:38:26 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:54264
Oct 26 16:38:26 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:54267
Oct 26 16:38:26 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:54264
Oct 26 16:38:26 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:54267
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.197:8088->172.31.141.195:54192: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:54192
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.197:8088->172.31.141.195:54189: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:54189
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.197:8088->172.31.141.195:54250: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd03 influxd[27411]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:54250

InfluxDB log on .199:

Oct 26 16:36:34 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:34 accept remote connection from 172.31.141.195:59441
Oct 26 16:36:34 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:34 close remote connection from 172.31.141.195:59441
Oct 26 16:36:34 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:34 accept remote connection from 172.31.141.195:59445
Oct 26 16:36:34 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:34 close remote connection from 172.31.141.195:59445
Oct 26 16:36:44 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:44 accept remote connection from 172.31.141.195:59452
Oct 26 16:36:44 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:44 accept remote connection from 172.31.141.195:59454
Oct 26 16:36:44 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:44 accept remote connection from 172.31.141.195:59457




Oct 26 16:36:58 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:58 accept remote connection from 172.31.141.195:59470
Oct 26 16:36:58 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:36:58 accept remote connection from 172.31.141.195:59469



Oct 26 16:38:12 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:12 accept remote connection from 172.31.141.195:59516



Oct 26 16:38:26 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:26 accept remote connection from 172.31.141.195:59530
Oct 26 16:38:26 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:26 close remote connection from 172.31.141.195:59530


Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59469: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59470: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59470
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59469
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59452: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59452
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59454: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59457: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59457
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59454
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 unable to read type-length-value read message type: read tcp 172.31.141.199:8088->172.31.141.195:59516: read: connection reset by peer
Oct 26 16:38:33 nl-ams02c-tsd05 influxd[43186]: [tcp] 2015/10/26 16:38:33 close remote connection from 172.31.141.195:59516

After SHOW MEASUREMENTS commands I see only accept message in log. There are no close message.

@sergzin
Copy link
Author

sergzin commented Oct 28, 2015

I see similar issues with test database if replication factor less than number of servers in cluster.

@beckettsean beckettsean mentioned this issue Nov 16, 2015
Closed
@li-ang li-ang mentioned this issue Nov 18, 2015
Merged
4 tasks
@li-ang
Copy link

li-ang commented Nov 18, 2015

@dgnorton I just fix the SHOW MEASUREMENTS. If the PR #4833 could be merged, I would like to try to fix SHOW TAG KEYS and SHOW TAG VALUES using the similar method.

@otoolep otoolep mentioned this issue Nov 23, 2015
Closed
@beckettsean
Copy link
Contributor

fixed by #4833

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dgnorton dgnorton

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dgnorton @otoolep @sergzin @priyadarsh @beckettsean @li-ang