-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[0.9.4.1] panic: interface conversion: interface {} is map[string]interface {}, not *influxql.meanMapOutput #4170
Comments
@ecables -- just to confirm, this data should be fully replicated, correct? |
It looks like I may have duped this, I can bring out a panic with an identical trace in response to queries. |
OK, I am working a hypothesis here. I believe your data is fully replicated, but occasionally when a new shard group is created, the system (due to the nature of the system) "sees" those new shards on remote nodes before it sees that the data is also local. This triggers a remote access to that shard. Secondly, there appears to be bug in our marshalling code that takes query output, and sends it over the network. It is this second issue I am debugging right now, as it is this issue that causes the panic. |
@dgnorton -- something is going wrong here, and I am not sure if it is related to the refactoring you did recently, or this issue was always here. The types emitted by the Mappers is getting "lost", and translated to simple maps. This is causing the receiving node to panic. That's what it looks like to me anyway. |
#4174 may allow an effective workaround for this issue, where clusters are fully replicated. |
@dgnorton - after further examination of the code, I think this issue may predate DQ work even, and that the model for marshalling Mapper data across the network may need to be reworked. #4174 should provide an effective workaround for the remote mapping operation in the first place (though it still needs to be established why remote mapping is occasionally occurring on a fully-replicated cluster), but it looks to me we may need to revisit marshaling of remotely-mapped Mapper outputs. |
@otoolep -- The replication factor is 3 on a 3 node cluster, so the expectation is that everything is fully replicated. |
Can you show us the output of
|
Here's output from one node in the cluster:
|
OK, thanks, this confirms that the shards should be fully replicated. We're continuing to look into the issue here. |
I believe I see what is going on here, with regard to the specific panic. I am going to start on a fix now, hope to have this fixed very soon. |
Should be fixed by #4191 |
The latest nightly builds includes new stats named cluster (available via |
Two nodes in a 3-node cluster panic'd with the following panic message.
The text was updated successfully, but these errors were encountered: