Failover requests would not work for partially used streams

[rahulreddy]

Identified by @alexandre-merle, streams which are consumed partially and failed over to another node would not work as it would only be able to pipe partial data

[ghost]

This would explain scality/MetaData#940 then.

[rahulreddy]

I guess. I just can't think of why the source stream would end up with an error, I would have assumed that there were TCP errors connecting to any of the nodes from start.

[ghost]

On the MD issue, @msegura analyzed for some connection closing related to the way the keepalive is configured on our Sproxyd's Tengine. Does it look to you like this could be the explanation ?

[rahulreddy]

I think it fits the picture. The log line with PUT chunk to sproxyd

{"name":"SproxydClient","error":{"code":"ECONNRESET"},"time":1484643601662,"req_id":"a7ea4c3151407e76b10e:7a6691a0746cb6785797","level":"error","message":"PUT chunk to sproxyd","hostname":"asvppdxobjs301.gecis.io","pid":59}

could only happen if a connection was established to the remote host, socket was assigned and then later destroyed by the remote host for some reason. In that case, the stream may have been partially consumed.
socket object has properties bytesRead / bytesWritten which we can use to do a failover, only when no bytes have been consumed.

[ThibaultRiviere]

That can explain the retry error but not the first one right ?

[rahulreddy]

Yeah I can't explain the first one.

[rachedbenmustapha]

Does this really happen though, given that production deployments only have 1 sproxyd endpoint configured and thus can not fail over?

[rahulreddy]

From the situation today, I realized that we have only 1 sproxyd endpoint. My speculation goes out of the window. It's a potential problem if there are multiple nodes in the bootstrap list.