pre-allocate decoding errors

[Dieterbe]

I saw that the dynamic creation of errors was contributing a lot to the heap profile (see below)
I haven't tested extensively yet, but this should fix it

(pprof) top30 -cum
62951786 of 63347642 total (99.38%)
Dropped 41 nodes (cum <= 316738)
      flat  flat%   sum%        cum   cum%
  57502851 90.77% 90.77%   63347642   100%  [metric_tank]
         0     0% 90.77%   35258445 55.66%  github.com/Shopify/sarama.(*realDecoder).getInt64Array
   3922373  6.19% 96.97%    3922373  6.19%  regexp/syntax.(*compiler).quest
    680349  1.07% 98.04%     680349  1.07%  crypto/x509/pkix.(*Name).FillFromRDNSequence
    444790   0.7% 98.74%     504731   0.8%  crypto/x509/pkix.Name.ToRDNSequence
         0     0% 98.74%     491660  0.78%  github.com/Shopify/sarama.(*OffsetFetchRequest).decode
    401423  0.63% 99.38%     401423  0.63%  regexp/syntax.(*parser).factor
(pprof) list getInt64Array
Total: 63347642
ROUTINE ======================== github.com/Shopify/sarama.(*realDecoder).getInt64Array in /home/dieter/go/src/github.com/Shopify/sarama/real_decoder.go
         0   35258445 (flat, cum) 55.66% of Total
         .          .    153:}
         .          .    154:
         .          .    155:func (rd *realDecoder) getInt64Array() ([]int64, error) {
         .          .    156:   if rd.remaining() < 4 {
         .          .    157:       rd.off = len(rd.raw)
         .      10923    158:       return nil, ErrInsufficientData
         .          .    159:   }
         .          .    160:   n := int(binary.BigEndian.Uint32(rd.raw[rd.off:]))
         .          .    161:   rd.off += 4
         .          .    162:
         .          .    163:   if rd.remaining() < 8*n {
         .    1773176    164:       rd.off = len(rd.raw)
         .          .    165:       return nil, ErrInsufficientData
         .          .    166:   }
         .          .    167:
         .          .    168:   if n == 0 {
         .    1180404    169:       return nil, nil
         .          .    170:   }
         .          .    171:
         .          .    172:   if n < 0 {
         .   32293942    173:       return nil, PacketDecodingError{"invalid array length"}
         .          .    174:   }
         .          .    175:
         .          .    176:   ret := make([]int64, n)
         .          .    177:   for i := range ret {
         .          .    178:       ret[i] = int64(binary.BigEndian.Uint64(rd.raw[rd.off:]))
(pprof) 

[eapache]

What kind of environment are you running that you're getting a substantial number of errors? I've never bothered optimising these paths because in practice kafka brokers always send well-formed messages (over TCP with a checksum too) so none of these errors are ever hit.

[eapache]

I think it might make more sense to make these unexported and move them to the top of real_decoder.go, I can't think of a case where the caller would need to be able to distinguish between them.

[Dieterbe]

This is in a development environment, and my first foray into Kafka (though with substantial traffic/load). I'll move the errors to real_decoder and privatise them. While the issue may be rare it should be a non-issue IMHO. I wouldn't want a flood of protocol errors to be exacerbated by excessive GC in prod.

[Dieterbe]

@eapache how does it look now?

[eapache]

Great, thanks!

[wvanbergen]

Thanks for contributing!