Skip to content

Latest commit

 

History

History
310 lines (262 loc) · 9.92 KB

README.md

File metadata and controls

310 lines (262 loc) · 9.92 KB
Raw

binary

Version Build status Coverage Status Go Report Card GoDoc License

Package binary is uesed to Encode/Decode between go data and byte slice.

The main purpose of this package is to replace package "std.binary".

Compare with other serialization package, this package is with full-feature as gob and protocol buffers, and with high-performance and lightweight as std.binary.

It is designed as a common solution to easily encode/decode between go data and byte slice.

It is recommended to use in net protocol serialization and go memory data serialization such as DB.

Install

$ go get -u github.com/vipally/binary
import(
	"github.com/vipally/binary"
)

Change log:

v1.2.0

1.use field tag `binary:"packed"` to encode ints value as varint/uvarint 
  for reged structs.
2.add method Encoder.ResizeBuffer.

v1.1.0

1.fix issue#1 nil pointer encode/decode error.
2.pack 8 bool values as bits in one byte.
3.put one bool bit for pointer fields to check if it is a nil pointer.
4.rename Pack/Unpack to Encode/Decode.

v1.0.0

1.full-type support like gob.
2.light-weight as std.binary.
3.high-performance as std.binary and gob.
4.encoding with fower bytes than std.binary and gob.
5.use RegStruct to improve performance of struct encoding/decoding.
6.take both advantages of std.binary and gob.
7.recommended using in net protocol serialization and DB serialization.

Todo:

1.[Encoder/Decoder].RegStruct to speed up local Coder.
2.[Encoder/Decoder].RegSerializer to speed up BinarySerializer search.
3.reg interface to using BinarySerializer interface.

CopyRight

CopyRight 2017 @Ally Dale. All rights reserved.

Author : Ally Dale(vipally@gmail.com)

Blog : http://blog.csdn.net/vipally

Site : https://github.com/vipally

1. Support all serialize-able basic types:

int, int8, int16, int32, int64,
uint, uint8, uint16, uint32, uint64,
float32, float64, complex64, complex128,
bool, string, slice, array, map, struct.
And their direct pointers. 
eg: *string, *struct, *map, *slice, *int32.

2. [recommended usage] Use Encode/Decode to read/write memory buffer directly.

Use RegStruct to improve struct encoding/decoding efficiency.

type someRegedStruct struct {
	A int `binary:"ignore"`
	B string
	C uint
}
binary.RegStruct((*someRegedStruct)(nil))

If data implements interface BinaryEncoder, it will use data.Encode/data.Decode 
to encode/decode data.
NOTE that data.Decode must implement on pointer receiever to enable modifying
receiever.Even though Size/Encode of data can implement on non-pointer receiever,
binary.Encode(&data, nil) is required if data has implement interface BinaryEncoder.
binary.Encode(data, nil) will probably NEVER use BinaryEncoder methods to Encode/Decode
data.
eg:

import "github.com/vipally/binary"

//1.Encode with default buffer
if bytes, err := binary.Encode(&data, nil); err==nil{
	//...
}

//2.Encode with existing buffer
size := binary.Sizeof(data)
buffer := make([]byte, size)
if bytes, err := binary.Encode(&data, buffer); err==nil{
	//...
}

//3.Decode from buffer
if err := binary.Decode(bytes, &data); err==nil{
	//...
}

3. [advanced usage] Encoder/Decoder are exported types aviable for encoding/decoding.

eg:
encoder := binary.NewEncoder(bufferSize)
encoder.Uint32(u32)
encoder.String(str)
encodeResult := encoder.Buffer()

decoder := binary.NewDecoder(buffer)
u32 := decoder.Uint32()
str := decoder.String()

4. Put an extra length field(uvarint,1~10 bytes) before string, slice, array, map.

eg: 
var s string = "hello"
will be encoded as:
[]byte{0x5, 0x68, 0x65, 0x6c, 0x6c, 0x6f}

5. Pack bool array with bits.

eg: 
[]bool{true, true, true, false, true, true, false, false, true}
will be encoded as:
[]byte{0x9, 0x37, 0x1}

6. Hide struct field when encoding/decoding.

Only encode/decode exported fields.
Support using field tag `binary:"ignore"` to disable encode/decode fields.
eg: 
type S struct{
    A uint32
	b uint32
	_ uint32
	C uint32 `binary:"ignore"`
}
Only field "A" will be encode/decode.

7. Auto allocate for slice, map and pointer.

eg: 
type S struct{
    A *uint32
	B *string
	C *[]uint8
	D []uint32
}
It will new pointers for fields "A, B, C",
and make new slice for fields "*C, D" when decode.

8. int/uint values will be encoded as varint/uvarint(1~10 bytes).

eg: 
uint(1)     will be encoded as: []byte{0x1}
uint(128)   will be encoded as: []byte{0x80, 0x1}
uint(32765) will be encoded as: []byte{0xfd, 0xff, 0x1}
int(-5)     will be encoded as: []byte{0x9}
int(-65)    will be encoded as: []byte{0x81, 0x1}

9. Test results.

Enncoding size(see example of Sizeof).

Encoding bytes is much shorter than std.binary and gob.

var s struct {
	Int8        int8
	Int16       int16
	Int32       int32
	Int64       int64
	Uint8       uint8
	Uint16      uint16
	Uint32      uint32
	Uint64      uint64
	Float32     float32
	Float64     float64
	Complex64   complex64
	Complex128  complex128
	Array       [10]uint8
	Bool        bool
	BoolArray   [100]bool
	Uint32Array [10]uint32
}

Output:
Sizeof(s)  = 133
std.Size(s)= 217
gob.Size(s)= 412

Benchmark test result.

Pack/Unpack unregisted struct is a little slower(80%) than std.binary.Read/std.binary.Write
Pack/Unpack registed struct is much faster(200%) than std.binary.Read/std.binary.Write
Pack/Unpack int array is much faster(230%) than std.binary.Read/std.binary.Write
Pack/Unpack string is a little faster(140%) than gob.Encode/gob.Decode

BenchmarkGobEncodeStruct     	 1000000	      1172 ns/op	 300.32 MB/s
BenchmarkStdWriteStruct      	 1000000	      3154 ns/op	  23.78 MB/s
BenchmarkWriteStruct         	  500000	      3954 ns/op	  18.71 MB/s
BenchmarkWriteRegedStruct    	 1000000	      1627 ns/op	  45.48 MB/s
BenchmarkPackStruct          	  500000	      3642 ns/op	  20.32 MB/s
BenchmarkPackRegedStruct     	 1000000	      1542 ns/op	  47.99 MB/s
BenchmarkGobDecodeStruct     	 3000000	       651 ns/op	 540.40 MB/s
BenchmarkStdReadStruct       	 1000000	      2008 ns/op	  37.35 MB/s
BenchmarkReadStruct          	  500000	      2386 ns/op	  31.01 MB/s
BenchmarkReadRegedStruct     	 1000000	      1194 ns/op	  61.97 MB/s
BenchmarkUnackStruct         	 1000000	      2293 ns/op	  32.27 MB/s
BenchmarkUnpackRegedStruct   	 2000000	       935 ns/op	  79.14 MB/s
BenchmarkGobEncodeInt1000    	  100000	     22871 ns/op	 219.58 MB/s
BenchmarkStdWriteInt1000     	   30000	     49502 ns/op	  80.80 MB/s
BenchmarkWriteInt1000        	   50000	     26001 ns/op	 153.91 MB/s
BenchmarkPackInt1000         	  100000	     21601 ns/op	 185.27 MB/s
BenchmarkStdReadInt1000      	   30000	     44035 ns/op	  90.84 MB/s
BenchmarkReadInt1000         	   50000	     29761 ns/op	 134.47 MB/s
BenchmarkUnackInt1000        	   50000	     25001 ns/op	 160.07 MB/s
BenchmarkGobEncodeString     	 3000000	       444 ns/op	 301.56 MB/s
BenchmarkStdWriteString 		unsupported 
BenchmarkWriteString         	 3000000	       455 ns/op	 285.49 MB/s
BenchmarkPackString          	 5000000	       337 ns/op	 385.05 MB/s
BenchmarkGobDecodeString     	 1000000	      1266 ns/op	 105.84 MB/s
BenchmarkStdReadString 			unsupported 
BenchmarkReadString          	 3000000	       550 ns/op	 236.06 MB/s
BenchmarkUnackString         	 5000000	       298 ns/op	 435.92 MB/s

With full-datatype support like gob.

Followed struct fullStruct is an aviable type that contains all supported types.
Which is not valid for std.binary.

type fullStruct struct {
	Int8       int8
	Int16      int16
	Int32      int32
	Int64      int64
	Uint8      uint8
	Uint16     uint16
	Uint32     uint32
	Uint64     uint64
	Float32    float32
	Float64    float64
	Complex64  complex64
	Complex128 complex128
	Array      [4]uint8
	Bool       bool
	BoolArray  [9]bool

	LittleStruct  littleStruct
	PLittleStruct *littleStruct
	String        string
	PString       *string
	PInt32        *int32
	Slice         []*littleStruct
	PSlice        *[]*string
	Float64Slice  []float64
	BoolSlice     []bool
	Uint32Slice   []uint32
	Map           map[string]*littleStruct
	Map2          map[string]uint16
	IntSlice      []int
	UintSlice     []uint
}

Do not supported types.

Followed struct TDoNotSupport is an invalid type that every field is invalid.

type TDoNotSupport struct {
	DeepPointer   **uint32
	Uintptr       uintptr
	UnsafePointer unsafe.Pointer
	Ch            chan bool
	Map           map[uintptr]uintptr
	Map2          map[int]uintptr
	Map3          map[uintptr]int
	Slice         []uintptr
	Array         [2]uintptr
	Array2        [2][2]uintptr
	Array3        [2]struct{ A uintptr }
	Func          func()
	Struct        struct {
		PStruct *struct {
			PPUintptr **uintptr
		}
	}
}

License.

Under MIT license.

Copyright (c) 2017 Ally Dale<vipally@gmail.com>
Author  : Ally Dale<vipally@gmail.com>
Site    : https://github.com/vipally
Origin  : https://github.com/vipally/binary