do ES bulks err

[hzruandd]

2016/12/12 16:33:39 binlogsyncer.go:71: [info] create BinlogSyncer with config &{100 mysql 10.8.6.182 3306 root passwd false false}
2016/12/12 16:33:39 dump.go:96: [info] skip dump, use last binlog replication pos (mysql-bin.000001, 30940085)
2016/12/12 16:33:39 sync.go:17: [info] start sync binlog at (mysql-bin.000001, 30940085)
2016/12/12 16:33:39 binlogsyncer.go:226: [info] begin to sync binlog from position (mysql-bin.000001, 30940085)
2016/12/12 16:33:39 binlogsyncer.go:130: [info] register slave for master server 10.8.6.182:3306
2016/12/12 16:33:39 status.go:52: [info] run status http server 10.8.6.170:12800
2016/12/12 16:33:39 binlogsyncer.go:552: [info] rotate to (mysql-bin.000001, 30940085)
2016/12/12 16:33:39 sync.go:53: [info] rotate binlog to (mysql-bin.000001, 30940085)
2016/12/12 16:34:52 sync.go:327: [error] sync docs err Post http://10.8.6.170:9200/_bulk: dial tcp 10.8.6.170:9200: connect: cannot assign requested address after binlog (mysql-bin.000001, 40298382)
2016/12/12 16:34:52 sync.go:54: [error] do ES bulks err Post http://10.8.6.170:9200/_bulk: dial tcp 10.8.6.170:9200: connect: cannot assign requested address, stop
2016/12/12 16:34:52 handler.go:35: [error] handle ESRiverRowsEventHandler err, interrupted
2016/12/12 16:34:52 sync.go:57: [error] handle rows event error do handler error, interrupted
2016/12/12 16:34:52 canal.go:151: [error] canal start sync binlog err: do handler error, interrupted

直接通过ES的restful接口做bulk操作测试，只要保证每个item都有'\n'即可成功。 但是目前却无法同步mysql的binlog数据了。

[hzruandd]

测试结果如下：
[root@1170 go-mysql-elasticsearch]# curl -XPOST '10.8.6.170:9200/demo-afe2016.12.08/kafka2logstash/_bulk?pretty' -d '

{"index":{"_id":"10"}}
{"name": "John Doe" }
{"index":{"_id":"20"}}
{"name": "Jane Doe" } '
{
"took" : 10,
"errors" : false,
"items" : [
{
"index" : {
"_index" : "demo-afe2016.12.08",
"_type" : "kafka2logstash",
"_id" : "10",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"created" : true,
"status" : 201
}
}
]
}

POST /test/type1/_bulk
{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

{
"took": 37,
"errors": true,
"items": [
{
"index": {
"_index": "test",
"_type": "type1",
"_id": "1",
"_version": 3,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": false,
"status": 200
}
},
{
"delete": {
"found": false,
"_index": "test",
"_type": "type1",
"_id": "2",
"_version": 2,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404
}
},
{
"create": {
"_index": "test",
"_type": "type1",
"_id": "3",
"status": 409,
"error": {
"type": "version_conflict_engine_exception",
"reason": "[type1][3]: version conflict, document already exists (current version [1])",
"index_uuid": "UYrliPSrSNq96vDM-x_7EA",
"shard": "4",
"index": "test"
}
}
},
{
"update": {
"_index": "test",
"_type": "type1",
"_id": "1",
"_version": 4,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200
}
}
]
}

[siddontang]

Thanks @hzruandd , I will see it later.

2016/12/12 16:34:52 sync.go:54: [error] do ES bulks err Post http://10.8.6.170:9200/_bulk: dial tcp 10.8.6.170:9200: connect: cannot assign requested address, stop

The error looks strange, can golang/go#16012 fix it?

[hzruandd]

it works well when i start, and it can sync binlog from mysql to es. maybe seconds latter, it appears the above error.

[siddontang]

Seem that this is the problem for socket, how many TIME_WAIT with netstat?

See http://stackoverflow.com/questions/26019164/too-many-time-wait-connections-getting-cannot-assign-requested-address

[siddontang]

@hzruandd

Please update the codes and try again.