panic: runtime error: index out of range

[Lyle-liang]

goroutine 3347 [running]:
github.com/pingcap/tidb/util/chunk.(*column).isNull(...)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/util/chunk/column.go:71
github.com/pingcap/tidb/util/chunk.Row.IsNull(...)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/util/chunk/row.go:222
github.com/pingcap/tidb/util/chunk.Row.GetDatum(0xc420194420, 0xffffffffffffffff, 0x0, 0xc4207ab3b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/util/chunk/row.go:152 +0xc4a
github.com/pingcap/tidb/statistics.(*Histogram).GetUpper(0xc420a63440, 0xffffffffffffffff, 0x132dd77)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/histogram.go:102 +0x83
github.com/pingcap/tidb/statistics.(*Histogram).bucketToString(0xc420a63440, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/histogram.go:356 +0x4d
github.com/pingcap/tidb/statistics.formatBuckets(0xc420a63440, 0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/feedback.go:767 +0x3d5
github.com/pingcap/tidb/statistics.colRangeToStr(0xc420a63440, 0xc425596c68, 0xffffffffffffffff, 0x3ffe0ef38f169eab, 0xc42029f690, 0x0)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/feedback.go:781 +0x190
github.com/pingcap/tidb/statistics.logForIndex(0xc421da8660, 0x22, 0xc42076a360, 0xc420a63710, 0xc4237cb440, 0x3, 0x3, 0xc426ef9f20, 0x3, 0x3, ...)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/feedback.go:858 +0xb45
github.com/pingcap/tidb/statistics.(*QueryFeedback).logDetailedInfo(0xc421a5de00, 0xc4203730e0)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/feedback.go:886 +0x3d1
github.com/pingcap/tidb/statistics.(*SessionStatsCollector).StoreQueryFeedback(0xc420727680, 0x1268400, 0xc421a5de00, 0xc4203730e0, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/statistics/update.go:196 +0x104
github.com/pingcap/tidb/session.(*session).StoreQueryFeedback(0xc4225b01c0, 0x1268400, 0xc421a5de00)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/session/session.go:237 +0x18b
github.com/pingcap/tidb/executor.(*IndexLookUpExecutor).startIndexWorker.func1(0x7fb69bcf1988, 0xc423242900, 0xc42194a580, 0x1451a20, 0xc4260e3380, 0xc424cc8340, 0xc423548e40)
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/executor/distsql.go:476 +0x10a
created by github.com/pingcap/tidb/executor.(*IndexLookUpExecutor).startIndexWorker
	/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/executor/distsql.go:470 +0x3ce
panic: runtime error: index out of range

[crazycs520]

@Lyle-liang Thanks for your feedback.
Could you provide more information, such as tidb_version.
And What did you do? How to reproduce it.

[lysu]

the backtrace is very similar to #7588 which is fixed in 2.1.0.rc1

[Lyle-liang]

@crazycs520
Release Version: v2.1.0-beta-251-g2fbd5e7
Git Commit Hash: 2fbd5e7
Git Branch: master
UTC Build Time: 2018-08-17 03:41:19
GoVersion: go version go1.10.2 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e

[Lyle-liang]

@lysu

I try to upgrade v2.1.0-rc2, but failure。

[lysu]

@Lyle-liang picture is broken 🤣

[Lyle-liang]

@lysu log like this:
failed: [localhost] (item={u'url': u'http://download.pingcap.org/tidb-v2.1.0-rc.2-linux-amd64-unportable.tar.gz', u'version': u'v2.1.0-rc.2', u'name': u'tidb'}) => {"attempts": 4, "changed": false, "dest": "/home/ec2-user/tidb-ansible/downloads/tidb-v2.1.0-rc.2.tar.gz", "item": {"name": "tidb", "url": "http://download.pingcap.org/tidb-v2.1.0-rc.2-linux-amd64-unportable.tar.gz", "version": "v2.1.0-rc.2"}, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "state": "absent", "status_code": 404, "url": "http://download.pingcap.org/tidb-v2.1.0-rc.2-linux-amd64-unportable.tar.gz"}

How can I upgrade v2.1.0-rc2 ? Can you help me?

[Lyle-liang]

@crazycs520
I change tidb_version = v2.1.0-rc.2, but error. How can I upgrade v2.1.0-rc2 via ansible? Can you help me?

[lysu]

@LinuxGit please help look ansible upgrade problem, download url 404..

[alivxxx]

@Lyle-liang You can first upgrade the tidb-ansible to the latest version and try again.

[Lyle-liang]

@lamxTyler what is the latest version? I have ansible 2.6.3

[alivxxx]

@Lyle-liang I mean tidb-ansible, not ansible.

[Lyle-liang]

@lamxTyler I agree what you said, I use tidb-ansible and version 2.6.3.

[Lyle-liang]

@lysu @LinuxGit Is there a solution? emergency...

[alivxxx]

@Lyle-liang I cannot see your picture. If you use git log under the directory of tidb-ansible, you can see something like:

commit 771cbe4ced1b9207680f6f5e9cdc928bbca24d11 (HEAD -> master, origin/master, origin/HEAD)
Author: Louis <liuzhi@pingcap.com>
Date:   Fri Sep 28 11:34:29 2018 +0800

    prometheus: modify TiKV_coprocessor_request_error alert rule (#555)

So if your date is before this, you can run git pull origin master to use the latest tidb-ansible. And you should not change the tidb-version, just keep it as latest.

[Lyle-liang]

@lamxTyler It's ok. I will be try. Another, I have a question, and when the components are rolling update, will the TiDB service stop?

[alivxxx]

@Lyle-liang We upgrade the tidb server one by one, so if you have multiple tidb servers, there is always some tidb server at service.

[LinuxGit]

1. Make sure you use tidb-ansible master branch, not release-2.0, see https://github.com/pingcap/docs/blob/master/op-guide/ansible-deployment.md#step-3-download-tidb-ansible-to-the-control-machine.

$ cd tidb-ansible
$ git branch
* master

2. Make sure tidb_version = latest in inventory.ini file

3. Delete the existing downloads directory /home/tidb/tidb-ansible/downloads/.

$ cd /home/tidb/tidb-ansible
$ rm -rf downloads

4. Download latest tidb packages.

$ ansible-playbook local_prepare.yml

5. Apply a rolling update to the TiDB node (only upgrade the TiDB service)

$ ansible-playbook rolling_update.yml --tags=tidb

See https://github.com/pingcap/docs/blob/master/op-guide/ansible-deployment-rolling-update.md#upgrade-the-component-version.