domain: load stats table asynchronously

[alivxxx]

We do not have to wait all stats table loaded into memory when starts TiDB server.

[shenli]

LGTM

[winoros]

LGTM

[shenli]

/run-all-tests

[shenli]

@lamxTyler CI failed.

[alivxxx]

@shenli It is not caused by this PR, @tiancaiamao is looking at it.

[darren]

@lamxTyler @shenli

Before this PR, tidb-server on our cluster takes 8 minutes to start before
it can accept connections. Most of the time is spent on loading stats.

After this PR, tidb-server starts very quickly and ready for accepting queries,
However it raises another problem: because stats loading is running in the background
and will take for a while. if tidb-server in the meantime accepts queries on tables whose
histograms are still in the loading queue, the queries will be very slow and resource intensive
thus bring significant impact on the cluster.

If that's the case then I would prefer that tidb-server load stats fully before accepting queries.

Is it feasible that loading the stats on demand?

[shenli]

@darren How many tables do you have? Could you provide us some of you schema and sql? We could check why them run slow with default statistic info.

[darren]

@shenli

We are still bulk loading data in the cluster and due to rotational disks on our TiKV servers, thus I expect that it will load slow. I think it's normal.

Here's a debug log of tidb-server start before this PR:

tidb-server.log.gz

[alivxxx]

@darren In #5215, we speed up the loading of stats when the server starts, could you try it and see if it is faster enough?

[darren]

@lamxTyler
If I stop loading data in another tidb-server instance, it loads faster enough, I estimate that it takes about 30 seconds or so (about 1500+ tables).

[shenli]

@lamxTyler Could we make it another 10x faster?

[shenli]

A native improvement: load stats_meta/stats_histograms/stats_buckets parallelly.

[darren]

@lamxTyler

Could you log some info about when stats loading starts and finishes? for now I have to set log level to debug and do some estimates by hand.

I think the print out some info level log about stats loading status is important for admins.

[alivxxx]

@darren Yes, I will do it.

[shenli]

@darren #5266 makes loading stats info faster and add time consuming in log for loading stats info.

[darren]

@lamxTyler @shenli

Good job!

It's really faster now:

2017/12/01 10:21:55.491 domain.go:613: [info] [stats] init stats info takes 7.019830276s