Add --stats to traces

[klauspost]

Description

Allows collecting live stats of all traced subsystems.

Examples:

λ mc admin trace --stats play
+----------------------------+-------------+------+-----------+--------+---------+---------+
| Call                       | Count       | RPM  | Avg Time  | Errors | RX Avg  | TX Avg  |
| s3.GetObject               | 247 (63.3%) | 82.5 | 2.391ms   | 0      | 133 B   | 179 KiB |
| s3.HeadBucket              | 102 (26.2%) | 34.1 | 1.114ms   | 0      | 147 B   | 0 B     |
| s3.GetBucketLocation       | 18 (4.6%)   | 6.0  | 1.942ms   | 0      | 147 B   | 149 B   |
| s3.PutBucket               | 6 (1.5%)    | 2.0  | 4.146ms   | 0      | 151 B   | 0 B     |
| s3.ListObjectsV1           | 4 (1.0%)    | 1.3  | 4.276ms   | 0      | 124 B   | 577 B   |
| s3.ListBuckets             | 3 (0.8%)    | 1.0  | 15.312ms  | 0      | 135 B   | 82 KiB  |
| s3.PutObject               | 3 (0.8%)    | 1.0  | 4.304ms   | 0      | 14 KiB  | 0 B     |
| s3.DeleteObject            | 2 (0.5%)    | 0.7  | 407µs     | 0      | 88 B    | 0 B     |
| s3.HeadObject              | 2 (0.5%)    | 0.7  | 1.36ms    | 0      | 108 B   | 0 B     |
| s3.CompleteMultipartUpload | 1 (0.3%)    | 0.3  | 7.943ms   | 0      | 301 B   | 449 B   |
| s3.NewMultipartUpload      | 1 (0.3%)    | 0.3  | 4.895ms   | 0      | 160 B   | 383 B   |
| s3.PutObjectPart           | 1 (0.3%)    | 0.3  | 103.782ms | 0      | 8.8 MiB | 0 B     |
+----------------------------+-------------+------+-----------+--------+---------+---------+

λ mc admin trace --stats -call=storage play
+------------------------------+---------------+----------+----------+--------+--------+--------+
| Call                         | Count         | RPM      | Avg Time | Errors | RX Avg | TX Avg |
| storage.ReadXL               | 70780 (33.3%) | 174700.3 | 43µs     | 200    | -      | -      |
| storage.RenameData           | 70633 (33.2%) | 174337.5 | 1.165ms  | 0      | -      | -      |
| storage.DiskInfo             | 70540 (33.1%) | 174107.9 | 1µs      | 0      | -      | -      |
| storage.ReadVersion          | 280 (0.1%)    | 691.1    | 70µs     | 0      | -      | -      |
| storage.StatVol              | 236 (0.1%)    | 582.5    | 13µs     | 0      | -      | -      |
| storage.DeleteAbandonedParts | 140 (0.1%)    | 345.6    | 149µs    | 0      | -      | -      |
| storage.CreateFile           | 88 (0.0%)     | 217.2    | 4.169ms  | 0      | -      | -      |
| storage.ListVols             | 88 (0.0%)     | 217.2    | 426µs    | 0      | -      | -      |
| storage.ReadFileStream       | 18 (0.0%)     | 44.4     | 28µs     | 0      | -      | -      |
| storage.VerifyFile           | 16 (0.0%)     | 39.5     | 2.166ms  | 0      | -      | -      |
+------------------------------+---------------+----------+----------+--------+--------+--------+

How to test this PR?

See examples above.

Types of changes

• New feature (non-breaking change which adds functionality)

[harshavardhana]

Why not enhance mc support top api @klauspost ?

[klauspost]

Mainly just because this has all the filtering set up already. But we could move it there.

I am not opinionated. Just want the same opinions.

[harshavardhana]

No it is easier to do it there since this table format is already present and its doing the same things anyways.

It is calculating TotalTX and TotalRX for each of these values - we can add flags to ensure we can capture other internal calls as well.

[harshavardhana]

I prefer we move it there.

[harshavardhana]

please resolve the conflicts @klauspost

[harshavardhana]

is this needed now since mc support top api has the relevant data? @klauspost

[harshavardhana]

taking this in anyways since this is specific to different call-types