Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

telemetry: Add telemetry initial implementation #18180

Merged
merged 26 commits into from
Jun 24, 2020
Merged

telemetry: Add telemetry initial implementation #18180

merged 26 commits into from
Jun 24, 2020

Conversation

breezewish
Copy link
Member

@breezewish breezewish commented Jun 23, 2020 •
edited
Loading

What problem does this PR solve?

This PR implements the telemetry service.

What is changed and how it works?

What's Changed:

A telemetry service running in background is introduced. It will report usage data every 24 hours to PingCAP.

  • TODO: Verify privileges for newly added ADMIN statements. Verified that ADMIN statements requires SUPER privilege.

Data Storing Compliance

The usage data is stored in AWS US region for non-China IPs, and stored in CN region for China IPs.

Data It Collects

  • A tracking UUID for current cluster
  • The following hardware information for each hosts in the cluster:
    • CPU (cache size, frequency, cores, vendor, etc)
    • Memory (capacity)
    • Disks (device name, capacity, used bytes, mount point, fs, etc)
      Only device name and mount point in our allowed list (whitelist) will be collected in plain text. Otherwise they will be a SHA1 checksum.
  • The following information for each component instances (TiDB, PD, TiKV, TiFlash) in the cluster:
    • Instance type (tidb / pd / ...)
    • SHA1 checksum of listen host
    • Listen port
    • SHA1 checksum of status server host
    • Status server port
    • Version
    • Git hash
    • Start time
    • Up time
  • The following information of one of the host that runs TiDB instance:
    • CPU flags (fpu, vme, mmx, ...)
    • CPU model name (like Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz)
    • OS (like linux)
    • Platform (like CentOS)
    • Platform family (like RHEL)
    • Platform version
    • Kernel version
    • Kernel arch (like x86_64)
    • Virtualization (like kvm)
    • Virtualization role (it is vm host / vm client)

More usage data may be collected in future versions.

Reset the Tracking UUID

The tracking UUID can be re-generated manually by:

ADMIN RESET TELEMETRY_ID;

Disable Telemetry On-The-Fly

SET @@global.tidb_enable_telemetry = 0;

Disable Telemetry Before Bootstrap

You can disable telemetry by setting config for all TiDB instances:

enable-telemetry = false

When telemetry is disabled in config, global variables will not taking effect (i.e. telemetry will be always disabled even if it is enabled in the global variable).

View Telemetry Status

ADMIN SHOW TELEMETRY;

Sample output:

TRACKING_ID: db8dba8f-8fb0-46c6-a5c4-3e3058a0e9e8
 LAST_STATUS: {
  "check_at": "2020-06-23T12:35:13+08:00",
  "is_error": false,
  "error_msg": "",
  "is_request_sent": true
}
DATA_PREVIEW: {
  "hardware": [
    {
      "instanceType": "pd",
         ...

You can see the status of the last telemetry attempt, as well as preview the usage data that will be send to PingCAP next time.

Sample Telemetry Data

CentOS 7 
{
  "hardware": [
    {
      "instanceType": "tidb",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "35543",
      "cpu": {
        "cache": "25600",
        "cpuFrequency": "3100.00MHz",
        "cpuLogicalCores": "40",
        "cpuPhysicalCores": "20"
      },
      "memory": {
        "capacity": "134905430016"
      },
      "disk": {
        "nvme0n1": {
          "deviceName": "nvme0n1",
          "free": "0",
          "freePercent": "0.00",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "df54ef6702a4cd3fea1b4ca7fce4744606129993",
          "total": "369061818368",
          "used": "350290862080",
          "usedPercent": "1.00"
        },
        "sda1": {
          "deviceName": "sda1",
          "free": "43447296",
          "freePercent": "0.24",
          "fstype": "ext4",
          "opts": "rw,relatime",
          "path": "/boot",
          "total": "198896640",
          "used": "140769280",
          "usedPercent": "0.76"
        },
        "sda2": {
          "deviceName": "sda2",
          "free": "3263488000",
          "freePercent": "0.00",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "/",
          "total": "1573233098752",
          "used": "1490030006272",
          "usedPercent": "1.00"
        },
        "sdb": {
          "deviceName": "sdb",
          "free": "210867863552",
          "freePercent": "0.14",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "ca4f0261aef9e2010ac4f69977c2be12ccd75035",
          "total": "1574498656256",
          "used": "1283626962944",
          "usedPercent": "0.86"
        }
      }
    },
    {
      "instanceType": "pd",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "2379",
      "cpu": {
        "cache": "25600",
        "cpuFrequency": "3100.00MHz",
        "cpuLogicalCores": "40",
        "cpuPhysicalCores": "20"
      },
      "memory": {
        "capacity": "134905430016"
      },
      "disk": {
        "nvme0n1": {
          "deviceName": "nvme0n1",
          "free": "0",
          "freePercent": "0.00",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "df54ef6702a4cd3fea1b4ca7fce4744606129993",
          "total": "369061818368",
          "used": "350290862080",
          "usedPercent": "1.00"
        },
        "sda1": {
          "deviceName": "sda1",
          "free": "43447296",
          "freePercent": "0.24",
          "fstype": "ext4",
          "opts": "rw,relatime",
          "path": "/boot",
          "total": "198896640",
          "used": "140769280",
          "usedPercent": "0.76"
        },
        "sda2": {
          "deviceName": "sda2",
          "free": "3263488000",
          "freePercent": "0.00",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "/",
          "total": "1573233098752",
          "used": "1490030006272",
          "usedPercent": "1.00"
        },
        "sdb": {
          "deviceName": "sdb",
          "free": "210867863552",
          "freePercent": "0.14",
          "fstype": "ext4",
          "opts": "rw,noatime",
          "path": "ca4f0261aef9e2010ac4f69977c2be12ccd75035",
          "total": "1574498656256",
          "used": "1283626962944",
          "usedPercent": "0.86"
        }
      }
    },
    {
      "instanceType": "tikv",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "39570",
      "cpu": {
        "cpuFrequency": "1839MHz",
        "cpuLogicalCores": "40",
        "cpuPhysicalCores": "20",
        "cpuVendorId": "GenuineIntel",
        "l1CacheLineSize": "64",
        "l1CacheSize": "32768",
        "l2CacheLineSize": "64",
        "l2CacheSize": "262144",
        "l3CacheLineSize": "64",
        "l3CacheSize": "26214400"
      },
      "memory": {
        "capacity": "134905430016"
      },
      "disk": {
        "devtmpfs": {
          "deviceName": "devtmpfs",
          "free": "67441336320",
          "freePercent": "1.00",
          "fstype": "devtmpfs",
          "path": "/dev",
          "total": "67441336320",
          "used": "0",
          "usedPercent": "0.00"
        },
        "nvme0n1": {
          "deviceName": "nvme0n1",
          "free": "0",
          "freePercent": "0.00",
          "fstype": "ext4",
          "path": "df54ef6702a4cd3fea1b4ca7fce4744606129993",
          "total": "369061818368",
          "used": "369061818368",
          "usedPercent": "1.00"
        },
        "rootfs": {
          "deviceName": "rootfs",
          "free": "3262820352",
          "freePercent": "0.00",
          "fstype": "rootfs",
          "path": "/",
          "total": "1573233098752",
          "used": "1569970278400",
          "usedPercent": "1.00"
        },
        "sda1": {
          "deviceName": "sda1",
          "free": "43447296",
          "freePercent": "0.22",
          "fstype": "ext4",
          "path": "/boot",
          "total": "198896640",
          "used": "155449344",
          "usedPercent": "0.78"
        },
        "sda2": {
          "deviceName": "sda2",
          "free": "3262820352",
          "freePercent": "0.00",
          "fstype": "ext4",
          "path": "/",
          "total": "1573233098752",
          "used": "1569970278400",
          "usedPercent": "1.00"
        },
        "sdb": {
          "deviceName": "sdb",
          "free": "210867863552",
          "freePercent": "0.13",
          "fstype": "ext4",
          "path": "ca4f0261aef9e2010ac4f69977c2be12ccd75035",
          "total": "1574498656256",
          "used": "1363630792704",
          "usedPercent": "0.87"
        }
      }
    }
  ],
  "instances": [
    {
      "instanceType": "tidb",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "35543",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "33548",
      "version": "4.0.0-beta.2",
      "gitHash": "66ffbca44d0bc9ba8590920eee866ad396d45633",
      "startTime": "2020-06-23T17:09:34+08:00",
      "upTime": "14.763772119s"
    },
    {
      "instanceType": "pd",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "2379",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "2379",
      "version": "4.1.0-alpha",
      "gitHash": "2c251bd8f1810bb0e26b8d972f7233417f60bce9",
      "startTime": "2020-06-23T17:09:28+08:00",
      "upTime": "20.763780133s"
    },
    {
      "instanceType": "tikv",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "39570",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "20180",
      "version": "4.1.0-alpha",
      "gitHash": "5994d418d4d0b5fd97a75dc996478e661edfd2e4",
      "startTime": "2020-06-23T17:09:31+08:00",
      "upTime": "17.763785836s"
    }
  ],
  "hostExtra": {
    "cpuFlags": [
      "fpu",
      "vme",
      "de",
      "pse",
      "tsc",
      "msr",
      "pae",
      "mce",
      "cx8",
      "apic",
      "sep",
      "mtrr",
      "pge",
      "mca",
      "cmov",
      "pat",
      "pse36",
      "clflush",
      "dts",
      "acpi",
      "mmx",
      "fxsr",
      "sse",
      "sse2",
      "ss",
      "ht",
      "tm",
      "pbe",
      "syscall",
      "nx",
      "pdpe1gb",
      "rdtscp",
      "lm",
      "constant_tsc",
      "arch_perfmon",
      "pebs",
      "bts",
      "rep_good",
      "nopl",
      "xtopology",
      "nonstop_tsc",
      "aperfmperf",
      "eagerfpu",
      "pni",
      "pclmulqdq",
      "dtes64",
      "monitor",
      "ds_cpl",
      "vmx",
      "smx",
      "est",
      "tm2",
      "ssse3",
      "sdbg",
      "fma",
      "cx16",
      "xtpr",
      "pdcm",
      "pcid",
      "dca",
      "sse4_1",
      "sse4_2",
      "x2apic",
      "movbe",
      "popcnt",
      "aes",
      "xsave",
      "avx",
      "f16c",
      "rdrand",
      "lahf_lm",
      "abm",
      "3dnowprefetch",
      "epb",
      "cat_l3",
      "cdp_l3",
      "intel_pt",
      "tpr_shadow",
      "vnmi",
      "flexpriority",
      "ept",
      "vpid",
      "fsgsbase",
      "tsc_adjust",
      "bmi1",
      "hle",
      "avx2",
      "smep",
      "bmi2",
      "erms",
      "invpcid",
      "rtm",
      "cqm",
      "rdt_a",
      "rdseed",
      "adx",
      "smap",
      "xsaveopt",
      "cqm_llc",
      "cqm_occup_llc",
      "cqm_mbm_total",
      "cqm_mbm_local",
      "dtherm",
      "ida",
      "arat",
      "pln",
      "pts"
    ],
    "cpuModelName": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
    "os": "linux",
    "platform": "centos",
    "platformFamily": "rhel",
    "platformVersion": "7.3.1611",
    "kernelVersion": "3.10.0-862.14.4.el7.x86_64",
    "kernelArch": "x86_64",
    "virtualizationSystem": "kvm",
    "virtualizationRole": "host"
  },
  "reportTimestamp": 1592903388,
  "trackingId": ""
}
MacOS 
{
  "hardware": [
    {
      "instanceType": "tidb",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "4000",
      "cpu": {
        "cache": "256",
        "cpuFrequency": "2900.00MHz",
        "cpuLogicalCores": "8",
        "cpuPhysicalCores": "4"
      },
      "memory": {
        "capacity": "17179869184"
      },
      "disk": {
        "disk1s1": {
          "deviceName": "disk1s1",
          "free": "90343526400",
          "freePercent": "0.19",
          "fstype": "apfs",
          "opts": "rw,suiddir,multilabel",
          "path": "/System/Volumes/Data",
          "total": "499963170816",
          "used": "387932069888",
          "usedPercent": "0.81"
        },
        "disk1s4": {
          "deviceName": "disk1s4",
          "free": "90343526400",
          "freePercent": "0.90",
          "fstype": "apfs",
          "opts": "rw,suiddir,multilabel",
          "path": "/private/var/vm",
          "total": "499963170816",
          "used": "9751994368",
          "usedPercent": "0.10"
        },
        "disk1s5": {
          "deviceName": "disk1s5",
          "free": "90343526400",
          "freePercent": "0.89",
          "fstype": "apfs",
          "opts": "ro,multilabel",
          "path": "/",
          "total": "499963170816",
          "used": "11163045888",
          "usedPercent": "0.11"
        },
        "disk3s1": {
          "deviceName": "disk3s1",
          "free": "51912704",
          "freePercent": "0.56",
          "fstype": "hfs",
          "opts": "ro,nosuid,suiddir,softdep,multilabel,nfs4acls",
          "path": "33ed3d305c2e762b6165fe5f818fd06eb04453d8",
          "total": "93282304",
          "used": "41369600",
          "usedPercent": "0.44"
        },
        "disk4s1": {
          "deviceName": "disk4s1",
          "free": "26824704",
          "freePercent": "0.36",
          "fstype": "hfs",
          "opts": "ro,nosuid,softdep,multilabel,nfs4acls",
          "path": "b3d72a182f0d53e3a6f9538342bfc48072531246",
          "total": "74448896",
          "used": "47624192",
          "usedPercent": "0.64"
        }
      }
    },
    {
      "instanceType": "pd",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "2379",
      "cpu": {
        "cache": "256",
        "cpuFrequency": "2900.00MHz",
        "cpuLogicalCores": "8",
        "cpuPhysicalCores": "4"
      },
      "memory": {
        "capacity": "17179869184"
      },
      "disk": {
        "disk1s1": {
          "deviceName": "disk1s1",
          "free": "90343526400",
          "freePercent": "0.19",
          "fstype": "apfs",
          "opts": "rw,suiddir,multilabel",
          "path": "/System/Volumes/Data",
          "total": "499963170816",
          "used": "387932069888",
          "usedPercent": "0.81"
        },
        "disk1s4": {
          "deviceName": "disk1s4",
          "free": "90343526400",
          "freePercent": "0.90",
          "fstype": "apfs",
          "opts": "rw,suiddir,multilabel",
          "path": "/private/var/vm",
          "total": "499963170816",
          "used": "9751994368",
          "usedPercent": "0.10"
        },
        "disk1s5": {
          "deviceName": "disk1s5",
          "free": "90343526400",
          "freePercent": "0.89",
          "fstype": "apfs",
          "opts": "ro,multilabel",
          "path": "/",
          "total": "499963170816",
          "used": "11163045888",
          "usedPercent": "0.11"
        },
        "disk3s1": {
          "deviceName": "disk3s1",
          "free": "51912704",
          "freePercent": "0.56",
          "fstype": "hfs",
          "opts": "ro,nosuid,suiddir,softdep,multilabel,nfs4acls",
          "path": "33ed3d305c2e762b6165fe5f818fd06eb04453d8",
          "total": "93282304",
          "used": "41369600",
          "usedPercent": "0.44"
        },
        "disk4s1": {
          "deviceName": "disk4s1",
          "free": "26824704",
          "freePercent": "0.36",
          "fstype": "hfs",
          "opts": "ro,nosuid,softdep,multilabel,nfs4acls",
          "path": "b3d72a182f0d53e3a6f9538342bfc48072531246",
          "total": "74448896",
          "used": "47624192",
          "usedPercent": "0.64"
        }
      }
    },
    {
      "instanceType": "tikv",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "20160",
      "cpu": {
        "cpuFrequency": "2900MHz",
        "cpuLogicalCores": "8",
        "cpuPhysicalCores": "4",
        "cpuVendorId": "Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz\u0000",
        "l1CacheLineSize": "64",
        "l1CacheSize": "32768",
        "l2CacheLineSize": "64",
        "l2CacheSize": "262144",
        "l3CacheLineSize": "64",
        "l3CacheSize": "8388608"
      },
      "memory": {
        "capacity": "17592185856"
      },
      "disk": {
        "a9c0a929f8530e587fd66a6bd075331de95690e2": {
          "deviceName": "a9c0a929f8530e587fd66a6bd075331de95690e2",
          "free": "90343526400",
          "freePercent": "0.18",
          "fstype": "apfs",
          "path": "/",
          "total": "499963170816",
          "used": "409619644416",
          "usedPercent": "0.82"
        }
      }
    }
  ],
  "instances": [
    {
      "instanceType": "tidb",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "4000",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "10080",
      "version": "4.0.0-beta.2",
      "gitHash": "66ffbca44d0bc9ba8590920eee866ad396d45633",
      "startTime": "2020-06-23T17:05:52+08:00",
      "upTime": "16.035407s"
    },
    {
      "instanceType": "pd",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "2379",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "2379",
      "version": "4.1.0-alpha",
      "gitHash": "a80b99ef0d70807d106bdc1b7c6e97a118679c7e",
      "startTime": "2020-06-23T17:05:47+08:00",
      "upTime": "21.03541s"
    },
    {
      "instanceType": "tikv",
      "listenHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "listenPort": "20160",
      "statusHostHash": "4b84b15bff6ee5796152495a230e45e3d7e947d9",
      "statusPort": "20180",
      "version": "4.1.0-alpha",
      "gitHash": "2be16853e00681f858e3902328ee44a7037ed8c1",
      "startTime": "2020-06-23T17:05:49+08:00",
      "upTime": "19.035413s"
    }
  ],
  "hostExtra": {
    "cpuFlags": [
      "fpu",
      "vme",
      "de",
      "pse",
      "tsc",
      "msr",
      "pae",
      "mce",
      "cx8",
      "apic",
      "sep",
      "mtrr",
      "pge",
      "mca",
      "cmov",
      "pat",
      "pse36",
      "clfsh",
      "ds",
      "acpi",
      "mmx",
      "fxsr",
      "sse",
      "sse2",
      "ss",
      "htt",
      "tm",
      "pbe",
      "sse3",
      "pclmulqdq",
      "dtes64",
      "mon",
      "dscpl",
      "vmx",
      "smx",
      "est",
      "tm2",
      "ssse3",
      "fma",
      "cx16",
      "tpr",
      "pdcm",
      "sse4.1",
      "sse4.2",
      "x2apic",
      "movbe",
      "popcnt",
      "aes",
      "pcid",
      "xsave",
      "osxsave",
      "seglim64",
      "tsctmr",
      "avx1.0",
      "rdrand",
      "f16c",
      "rdwrfsgs",
      "tsc_thread_offset",
      "sgx",
      "bmi1",
      "hle",
      "avx2",
      "smep",
      "bmi2",
      "erms",
      "invpcid",
      "rtm",
      "fpu_csds",
      "mpx",
      "rdseed",
      "adx",
      "smap",
      "clfsopt",
      "ipt",
      "mdclear",
      "tsxfa",
      "ibrs",
      "stibp",
      "l1df",
      "ssbd",
      "syscall",
      "xd",
      "1gbpage",
      "em64t",
      "lahf",
      "lzcnt",
      "prefetchw",
      "rdtscp",
      "tsci"
    ],
    "cpuModelName": "Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz",
    "os": "darwin",
    "platform": "darwin",
    "platformFamily": "Standalone Workstation",
    "platformVersion": "10.15.5",
    "kernelVersion": "19.5.0",
    "kernelArch": "x86_64"
  },
  "reportTimestamp": 1592903168,
  "trackingId": "aec72db7-794a-4245-bef0-3365d8b53970"
}

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn: TBD
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Manual test

Release note

  • Support telemetry

@breezewish
Add global variable
1411579
@breezewish
Implement
a0de4fe 
TODO: interval need to be changed

Signed-off-by: Breezewish <me@breeswish.org>
@breezewish
Update telemetry status even when it is disabled from config
07beddd 
Signed-off-by: Breezewish <me@breeswish.org>
@breezewish
Update report interval
62df36d 
Signed-off-by: Breezewish <me@breeswish.org>
@breezewish breezewish requested review from a team as code owners June 23, 2020 04:58
@breezewish breezewish requested review from SunRunAway and removed request for a team June 23, 2020 04:58
crazycs520 and others added 2 commits June 23, 2020 13:01
@crazycs520
Merge branch 'master' into telemetry
e4f843f
@breezewish
Make lint happy
89e7832 
Signed-off-by: Breezewish <me@breeswish.org>
@breezewish breezewish mentioned this pull request Jun 23, 2020
Merged
@breezewish
Copy link
Member Author

/label component/executor component/server needs-cherry-pick-4.0 priority/release-blocker

@ti-srebot
Copy link
Contributor

These labels are not found component/executor component/server needs-cherry-pick-4.

@breezewish
Tidy parser
49db344 
Signed-off-by: Breezewish <me@breeswish.org>
@SunRunAway SunRunAway added needs-cherry-pick-4.0 priority/release-blocker This issue blocks a release. Please solve it ASAP. labels Jun 23, 2020
bb7133
bb7133 reviewed Jun 23, 2020
View reviewed changes
executor/admin_telemetry.go Outdated Show resolved Hide resolved
lonng
lonng reviewed Jun 23, 2020
View reviewed changes
domain/domain.go
err := telemetry.ReportUsageData(ctx, do.GetEtcdClient())
if err != nil {
// Only status update errors will be printed out
logutil.BgLogger().Warn("handleTelemetryLoop status update failed", zap.Error(err))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can ignore the error to prevent a bunch of error messages in the offline environment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errors will be recorded in status. Only status update failures (which happens when TiDB cannot connect PD) will be printed out.

lonng
lonng reviewed Jun 23, 2020
View reviewed changes
Copy link
Contributor

@lonng lonng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to put the telemetry package into util instead of make it a top-level package.

@SunRunAway SunRunAway requested review from qw4990 and removed request for a team and SunRunAway June 23, 2020 05:57
@breezewish
Report every 24 hours
3c2e6b6 
Signed-off-by: Breezewish <me@breeswish.org>
@AndreMouche
Copy link
Contributor

/label component/executor, component/server, needs-cherry-pick-4.0, priority/release-blocker

@ti-srebot
Copy link
Contributor

These labels are not found needs-cherry-pick-4.

@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot
Copy link
Contributor

@breeswish merge failed.

@breezewish
Copy link
Member Author

/unlabel can-merge

@breezewish
Copy link
Member Author

@ti-srebot /merge

@ti-srebot
Copy link
Contributor

@breeswish No command or invalid command

@SunRunAway SunRunAway mentioned this pull request Jun 24, 2020
Closed
@zimulala
Copy link
Contributor

@SunRunAway
PTAL

[2020-06-24T05:41:39.836Z] WARNING: DATA RACE
[2020-06-24T05:41:39.836Z] Read at 0x00c05d13ca30 by goroutine 68:
[2020-06-24T05:41:39.836Z]   runtime.chansend()
[2020-06-24T05:41:39.836Z]       /usr/local/go/src/runtime/chan.go:142 +0x0
[2020-06-24T05:41:39.836Z]   github.com/pingcap/tidb/executor.(*innerMergeWorker).doMergeJoin.func1()
[2020-06-24T05:41:39.836Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join.go:535 +0x344
[2020-06-24T05:41:39.836Z]   github.com/pingcap/tidb/executor.(*innerMergeWorker).doMergeJoin()
[2020-06-24T05:41:39.836Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join.go:591 +0x99f
[2020-06-24T05:41:39.836Z]   github.com/pingcap/tidb/executor.(*innerMergeWorker).handleTask()
[2020-06-24T05:41:39.836Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join.go:500 +0xc33
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*innerMergeWorker).run()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join.go:420 +0x28d
[2020-06-24T05:41:39.837Z] 
[2020-06-24T05:41:39.837Z] Previous write at 0x00c05d13ca30 by goroutine 758:
[2020-06-24T05:41:39.837Z]   runtime.closechan()
[2020-06-24T05:41:39.837Z]       /usr/local/go/src/runtime/chan.go:334 +0x0
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*IndexLookUpMergeJoin).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join.go:715 +0xef
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*baseExecutor).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:171 +0xad
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*LimitExec).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:1058 +0x6c
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*baseExecutor).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:171 +0xad
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:317 +0x24e
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor.(*recordSet).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:151 +0x59
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/session.(*execStmtResult).Close()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1246 +0x7b
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).MustExec()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:209 +0x426
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor_test.(*testSuite9).TestIssue18068()
[2020-06-24T05:41:39.837Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/index_lookup_merge_join_test.go:46 +0x469
[2020-06-24T05:41:39.837Z]   github.com/pingcap/tidb/executor_test.(*testSuite9).TestIssue18068()

@zimulala
Copy link
Contributor

/merge

@ti-srebot
Copy link
Contributor

Sorry @zimulala, you don't have permission to trigger auto merge event on this branch. You are not a committer for the related sigs:ddl(slack),execution(slack).

@zimulala
Copy link
Contributor

/run-all-test

@zimulala zimulala added status/can-merge Indicates a PR has been approved by a committer. and removed status/can-merge Indicates a PR has been approved by a committer. labels Jun 24, 2020
@ti-srebot
Copy link
Contributor

Sorry @zimulala, you don't have permission to trigger auto merge event on this branch. You are not a committer for the related sigs:ddl(slack),execution(slack).

@SunRunAway
Copy link
Contributor

/merge

@ti-srebot
Copy link
Contributor

Sorry @SunRunAway, you don't have permission to trigger auto merge event on this branch. You are not a committer for the related sigs:ddl(slack),execution(slack).

@SunRunAway
Copy link
Contributor

/run-unit-test

@SunRunAway SunRunAway merged commit 3b63270 into pingcap:master Jun 24, 2020
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 24, 2020
@breezewish @ti-srebot
cherry pick pingcap#18180 to release-4.0
0aa0183 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Jun 24, 2020
Merged
1 task
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #18197

@SunRunAway
Copy link
Contributor

/merge

@ti-srebot
Copy link
Contributor

Your auto merge job has been accepted, waiting for:

  • 18102

@Yisaer
Copy link
Contributor

Yisaer commented Jun 24, 2020 •
edited
Loading

I think we should check the performance in the Kubernetes for this request. @breeswish @SunRunAway

@breezewish breezewish deleted the telemetry branch June 24, 2020 14:23
jebter pushed a commit that referenced this pull request Jun 24, 2020
@ti-srebot @breezewish
telemetry: Add telemetry initial implementation (#18180) (#18197)
c6e7277 
* cherry pick #18180 to release-4.0

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

* Fix merge conflicts

Signed-off-by: Breezewish <me@breeswish.org>

* Fix different names

Signed-off-by: Breezewish <me@breeswish.org>

* Trigger

Co-authored-by: Wenxuan <hi@breeswish.org>
Co-authored-by: Breezewish <me@breeswish.org>
@breezewish breezewish mentioned this pull request Jun 30, 2020
Closed
@nolouch nolouch mentioned this pull request Sep 17, 2020
Open
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qw4990 qw4990 qw4990 left review comments

@lonng lonng lonng left review comments

@bb7133 bb7133 bb7133 left review comments

@zimulala zimulala zimulala approved these changes

Assignees
No one assigned
Labels
component/config component/server priority/release-blocker This issue blocks a release. Please solve it ASAP. sig/execution SIG execution sig/sql-infra SIG: SQL Infra status/can-merge Indicates a PR has been approved by a committer.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.