Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiFlash process get blocked when cpu_quota is less than 1 cpu #9488

Open
JaySon-Huang opened this issue Sep 27, 2024 · 1 comment
Open

TiFlash process get blocked when cpu_quota is less than 1 cpu #9488

JaySon-Huang opened this issue Sep 27, 2024 · 1 comment
Labels
severity/moderate type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Deploy a tidb cluster using tiup with cpu_quota less than 100%

tiflash_servers:
- host: 127.0.0.1
  ...
  resource_control:
    cpu_quota: 80%

2. What did you expect to see? (Required)

tiflash starts normally or exit with explicit error message

3. What did you see instead (Required)

tiflash blocks forever

4. What is your TiFlash version? (Required)

master

@JaySon-Huang JaySon-Huang added type/bug The issue is confirmed as a bug. severity/moderate labels Sep 27, 2024
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Sep 27, 2024

the logcial cpu is int, so 80% cpu_quota turns into 0 logical CPU cores. And some thread pools inited incorrectly.

[2024/09/27 18:49:50.158 +08:00] [INFO] [getNumberOfCPUCores.cpp:76] ["logical cpu cores: 0, hardware logical cpu cores: 72, hardware physical cpu cores: 36, physical cpu cores: 0, number_of_physical_cpu_cores: 1"] [thread_id=1]
[2024/09/27 18:49:50.158 +08:00] [INFO] [Server.cpp:1093] ["ServerInfo: CPU: \n     logical cores: 0\n     physical cores: 36\n     frequency: 0MHz\n     l1 cache size: 32768\n     l1 cache line size: 64\n     l2 cache size: 1048576\n     l2 cache line size: 64\n     l3 cache size: 25952256\n     l3 cache line size: 64\n     arch: amd64\nDisk: \n     name: /de
v/nvme0n1\n     type: SSD\n     total space: 3936820731904\n     free space: 1425970372608\n     mount point: /DATA/disk1\n     fstype: ext4\nDisk: \n     name: /dev/nvme1n1\n     type: SSD\n     total space: 3936820731904\n     free space: 2052457152512\n     mount point: /DATA/disk2\n     fstype: ext4\nDisk: \n     name: /dev/nvme2n1\n     type: SSD\n     to
tal space: 3936820731904\n     free space: 1677164838912\n     mount point: /DATA/disk3\n     fstype: ext4\nDisk: \n     name: /dev/nvme3n1\n     type: SSD\n     total space: 3936820731904\n     free space: 1707853307904\n     mount point: /DATA/disk4\n     fstype: ext4\nDisk: \n     name: /dev/sda2\n     type: SSD\n     total space: 3953205248\n     free spac
e: 3457921024\n     mount point: /boot\n     fstype: ext4\nDisk: \n     name: /dev/sda3\n     type: SSD\n     total space: 467794132992\n     free space: 234318372864\n     mount point: /\n     fstype: ext4\nDisk: \n     name: /dev/sda3\n     type: SSD\n     total space: 467794132992\n     free space: 234318372864\n     mount point: /var/lib/containers/storage
/overlay\n     fstype: ext4\nMemory: \n     capacity: 403782103040\n"] [thread_id=1]
...
[2024/09/27 18:49:50.832 +08:00] [INFO] [SegmentReader.cpp:146] ["Create start, thread_count=0 cpus=[11, 9, 48, 7, 38, 46, 5, 36, 16, 44, 3, 52, 14, 42, 1, 50, 12, 40, 49, 10, 8, 39, 47, 6, 37, 17, 45, 4, 53, 15, 43, 2, 51, 13, 41, 0]"] [thread_id=1]
[2024/09/27 18:49:50.832 +08:00] [INFO] [SegmentReader.cpp:151] ["Create end, thread_count=0 cpus=[11, 9, 48, 7, 38, 46, 5, 36, 16, 44, 3, 52, 14, 42, 1, 50, 12, 40, 49, 10, 8, 39, 47, 6, 37, 17, 45, 4, 53, 15, 43, 2, 51, 13, 41, 0]"] [thread_id=1]
[2024/09/27 18:49:50.832 +08:00] [INFO] [SegmentReader.cpp:146] ["Create start, thread_count=0 cpus=[21, 68, 58, 66, 28, 56, 18, 64, 26, 54, 34, 62, 24, 70, 32, 60, 22, 69, 30, 59, 20, 67, 29, 57, 19, 65, 27, 55, 35, 63, 25, 71, 33, 61, 23, 31]"] [thread_id=1]
[2024/09/27 18:49:50.832 +08:00] [INFO] [SegmentReader.cpp:151] ["Create end, thread_count=0 cpus=[21, 68, 58, 66, 28, 56, 18, 64, 26, 54, 34, 62, 24, 70, 32, 60, 22, 69, 30, 59, 20, 67, 29, 57, 19, 65, 27, 55, 35, 63, 25, 71, 33, 61, 23, 31]"] [thread_id=1]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

1 participant