Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Defect: 使用 httpx 情况下 ForwardDriver.request 在并发时存在性能问题 #2617

Closed
Ailitonia opened this issue Mar 31, 2024 · 1 comment · Fixed by #2627
Closed

Defect: 使用 httpx 情况下 ForwardDriver.request 在并发时存在性能问题 #2617

Ailitonia opened this issue Mar 31, 2024 · 1 comment · Fixed by #2627
Labels
enhancement New feature or request

Comments

@Ailitonia
Copy link
Contributor

操作系统

Windows

Python 版本

3.11.8

NoneBot 版本

2.2.1

适配器

无

协议端

无

描述问题

问题

如题, ForwardDriver.request 在 3.11 与 3.10 并发时性能存在明显差异, 3.11 中存在明显的性能下降

不排除是运行环境、硬件等造成的, 请求更多测试

原因

可能是由于 httpx 库造成的

造成的影响

可能会使所有使用客户端型驱动器的 Adapter 产生性能问题

复现步骤

项目依赖环境

[tool.poetry.dependencies]
python = "^3.10"
nonebot2 = {version = "2.2.1", extras = ["fastapi", "aiohttp", "httpx", "websockets"]}
pydantic = ">=2.6.4,<2.7.0"
nonebot-adapter-console = {version = "^0.5.0", optional = true}
nonebot-adapter-onebot = {version = "^2.4.3", optional = true}
nonebot-adapter-qq = {version = "^1.4.2", optional = true}
nonebot-adapter-telegram = {version = "^0.1.0b17", optional = true}
sqlalchemy = ">=2.0.22,<3.0.0"
asyncmy = {version = "^0.2.9", optional = true}
aiomysql = {version = "^0.2.0", optional = true}
asyncpg = {version = "^0.29.0", optional = true}
aiosqlite = {version = "^0.20.0", optional = true}
apscheduler = "^3.10.4"
aiofiles = "^23.2.1"
ujson = "^5.9.0"
lxml = "^5.1.0"
msgpack = "^1.0.8"
numpy = "^1.26.4"
matplotlib = "^3.8.3"
pillow = "^10.2.0"
imageio = "^2.34.0"
psutil = "^5.9.8"
pycryptodome = "^3.20.0"
py7zr = "^0.21.0"
pytz = "^2024.1"
zhconv = "^1.4.3"
rapidfuzz = "^3.6.2"
emoji = "^2.10.1"
openpyxl = "^3.1.2"

使用代码

import asyncio
import datetime
import sys
from typing import cast

import nonebot
from nonebot import get_driver
from nonebot.drivers import Request, HTTPClientMixin


async def get(url):
    start_time = datetime.datetime.now()
    print(f'starting get {url} at {start_time}')

    driver = cast(HTTPClientMixin, get_driver())
    setup = Request(method='GET', url=url)
    response = await driver.request(setup=setup)

    end_time = datetime.datetime.now()
    print(f'ending get {url} at {end_time}, using time {end_time - start_time}')
    return response


async def main():
    tasks = [get('https://baidu.com') for _ in range(20)]
    print(await asyncio.gather(*tasks))


if __name__ == '__main__':
    print(sys.version)
    nonebot.init(driver='~fastapi+~httpx')
    nonebot.get_driver().on_startup(main)
    nonebot.run()

分别在 3.10.11 和 3.11.8 环境中执行

结果

3.10.11 和 3.11.8 两种环境下有 10 倍甚至 9 倍 非常明显的的性能差距

3.10.11

C:\Users\Ailit\PycharmProjects\nonebot2_miya\venv310\Scripts\python.exe C:\Users\Ailit\PycharmProjects\nonebot2_miya\test\v2\async_test.py 
3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
03-31 16:27:03 [SUCCESS] nonebot | NoneBot is initializing...
03-31 16:27:03 [INFO] nonebot | Current Env: prod
03-31 16:27:03 [SUCCESS] nonebot | Running NoneBot...
03-31 16:27:03 [INFO] uvicorn | Started server process [26012]
03-31 16:27:03 [INFO] uvicorn | Waiting for application startup.
starting get https://baidu.com at 2024-03-31 16:27:03.982470
starting get https://baidu.com at 2024-03-31 16:27:03.991470
starting get https://baidu.com at 2024-03-31 16:27:03.997470
starting get https://baidu.com at 2024-03-31 16:27:04.003470
starting get https://baidu.com at 2024-03-31 16:27:04.009470
starting get https://baidu.com at 2024-03-31 16:27:04.015473
starting get https://baidu.com at 2024-03-31 16:27:04.021470
starting get https://baidu.com at 2024-03-31 16:27:04.027471
starting get https://baidu.com at 2024-03-31 16:27:04.033471
starting get https://baidu.com at 2024-03-31 16:27:04.039471
starting get https://baidu.com at 2024-03-31 16:27:04.044974
starting get https://baidu.com at 2024-03-31 16:27:04.049974
starting get https://baidu.com at 2024-03-31 16:27:04.055974
starting get https://baidu.com at 2024-03-31 16:27:04.060974
starting get https://baidu.com at 2024-03-31 16:27:04.066974
starting get https://baidu.com at 2024-03-31 16:27:04.071974
starting get https://baidu.com at 2024-03-31 16:27:04.077589
starting get https://baidu.com at 2024-03-31 16:27:04.083589
starting get https://baidu.com at 2024-03-31 16:27:04.088589
starting get https://baidu.com at 2024-03-31 16:27:04.094209
ending get https://baidu.com at 2024-03-31 16:27:04.474663, using time 0:00:00.447192
ending get https://baidu.com at 2024-03-31 16:27:04.478257, using time 0:00:00.495787
ending get https://baidu.com at 2024-03-31 16:27:04.478257, using time 0:00:00.428283
ending get https://baidu.com at 2024-03-31 16:27:04.479521, using time 0:00:00.482051
ending get https://baidu.com at 2024-03-31 16:27:04.481523, using time 0:00:00.442052
ending get https://baidu.com at 2024-03-31 16:27:04.483989, using time 0:00:00.492519
ending get https://baidu.com at 2024-03-31 16:27:04.492864, using time 0:00:00.436890
ending get https://baidu.com at 2024-03-31 16:27:04.510417, using time 0:00:00.500947
ending get https://baidu.com at 2024-03-31 16:27:04.511417, using time 0:00:00.427828
ending get https://baidu.com at 2024-03-31 16:27:04.512417, using time 0:00:00.496944
ending get https://baidu.com at 2024-03-31 16:27:04.513417, using time 0:00:00.452443
ending get https://baidu.com at 2024-03-31 16:27:04.514083, using time 0:00:00.469109
ending get https://baidu.com at 2024-03-31 16:27:04.515669, using time 0:00:00.482198
ending get https://baidu.com at 2024-03-31 16:27:04.521670, using time 0:00:00.433081
ending get https://baidu.com at 2024-03-31 16:27:04.535537, using time 0:00:00.441328
ending get https://baidu.com at 2024-03-31 16:27:04.538109, using time 0:00:00.534639
ending get https://baidu.com at 2024-03-31 16:27:04.544011, using time 0:00:00.477037
ending get https://baidu.com at 2024-03-31 16:27:04.547010, using time 0:00:00.469421
ending get https://baidu.com at 2024-03-31 16:27:04.548011, using time 0:00:00.526541
ending get https://baidu.com at 2024-03-31 16:27:04.566239, using time 0:00:00.494265
[Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200)]
03-31 16:27:04 [INFO] uvicorn | Application startup complete.
03-31 16:27:04 [INFO] uvicorn | Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

3.11.8

C:\Users\Ailit\PycharmProjects\nonebot2_miya\venv\Scripts\python.exe C:\Users\Ailit\PycharmProjects\nonebot2_miya\test\v2\async_test.py 
3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]
03-31 16:28:16 [SUCCESS] nonebot | NoneBot is initializing...
03-31 16:28:16 [INFO] nonebot | Current Env: prod
03-31 16:28:16 [SUCCESS] nonebot | Running NoneBot...
03-31 16:28:16 [INFO] uvicorn | Started server process [7052]
03-31 16:28:16 [INFO] uvicorn | Waiting for application startup.
starting get https://baidu.com at 2024-03-31 16:28:16.553693
starting get https://baidu.com at 2024-03-31 16:28:16.931345
starting get https://baidu.com at 2024-03-31 16:28:17.307382
starting get https://baidu.com at 2024-03-31 16:28:17.677874
starting get https://baidu.com at 2024-03-31 16:28:18.054005
starting get https://baidu.com at 2024-03-31 16:28:18.424357
starting get https://baidu.com at 2024-03-31 16:28:18.793891
starting get https://baidu.com at 2024-03-31 16:28:19.163367
starting get https://baidu.com at 2024-03-31 16:28:19.531708
starting get https://baidu.com at 2024-03-31 16:28:19.900126
starting get https://baidu.com at 2024-03-31 16:28:20.270024
starting get https://baidu.com at 2024-03-31 16:28:20.639933
starting get https://baidu.com at 2024-03-31 16:28:21.009723
starting get https://baidu.com at 2024-03-31 16:28:21.379113
starting get https://baidu.com at 2024-03-31 16:28:21.756936
starting get https://baidu.com at 2024-03-31 16:28:22.133422
starting get https://baidu.com at 2024-03-31 16:28:22.505594
starting get https://baidu.com at 2024-03-31 16:28:22.875434
starting get https://baidu.com at 2024-03-31 16:28:23.245732
starting get https://baidu.com at 2024-03-31 16:28:23.615300
ending get https://baidu.com at 2024-03-31 16:28:24.308571, using time 0:00:03.298848
ending get https://baidu.com at 2024-03-31 16:28:24.316799, using time 0:00:04.046775
ending get https://baidu.com at 2024-03-31 16:28:24.334603, using time 0:00:01.829009
ending get https://baidu.com at 2024-03-31 16:28:24.338694, using time 0:00:01.092962
ending get https://baidu.com at 2024-03-31 16:28:24.339378, using time 0:00:06.661504
ending get https://baidu.com at 2024-03-31 16:28:24.342182, using time 0:00:05.548291
ending get https://baidu.com at 2024-03-31 16:28:24.344065, using time 0:00:05.919708
ending get https://baidu.com at 2024-03-31 16:28:24.345066, using time 0:00:05.181699
ending get https://baidu.com at 2024-03-31 16:28:24.347370, using time 0:00:02.590434
ending get https://baidu.com at 2024-03-31 16:28:24.348007, using time 0:00:07.416662
ending get https://baidu.com at 2024-03-31 16:28:24.348509, using time 0:00:00.733209
ending get https://baidu.com at 2024-03-31 16:28:24.350591, using time 0:00:07.043209
ending get https://baidu.com at 2024-03-31 16:28:24.353281, using time 0:00:07.799588
ending get https://baidu.com at 2024-03-31 16:28:24.358896, using time 0:00:02.979783
ending get https://baidu.com at 2024-03-31 16:28:24.360140, using time 0:00:02.226718
ending get https://baidu.com at 2024-03-31 16:28:24.366531, using time 0:00:04.466405
ending get https://baidu.com at 2024-03-31 16:28:24.367531, using time 0:00:06.313526
ending get https://baidu.com at 2024-03-31 16:28:24.371531, using time 0:00:01.496097
ending get https://baidu.com at 2024-03-31 16:28:24.392803, using time 0:00:03.752870
ending get https://baidu.com at 2024-03-31 16:28:24.410935, using time 0:00:04.879227
[Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200)]
03-31 16:28:24 [INFO] uvicorn | Application startup complete.
03-31 16:28:24 [INFO] uvicorn | Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

可能的原因

ForwardDriver 在每次 request 时都新建 Client 导致额外的性能开销, 多次请求共享 Client 则不会有此问题,

nonebot2/nonebot/drivers/httpx.py

Lines 48 to 54 in 142a61c

async def request(self, setup: Request) -> Response:
async with httpx.AsyncClient(
cookies=setup.cookies.jar,
http2=setup.version == HTTPVersion.H2,
proxies=setup.proxy,
follow_redirects=True,
) as client:

其他

意外的是, aiohttp 则没有上述问题, 在 3.10 和 3.11 均表现一致的性能

    nonebot.init(driver='~fastapi+~aiohttp')

3.10.11

C:\Users\Ailit\PycharmProjects\nonebot2_miya\venv310\Scripts\python.exe C:\Users\Ailit\PycharmProjects\nonebot2_miya\test\v2\async_test.py 
3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
03-31 16:37:12 [SUCCESS] nonebot | NoneBot is initializing...
03-31 16:37:12 [INFO] nonebot | Current Env: prod
03-31 16:37:12 [SUCCESS] nonebot | Running NoneBot...
03-31 16:37:12 [INFO] uvicorn | Started server process [27440]
03-31 16:37:12 [INFO] uvicorn | Waiting for application startup.
starting get https://baidu.com at 2024-03-31 16:37:12.718504
starting get https://baidu.com at 2024-03-31 16:37:12.727504
starting get https://baidu.com at 2024-03-31 16:37:12.728504
starting get https://baidu.com at 2024-03-31 16:37:12.728504
starting get https://baidu.com at 2024-03-31 16:37:12.728504
starting get https://baidu.com at 2024-03-31 16:37:12.728504
starting get https://baidu.com at 2024-03-31 16:37:12.729504
starting get https://baidu.com at 2024-03-31 16:37:12.729504
starting get https://baidu.com at 2024-03-31 16:37:12.729504
starting get https://baidu.com at 2024-03-31 16:37:12.729504
starting get https://baidu.com at 2024-03-31 16:37:12.730504
starting get https://baidu.com at 2024-03-31 16:37:12.730504
starting get https://baidu.com at 2024-03-31 16:37:12.730504
starting get https://baidu.com at 2024-03-31 16:37:12.730504
starting get https://baidu.com at 2024-03-31 16:37:12.730504
starting get https://baidu.com at 2024-03-31 16:37:12.731618
starting get https://baidu.com at 2024-03-31 16:37:12.731618
starting get https://baidu.com at 2024-03-31 16:37:12.732120
starting get https://baidu.com at 2024-03-31 16:37:12.732120
starting get https://baidu.com at 2024-03-31 16:37:12.732120
ending get https://baidu.com at 2024-03-31 16:37:13.044873, using time 0:00:00.313255
ending get https://baidu.com at 2024-03-31 16:37:13.045145, using time 0:00:00.317641
ending get https://baidu.com at 2024-03-31 16:37:13.066502, using time 0:00:00.336998
ending get https://baidu.com at 2024-03-31 16:37:13.070809, using time 0:00:00.338689
ending get https://baidu.com at 2024-03-31 16:37:13.073724, using time 0:00:00.341604
ending get https://baidu.com at 2024-03-31 16:37:13.080353, using time 0:00:00.349849
ending get https://baidu.com at 2024-03-31 16:37:13.083827, using time 0:00:00.353323
ending get https://baidu.com at 2024-03-31 16:37:13.084328, using time 0:00:00.355824
ending get https://baidu.com at 2024-03-31 16:37:13.089450, using time 0:00:00.357330
ending get https://baidu.com at 2024-03-31 16:37:13.096786, using time 0:00:00.366282
ending get https://baidu.com at 2024-03-31 16:37:13.097394, using time 0:00:00.368890
ending get https://baidu.com at 2024-03-31 16:37:13.097394, using time 0:00:00.367890
ending get https://baidu.com at 2024-03-31 16:37:13.105054, using time 0:00:00.386550
ending get https://baidu.com at 2024-03-31 16:37:13.105555, using time 0:00:00.377051
ending get https://baidu.com at 2024-03-31 16:37:13.112123, using time 0:00:00.382619
ending get https://baidu.com at 2024-03-31 16:37:13.116240, using time 0:00:00.385736
ending get https://baidu.com at 2024-03-31 16:37:13.126906, using time 0:00:00.398402
ending get https://baidu.com at 2024-03-31 16:37:13.126906, using time 0:00:00.397402
ending get https://baidu.com at 2024-03-31 16:37:13.129273, using time 0:00:00.397655
ending get https://baidu.com at 2024-03-31 16:37:13.146615, using time 0:00:00.416111
[Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200)]
03-31 16:37:13 [INFO] uvicorn | Application startup complete.
03-31 16:37:13 [INFO] uvicorn | Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

3.11.8

C:\Users\Ailit\PycharmProjects\nonebot2_miya\venv\Scripts\python.exe C:\Users\Ailit\PycharmProjects\nonebot2_miya\test\v2\async_test.py 
3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]
03-31 16:37:26 [SUCCESS] nonebot | NoneBot is initializing...
03-31 16:37:26 [INFO] nonebot | Current Env: prod
03-31 16:37:26 [SUCCESS] nonebot | Running NoneBot...
03-31 16:37:26 [INFO] uvicorn | Started server process [23040]
03-31 16:37:26 [INFO] uvicorn | Waiting for application startup.
starting get https://baidu.com at 2024-03-31 16:37:26.920084
starting get https://baidu.com at 2024-03-31 16:37:26.934043
starting get https://baidu.com at 2024-03-31 16:37:26.935043
starting get https://baidu.com at 2024-03-31 16:37:26.935043
starting get https://baidu.com at 2024-03-31 16:37:26.935043
starting get https://baidu.com at 2024-03-31 16:37:26.936043
starting get https://baidu.com at 2024-03-31 16:37:26.936043
starting get https://baidu.com at 2024-03-31 16:37:26.936043
starting get https://baidu.com at 2024-03-31 16:37:26.937043
starting get https://baidu.com at 2024-03-31 16:37:26.937043
starting get https://baidu.com at 2024-03-31 16:37:26.937043
starting get https://baidu.com at 2024-03-31 16:37:26.938043
starting get https://baidu.com at 2024-03-31 16:37:26.938043
starting get https://baidu.com at 2024-03-31 16:37:26.938043
starting get https://baidu.com at 2024-03-31 16:37:26.938043
starting get https://baidu.com at 2024-03-31 16:37:26.939043
starting get https://baidu.com at 2024-03-31 16:37:26.939043
starting get https://baidu.com at 2024-03-31 16:37:26.939043
starting get https://baidu.com at 2024-03-31 16:37:26.939043
starting get https://baidu.com at 2024-03-31 16:37:26.940043
ending get https://baidu.com at 2024-03-31 16:37:27.282537, using time 0:00:00.344494
ending get https://baidu.com at 2024-03-31 16:37:27.288856, using time 0:00:00.349813
ending get https://baidu.com at 2024-03-31 16:37:27.290611, using time 0:00:00.370527
ending get https://baidu.com at 2024-03-31 16:37:27.291719, using time 0:00:00.352676
ending get https://baidu.com at 2024-03-31 16:37:27.298761, using time 0:00:00.359718
ending get https://baidu.com at 2024-03-31 16:37:27.299262, using time 0:00:00.359219
ending get https://baidu.com at 2024-03-31 16:37:27.304273, using time 0:00:00.369230
ending get https://baidu.com at 2024-03-31 16:37:27.304697, using time 0:00:00.368654
ending get https://baidu.com at 2024-03-31 16:37:27.305940, using time 0:00:00.370897
ending get https://baidu.com at 2024-03-31 16:37:27.312162, using time 0:00:00.378119
ending get https://baidu.com at 2024-03-31 16:37:27.312678, using time 0:00:00.375635
ending get https://baidu.com at 2024-03-31 16:37:27.319581, using time 0:00:00.381538
ending get https://baidu.com at 2024-03-31 16:37:27.326453, using time 0:00:00.389410
ending get https://baidu.com at 2024-03-31 16:37:27.329068, using time 0:00:00.390025
ending get https://baidu.com at 2024-03-31 16:37:27.333185, using time 0:00:00.396142
ending get https://baidu.com at 2024-03-31 16:37:27.336978, using time 0:00:00.401935
ending get https://baidu.com at 2024-03-31 16:37:27.340575, using time 0:00:00.402532
ending get https://baidu.com at 2024-03-31 16:37:27.344610, using time 0:00:00.408567
ending get https://baidu.com at 2024-03-31 16:37:27.347666, using time 0:00:00.411623
ending get https://baidu.com at 2024-03-31 16:37:27.350981, using time 0:00:00.412938
[Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200), Response(status_code=200)]
03-31 16:37:27 [INFO] uvicorn | Application startup complete.
03-31 16:37:27 [INFO] uvicorn | Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

期望的结果

No response

截图或日志

No response
@Ailitonia Ailitonia added the bug Something isn't working label Mar 31, 2024
@yanyongyu yanyongyu added enhancement New feature or request and removed bug Something isn't working labels Mar 31, 2024
@yanyongyu
Copy link
Member

可以提供一个获取session的方法来共享client

@yanyongyu yanyongyu mentioned this issue Apr 4, 2024
Merged
@Ailitonia Ailitonia mentioned this issue Jun 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature: 支持 HTTP 客户端会话 nonebot/nonebot2
2 participants
@Ailitonia @yanyongyu