Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI][SOT] Remove package github installation in CI and update SOT trigger file list #63848

Merged

Conversation

SigureMo
Copy link
Member

@SigureMo SigureMo commented Apr 24, 2024

PR Category

Environment Adaptation

PR Types

Bug fixes

Description

#63189paddle_build.sh 添加 pip install github 导致 PR-CI-SOT 流水线在安装 3.11 依赖(run_setup)的时候挂掉,如 https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/10555965/job/26019581

2024-04-24 23:24:17 + pip install github
2024-04-24 23:24:18 DEPRECATION: Loading egg at /usr/local/lib/python3.11/dist-packages/pip-23.3.1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
2024-04-24 23:24:18 DEPRECATION: Loading egg at /usr/local/lib/python3.11/dist-packages/setuptools-68.2.2-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
2024-04-24 23:24:20 Collecting github
2024-04-24 23:24:20   Using cached github-1.2.7-py3-none-any.whl.metadata (1.7 kB)
2024-04-24 23:24:22 Collecting aiohttp==3.8.1 (from github)
2024-04-24 23:24:23   Using cached aiohttp-3.8.1.tar.gz (7.3 MB)
2024-04-24 23:24:23   Installing build dependencies: started
2024-04-24 23:24:29   Installing build dependencies: finished with status 'done'
2024-04-24 23:24:29   Getting requirements to build wheel: started
2024-04-24 23:24:30   Getting requirements to build wheel: finished with status 'done'
2024-04-24 23:24:30   Installing backend dependencies: started
2024-04-24 23:24:35   Installing backend dependencies: finished with status 'done'
2024-04-24 23:24:35   Preparing metadata (pyproject.toml): started
2024-04-24 23:24:35   Preparing metadata (pyproject.toml): finished with status 'done'
2024-04-24 23:24:35 Requirement already satisfied: typing-extensions in /usr/local/lib/python3.11/dist-packages (from github) (4.10.0)
2024-04-24 23:24:35 Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp==3.8.1->github) (23.2.0)
2024-04-24 23:24:36 Collecting charset-normalizer<3.0,>=2.0 (from aiohttp==3.8.1->github)
2024-04-24 23:24:37   Using cached charset_normalizer-2.1.1-py3-none-any.whl.metadata (11 kB)
2024-04-24 23:24:39 Collecting multidict<7.0,>=4.5 (from aiohttp==3.8.1->github)
2024-04-24 23:24:39   Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
2024-04-24 23:24:41 Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp==3.8.1->github)
2024-04-24 23:24:41   Using cached async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
2024-04-24 23:24:43 Collecting yarl<2.0,>=1.0 (from aiohttp==3.8.1->github)
2024-04-24 23:24:43   Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
2024-04-24 23:24:45 Collecting frozenlist>=1.1.1 (from aiohttp==3.8.1->github)
2024-04-24 23:24:45   Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
2024-04-24 23:24:47 Collecting aiosignal>=1.1.2 (from aiohttp==3.8.1->github)
2024-04-24 23:24:47   Using cached aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
2024-04-24 23:24:47 Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.0->aiohttp==3.8.1->github) (2.8)
2024-04-24 23:24:48 Using cached github-1.2.7-py3-none-any.whl (15 kB)
2024-04-24 23:24:48 Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
2024-04-24 23:24:49 Using cached async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
2024-04-24 23:24:49 Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
2024-04-24 23:24:50 Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (272 kB)
2024-04-24 23:24:50 Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB)
2024-04-24 23:24:51 Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (328 kB)
2024-04-24 23:24:51 Building wheels for collected packages: aiohttp
2024-04-24 23:24:51   Building wheel for aiohttp (pyproject.toml): started
2024-04-24 23:24:51   Building wheel for aiohttp (pyproject.toml): finished with status 'error'
2024-04-24 23:24:51   error: subprocess-exited-with-error
2024-04-24 23:24:51   
2024-04-24 23:24:51   ?? Building wheel for aiohttp (pyproject.toml) did not run successfully.
2024-04-24 23:24:51   ??? exit code: 1
2024-04-24 23:24:51   ??????> [107 lines of output]
2024-04-24 23:24:51       *********************
2024-04-24 23:24:51       * Accelerated build *
2024-04-24 23:24:51       *********************
2024-04-24 23:24:51       running bdist_wheel
2024-04-24 23:24:51       running build
2024-04-24 23:24:51       running build_py
2024-04-24 23:24:51       creating build
2024-04-24 23:24:51       creating build/lib.linux-x86_64-cpython-311
2024-04-24 23:24:51       creating build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/typedefs.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/resolver.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_response.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/client_reqrep.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_server.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/__init__.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/hdrs.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/payload.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_log.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/base_protocol.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/abc.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_urldispatcher.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_middlewares.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/http.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/client_exceptions.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/http_writer.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_routedef.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_app.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_fileresponse.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/worker.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_protocol.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_ws.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/client_proto.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/http_exceptions.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/log.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/http_websocket.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/test_utils.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/http_parser.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/locks.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/client.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/tracing.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_runner.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/client_ws.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_request.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/cookiejar.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/tcp_helpers.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/formdata.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/web_exceptions.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/multipart.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/pytest_plugin.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/payload_streamer.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/streams.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/connector.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/helpers.py -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       running egg_info
2024-04-24 23:24:51       writing aiohttp.egg-info/PKG-INFO
2024-04-24 23:24:51       writing dependency_links to aiohttp.egg-info/dependency_links.txt
2024-04-24 23:24:51       writing requirements to aiohttp.egg-info/requires.txt
2024-04-24 23:24:51       writing top-level names to aiohttp.egg-info/top_level.txt
2024-04-24 23:24:51       reading manifest file 'aiohttp.egg-info/SOURCES.txt'
2024-04-24 23:24:51       reading manifest template 'MANIFEST.in'
2024-04-24 23:24:51       warning: no files found matching 'aiohttp' anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.pyc' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.pyd' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.so' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.lib' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.dll' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.a' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files matching '*.obj' found anywhere in distribution
2024-04-24 23:24:51       warning: no previously-included files found matching 'aiohttp/*.html'
2024-04-24 23:24:51       no previously-included directories found matching 'docs/_build'
2024-04-24 23:24:51       adding license file 'LICENSE.txt'
2024-04-24 23:24:51       writing manifest file 'aiohttp.egg-info/SOURCES.txt'
2024-04-24 23:24:51       copying aiohttp/_cparser.pxd -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_find_header.c -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_find_header.h -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_find_header.pxd -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_headers.pxi -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_helpers.c -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_helpers.pyi -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_helpers.pyx -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_http_parser.c -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_http_parser.pyx -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_http_writer.c -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_http_writer.pyx -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_websocket.c -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/_websocket.pyx -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       copying aiohttp/py.typed -> build/lib.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       creating build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_cparser.pxd.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_find_header.pxd.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_helpers.pyi.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_helpers.pyx.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_http_parser.pyx.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_http_writer.pyx.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/_websocket.pyx.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       copying aiohttp/.hash/hdrs.py.hash -> build/lib.linux-x86_64-cpython-311/aiohttp/.hash
2024-04-24 23:24:51       running build_ext
2024-04-24 23:24:51       building 'aiohttp._websocket' extension
2024-04-24 23:24:51       creating build/temp.linux-x86_64-cpython-311
2024-04-24 23:24:51       creating build/temp.linux-x86_64-cpython-311/aiohttp
2024-04-24 23:24:51       x86_64-linux-gnu-gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.11 -c aiohttp/_websocket.c -o build/temp.linux-x86_64-cpython-311/aiohttp/_websocket.o
2024-04-24 23:24:51       aiohttp/_websocket.c:198:12: fatal error: longintrepr.h: No such file or directory
2024-04-24 23:24:51         198 |   #include "longintrepr.h"
2024-04-24 23:24:51             |            ^~~~~~~~~~~~~~~
2024-04-24 23:24:51       compilation terminated.
2024-04-24 23:24:51       error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
2024-04-24 23:24:51       [end of output]
2024-04-24 23:24:51   
2024-04-24 23:24:51   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-04-24 23:24:51   ERROR: Failed building wheel for aiohttp
2024-04-24 23:24:51 Failed to build aiohttp
2024-04-24 23:24:51 ERROR: Could not build wheels for aiohttp, which is required to install pyproject.toml-based projects
2024-04-24 23:24:52 + EXCODE=1

这里 Python 3.11 装了 githubgithub 依赖 aiohttp>=3.8.1,虽然并不是很清楚 resolve 后仍然选择的是最低版本 aiohttp 3.8.1,但就 3.8.1 来看,明显是没有 Python 3.11 的预编译 wheel 包的,触发 build 导致挂掉(这个版本应该也没适配 Python 3.11,挂掉很正常)

PyPI 里 github 这个包来自于 https://github.com/VarMonke/Github-Api-Wrapper,它与 PyGithub 是有冲突的,它们同样通过 import github 来使用,但使用方式完全不同,我们现有脚本里明显都是使用 PyGithub 的,github 是没有用的,因此直接删掉就好了

另外,没能成功拦截 #63189 的原因是 SOT CI 只对修改部分文件的情况下才会触发,而 paddle_build.sh 并不在其中,因此修改监控文件列表,避免类似问题

PCard-66972

Copy link

paddle-bot bot commented Apr 24, 2024

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@SigureMo SigureMo changed the title [CI] Remove github installation in CI [CI] Remove package github installation in CI Apr 24, 2024
@SigureMo SigureMo requested a review from gouzil April 24, 2024 17:19
@SigureMo SigureMo changed the title [CI] Remove package github installation in CI [CI][SOT] Remove package github installation in CI Apr 24, 2024
@SigureMo SigureMo changed the title [CI][SOT] Remove package github installation in CI [CI][SOT] Remove package github installation in CI and update SOT check file list Apr 24, 2024
@SigureMo SigureMo changed the title [CI][SOT] Remove package github installation in CI and update SOT check file list [CI][SOT] Remove package github installation in CI and update SOT trigger file list Apr 24, 2024
@SigureMo SigureMo merged commit d7da07f into PaddlePaddle:develop Apr 25, 2024
28 of 30 checks passed
@SigureMo SigureMo deleted the ci/remove-github-installation-in-ci branch April 25, 2024 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants