Jenkins builds failing due to lack of free space #428

jleveque · 2017-03-23T17:49:31Z

https://sonic-jenkins.westus.cloudapp.azure.com/job/common/job/buildimage-baseimage-pr/413/:

make[4]: Entering directory '/sonic/src/sonic-linux-kernel/linux-3.16.36'
dh_installdocs 
dh_installchangelogs
dh_strip
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol -- -Vkernel:Recommends=irqbalance,
dh_md5sums
dh_builddeb -- -Zxz 
dpkg-deb: building package `linux-image-3.16.0-4-amd64' in `../linux-image-3.16.0-4-amd64_3.16.36-1+deb8u2_amd64.deb'.
make[4]: Leaving directory '/sonic/src/sonic-linux-kernel/linux-3.16.36'
dpkg-deb (subprocess): compressing data member: lzma write error: No space left on device
dpkg-deb: error: subprocess <compress> from tar -cf returned error exit status 2
dh_builddeb: dpkg-deb -Zxz --build debian/linux-image-3.16.0-4-amd64-dbg .. returned exit code 2
debian/rules.real:173: recipe for target 'install-base' failed

https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all-pr/83/:

running build
running build_ext
error: [Errno 28] No space left on device: 'build/temp.linux-x86_64-3.6-pydebug'
Makefile:627: recipe for target 'sharedmods' failed
make[3]: *** [sharedmods] Error 1

https://sonic-jenkins.westus.cloudapp.azure.com/job/cavium/job/buildimage-cavm-all-pr/410/:

/tmp/ccJe73Wu.s: Assembler messages:
/tmp/ccJe73Wu.s: Fatal error: can't write tests-converter_ut.o: No space left on device

The text was updated successfully, but these errors were encountered:

stcheng · 2017-03-28T00:43:22Z

https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all/130/console

dpkg-deb: building package `swss' in `../swss_1.0.0_amd64.deb'.
dpkg-deb (subprocess): compressing control member: internal gzip write error: No space left on device
dpkg-deb: error: subprocess gzip -9c returned error exit status 2
dh_builddeb: dpkg-deb --build debian/swss .. returned exit code 2
dpkg-deb: building package `swss-dbg' in `../swss-dbg_1.0.0_amd64.deb'.
dpkg-deb (subprocess): compressing control member: internal gzip write error: No space left on device
dpkg-deb: error: subprocess gzip -9c returned error exit status 2
dh_builddeb: dpkg-deb --build debian/swss-dbg .. returned exit code 2
debian/rules:22: recipe for target 'binary' failed

lguohan · 2017-04-06T18:26:07Z

allocated new 1T disk for jenkins build.

stcheng · 2017-04-14T00:55:58Z

fatal error: error writing to /tmp/ccCK5Whn.s: No space left on device

lguohan · 2017-04-14T06:57:31Z

sonic-slave consume the disk, manually cleaned them up. Should be good for a while. Will figure out way to automatically clean up the obseleted sonic-slave

stcheng · 2017-05-01T22:29:16Z

#560
https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/765/console

jleveque · 2017-05-01T22:47:19Z

Also: https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/764/

stcheng · 2017-05-02T05:17:17Z

Also: https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/771/consoleFull

jleveque · 2017-07-24T19:45:28Z

Looks like it's happening once again: https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all-pr/752/console

stcheng · 2017-07-26T02:15:57Z

🚑

lguohan · 2017-07-26T05:30:12Z

on worker-1 we have 221G flash for the docker

/dev/sdb1 221G 126G 84G 61% /mnt

it looks like when there are 10 concurrent jobs running at the same time, the /mnt partition will become full.

I clean up the /mnt partition about 40G garbage data. If this disk partition happens again, I will increase the /mnt partition to 512G.

jleveque · 2017-11-10T22:44:04Z

Closing as we haven't experienced this for a while now.

Submodule src/sonic-sairedis 54b02a1..8182916: > Add pre match to comparison logic and unittests (sonic-net#423) > Drop FDB notifications if they contain invalid OIDs (sonic-net#428) Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Submodule src/sonic-sairedis 54b02a1..8182916: > Add pre match to comparison logic and unittests (#423) > Drop FDB notifications if they contain invalid OIDs (#428) Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Submodule src/sonic-sairedis 54b02a1..8182916: > Add pre match to comparison logic and unittests (sonic-net#423) > Drop FDB notifications if they contain invalid OIDs (sonic-net#428) Signed-off-by: Ying Xie <ying.xie@microsoft.com>

…2755) sonic-sairedis: e8cb879 Make object list deterministic when iterating (#438) 5486f97 Ignore ACL_COUNTER bytes and packets during comparison logic (#443) b138ff9 Notify OA about exception and process only restart query events (#437) 0974a43 Set MTU value on created tap device for virtual switch (#436) e2f50e8 Increase eth buffer size to 16k for virtual switch (#435) 79fb388 [bfn] Ged rid of ld_preload. Link against libsai only (#429) 2e47b78 Remove MAC alignment WA for Mellanox platforms. (#430) e1354fe Add pre match to comparison logic and unittests (#423) 18a5ebb Drop FDB notifications if they contain invalid OIDs (#428) sonic-swss-common: 8af58ad sonic-swss-common: Add vxlan macros to schema (#269) 76837bf Make class Select support batch read from selectables (#270) 4cf643e Add multiple fields hdel support (#267) a710529 Update PFC_WD table name in CONFIG_DB (#266) 3c452c1 Update README.md (#268) sonic-swss: e329dbd Survive pfc watchdog storm action across warm-reboot (#794) sonic-utilities: 6ee0aea (HEAD, origin/master, origin/HEAD) [config]: Change the order of interface commands (#504) 5ae30d2 [show vlan brief] Support 'alias' interface naming mode (#497) bafebf9 Update neighbor advertiser (#498) fa90083 [clear/main.py]: clear ndp command. (#450) 65f69e4 [show interface neighbor expected] Support 'alias' interface naming mode (#495) aae39e7 updated show ipv6 interface for alias mode (#493) 170fed9 [warm-reboot] initialize warm reboot state table before warm rebooting (#492) 06cd99f Allow config shutdown and startup operations on valid PortChannel interface names (#474) 98cdebb [show ip interface] Add support for 'alias' interface naming mode (#486) 5f1de81 [show] Add serial numbers/uptime/hwinfo to 'show version' output (#488) e78a866 [route_check] Move scripts under scripts/ directory, add to setup.py (#489) d347527 Change PFC watchdog CONFIG_DB table name from PFC_WD_TABLE to PFC_WD (#475) Signed-off-by: Wenda Ni <wenni@microsoft.com>

… define in swss common (sonic-net#428) Signed-off-by: Sihui Han <sihan@microsoft.com>

Including commits in sonic-swss-common repo: b423b9c Add support for hexists call (#432) [Tony Titus] 0982996 Remove extension of tableNameSeparatorMap (#430) [Qi Luo] d16cc76 [build]: add azure pipeline build badge (#429) [lguohan] f2aaf55 Set up CI with Azure Pipelines (#428) [lguohan]

…tomatically (#20068) #### Why I did it src/sonic-linux-kernel ``` * 2cfa620 - (HEAD -> 202311, origin/202311) tg3: fix broadcom NIC 57766 staying down issue (#421) (#428) (7 hours ago) [Vasundhara Volam] ``` #### How I did it #### How to verify it #### Description for the changelog

jleveque added Bug 🐛 Critical ❗ labels Mar 23, 2017

jleveque mentioned this issue Mar 23, 2017

[interfaces]: Restart networking service after the file is updated #430

Merged

lguohan closed this as completed Apr 6, 2017

stcheng reopened this Apr 14, 2017

lguohan self-assigned this Apr 14, 2017

stcheng removed the Critical ❗ label Apr 14, 2017

stcheng added the Critical ❗ label May 1, 2017

stcheng closed this as completed Jun 7, 2017

jleveque reopened this Jul 24, 2017

jleveque closed this as completed Nov 10, 2017

yxieca mentioned this issue Mar 7, 2019

[201811 sub module] advance sairedis sub module #2657

Merged

wendani mentioned this issue Apr 9, 2019

Submodule reference pointer change for PFC watchdog warm reboot commits #2755

Merged

madhanmellanox pushed a commit to madhanmellanox/sonic-buildimage that referenced this issue Mar 23, 2020

[pfcwd]: rename FLEX_COUNTER_POLL_MSECS to PFC_WD_POLL_MSECS and move…

88636f9

… define in swss common (sonic-net#428) Signed-off-by: Sihui Han <sihan@microsoft.com>

dmytroxshevchuk pushed a commit to dmytroxshevchuk/sonic-buildimage that referenced this issue Aug 31, 2020

Drop FDB notifications if they contain invalid OIDs (sonic-net#428)

18a5ebb

tonytitus mentioned this issue Dec 29, 2020

[sonic-swss-common] Update submodule #6317

Merged

guillaumelambert mentioned this issue Dec 15, 2022

[build] create a clean_docker_images profile #13067

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkins builds failing due to lack of free space #428

Jenkins builds failing due to lack of free space #428

jleveque commented Mar 23, 2017 •

edited

Loading

stcheng commented Mar 28, 2017

lguohan commented Apr 6, 2017

stcheng commented Apr 14, 2017

lguohan commented Apr 14, 2017

stcheng commented May 1, 2017 •

edited

Loading

jleveque commented May 1, 2017

stcheng commented May 2, 2017

jleveque commented Jul 24, 2017

stcheng commented Jul 26, 2017

lguohan commented Jul 26, 2017

jleveque commented Nov 10, 2017

Jenkins builds failing due to lack of free space #428

Jenkins builds failing due to lack of free space #428

Comments

jleveque commented Mar 23, 2017 • edited Loading

stcheng commented Mar 28, 2017

lguohan commented Apr 6, 2017

stcheng commented Apr 14, 2017

lguohan commented Apr 14, 2017

stcheng commented May 1, 2017 • edited Loading

jleveque commented May 1, 2017

stcheng commented May 2, 2017

jleveque commented Jul 24, 2017

stcheng commented Jul 26, 2017

lguohan commented Jul 26, 2017

jleveque commented Nov 10, 2017

jleveque commented Mar 23, 2017 •

edited

Loading

stcheng commented May 1, 2017 •

edited

Loading