[PY312] Update various base level tools e.g zlib, xz, cmake and more #9410

smuzaffar · 2024-09-17T15:39:49Z

This PR updates many base level packages

autotools:
- autoconf: 2.72
- automake: 1.16.5 ( 1.17 does not work see Failure to build static library on Github MacOS libffi/libffi#853)
- libtool: 2.4.7
- m4: 1.4.19 (dropped m4-centos8 patch: already included in new version)
- gettext: 0.22.5
bz2lib: 1.08
cmake: 3.30.3
curl: 8.10.1
expat: 2.6.3
gdbm: 1.24
gmake: 4.4.1
libffi: 3.4.6
libjpeg-turbo: 3.0.4
libpng: 1.6.44
libtiff: 4.6.0
libuuid: 2.40
nasm: 2.16.0
sqlite: 3.46.1
zlib: 1.3.1 and use consistent version for all archs
zstd: 1.5.6
xz: 5.6.2
Used -fno-fast-math to build xz (see re-evaluate the optimisation flags for liblzma.so cmssw#41773). Looks like new xz version 5.6.2 is relatively faster than 5.2.5. ROOT bench mark test re-evaluate the optimisation flags for liblzma.so cmssw#41773 (comment) timings are below (avg of 5 runs)
- xz 5.2.5 with -Ofast: 8.251s
- xz 5.6.2 with -Ofast -fno-fast-math: 8.1s
- Generated root file size is exactly same for both.\

Also added missing BuildRequires on gmake/autotools where needed

smuzaffar · 2024-09-17T15:40:00Z

please test

cmsbuild · 2024-09-17T15:40:10Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_2_X/master.

@aandvalenzuela, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-09-17T15:40:11Z

cms-bot internal usage

cmsbuild · 2024-09-18T10:05:25Z

Pull request #9410 was updated.

smuzaffar · 2024-09-18T10:06:39Z

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

smuzaffar · 2024-09-18T20:55:09Z

please test for CMSSW_14_2_PY312_X/el8_amd64_gcc12

cmsbuild · 2024-09-19T12:06:55Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41578/summary.html
COMMIT: 255ed5b
CMSSW: CMSSW_14_2_TF_X_2024-09-17-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9410/41578/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41578/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41578/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test TestFWCoreSkeletons had ERRORS

Comparison Summary

Summary:

You potentially added 138 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 28282 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3331158
DQMHistoTests: Total failures: 62047
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3269091
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -1.304 KiB( 43 files compared)
DQMHistoSizes: changed ( 250202.181 ): 0.182 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.127 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): -0.120 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 8.0 ): -1.239 KiB SiStrip/MechanicalView
Checked 193 log files, 163 edm output root files, 44 DQM output files
TriggerResults: found differences in 6 / 42 workflows

cmsbuild · 2024-09-19T16:45:47Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41611/summary.html
COMMIT: 255ed5b
CMSSW: CMSSW_14_2_PY312_X_2024-09-18-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9410/41611/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/py3-tensorflow/2.16.1-7f498ea53b2a28aeb5b5e7d621693f56/lib/python3.12/site-packages/tensorflow/lite/python/authoring/authoring.py", line 43, in 
from tensorflow.lite.python import convert
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/py3-tensorflow/2.16.1-7f498ea53b2a28aeb5b5e7d621693f56/lib/python3.12/site-packages/tensorflow/lite/python/convert.py", line 25, in 
import shutils
ModuleNotFoundError: No module named 'shutils'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.AfyjSq (%build)


RPM build errors:
line 42: It's not recommended to have unversioned Obsoletes: Obsoletes: external+tfaot-model-test-simple+1.0.1-4f700e2ecb2c4fecbd024b559b88636f
Macro expanded in comment on line 393: %{aot_config}, pointing to the aot config file of the model to compile (required)

smuzaffar · 2024-09-19T21:02:14Z

please test for CMSSW_14_2_PY312_X/el8_amd64_gcc12

smuzaffar · 2024-09-20T10:33:02Z

looks good to go in for PY312_X IBs

cmsbuild · 2024-09-20T11:43:17Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41652/summary.html
COMMIT: 255ed5b
CMSSW: CMSSW_14_2_PY312_X_2024-09-18-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9410/41652/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41652/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fcd03/41652/git-merge-result

Unit Tests

I found 5 errors in the following unit tests:

---> test TestDQMOnlineClient-hcal_dqm_sourceclient had ERRORS
---> test TestFWCoreParameterSetDriver had ERRORS
---> test TestFWCoreSkeletons had ERRORS
and more ...

Comparison Summary

Summary:

You potentially added 95 lines to the logs
Reco comparison results: 4549 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3331158
DQMHistoTests: Total failures: 22598
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3308540
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 193 log files, 163 edm output root files, 44 DQM output files
TriggerResults: no differences found

[xz] version 5.6.2 and build with -fnofast-math

2ec353e

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Sep 17, 2024

smuzaffar mentioned this pull request Sep 17, 2024

[TF][xz] version 5.6.2 and build with -fnofast-math #9404

Closed

Update various base level tools e.g zlib, xz, cmake

255ed5b

cmsbuild removed the tests-started label Sep 18, 2024

cmsbuild added the tests-pending label Sep 18, 2024

smuzaffar changed the title ~~[xz] version 5.6.2 and build with -fnofast-math~~ Update various base level tools e.g zlib, xz, cmake and more Sep 18, 2024

smuzaffar changed the base branch from IB/CMSSW_14_2_X/master to IB/CMSSW_14_2_X/tf September 18, 2024 10:06

cmsbuild added tests-started and removed tests-pending labels Sep 18, 2024

smuzaffar changed the base branch from IB/CMSSW_14_2_X/tf to IB/CMSSW_14_2_X/py312 September 18, 2024 15:37

smuzaffar changed the title ~~Update various base level tools e.g zlib, xz, cmake and more~~ [PY312] Update various base level tools e.g zlib, xz, cmake and more Sep 18, 2024

cmsbuild added tests-rejected and removed tests-started labels Sep 19, 2024

cmsbuild added tests-started and removed tests-rejected labels Sep 19, 2024

smuzaffar merged commit 13831e5 into IB/CMSSW_14_2_X/py312 Sep 20, 2024
12 of 15 checks passed

cmsbuild added tests-rejected and removed tests-started labels Sep 20, 2024

smuzaffar deleted the update-xz-562-no-fast-math branch October 8, 2024 08:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PY312] Update various base level tools e.g zlib, xz, cmake and more #9410

[PY312] Update various base level tools e.g zlib, xz, cmake and more #9410

smuzaffar commented Sep 17, 2024 •

edited

Loading

smuzaffar commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

cmsbuild commented Sep 17, 2024 •

edited

Loading

cmsbuild commented Sep 18, 2024

smuzaffar commented Sep 18, 2024

smuzaffar commented Sep 18, 2024

cmsbuild commented Sep 19, 2024

cmsbuild commented Sep 19, 2024

smuzaffar commented Sep 19, 2024

smuzaffar commented Sep 20, 2024

cmsbuild commented Sep 20, 2024

[PY312] Update various base level tools e.g zlib, xz, cmake and more #9410

[PY312] Update various base level tools e.g zlib, xz, cmake and more #9410

Conversation

smuzaffar commented Sep 17, 2024 • edited Loading

smuzaffar commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

cmsbuild commented Sep 17, 2024 • edited Loading

cmsbuild commented Sep 18, 2024

smuzaffar commented Sep 18, 2024

smuzaffar commented Sep 18, 2024

cmsbuild commented Sep 19, 2024

Unit Tests

Comparison Summary

cmsbuild commented Sep 19, 2024

External Build

smuzaffar commented Sep 19, 2024

smuzaffar commented Sep 20, 2024

cmsbuild commented Sep 20, 2024

Unit Tests

Comparison Summary

smuzaffar commented Sep 17, 2024 •

edited

Loading

cmsbuild commented Sep 17, 2024 •

edited

Loading