external lib for HYDJET++ #7483

wouf · 2021-12-01T14:59:08Z

Hydjet++ (v2.4.3) integration as external.
RP cmssw: cms-sw/cmssw#36316

cmsbuild · 2021-12-01T14:59:32Z

A new Pull Request was created by @wouf for branch IB/CMSSW_12_2_X/master.

@cmsbuild, @smuzaffar, @iarspider, @ddaina can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2021-12-01T15:57:09Z

Pull request #7483 was updated.

smuzaffar · 2021-12-02T07:44:39Z

assign generators

cmsbuild · 2021-12-02T07:45:02Z

New categories assigned: generators

@mkirsano,@alberto-sanchez,@SiewYan,@GurpreetSinghChahal,@Saptaparna,@agrohsje you have been requested to review this Pull request/Issue and eventually sign? Thanks

smuzaffar · 2021-12-02T07:48:46Z

hydjet2.spec

@@ -0,0 +1,20 @@
+### RPM external hydjet2 2.x.y 


please provide the exact version

Can we put here the link to the actual version instead of hardcoded number?

what do you mean ? can you please give an example

Something like

RPM external hydjet2 version

where the variable version would be determined from the link to the json file on the author's site, or just from the some text file, placed on the author's website.
Mayby ### RPM external hydjet2 http://..../version
Would it work?

no, it will not work. One needs to explicitly provide the version to be used here and then you can use the version to auto generate the source url e.g.

### RPM externals hydjet2 2.4.3 Source: http://cern.ch/lokhtin/hydjet++/%{n}-%{realversion}.tar.gz

So in future when you update the version to e.g. 2.4.4 then package system will try to download http://cern.ch/lokhtin/hydjet++/hydjet2-2.4.4.tar.gz

But only after PR, right? I'm looking a way to avoid of necessity to make PR for every minor update of HYDJET++

We do not want our build system to automatically pick up new updates which can break the whole release. imagine that we have tested one version in IBs and when we want to build a release then author updates the version which can break the whole system.

When there is a new version then it should be explicitly requested, tested via PR before we integrate it.

Ok, I see, thanks!

smuzaffar · 2021-12-02T07:50:29Z

hydjet2.spec

@@ -0,0 +1,20 @@
+### RPM external hydjet2 2.x.y 
+
+Source: http://cern.ch/lokhtin/hydjet++/%{n}.tar.gz


the URL http://cern.ch/lokhtin/hydjet++/hydjet2.tar.gz is very generic and our packaging system is not going to redownload it when you need to update the version. Can you please provide separate tar files for each version?

At the author's site is just one version for the new HYDJET++ - the actual one. This naming policy implies to move all outdated version to other links/tarballs. The author is the member of CMS HI, since very beginning and he was updating PYQUEN and HYDJETs focusing primarily on the interests of the CMS. As far as I understand, during compiling of each CMSSW version, the external archive downloaded from the source website, so if author will update the version, it will be available for the next CMSSW version. It's ok for CMS HI group, we always need the more actual version on top.

Or You mean difficulties related to the backport for the new version?

For each version of a package there should be a unique url. If author just keeps on updating http://cern.ch/lokhtin/hydjet++/hydjet2.tar.gz (pointing to latest version) then there is no guarantee that two cmssw builds will have the same contents. I have no issues with http://cern.ch/lokhtin/hydjet++/hydjet2.tar.gz pointing to latest version but there should link for specific versions too e.g. http://cern.ch/lokhtin/hydjet++/hydjet2-%{version}.tar.gz and this should be used in our spec files.

Note that cmsBuild ( our packaging system) also caches the contents of each source URL. So cmsBuild will always use the previously cached contents if url itself is not changed.

I see, thanks. I will ask for this changes.

davidlange6 · 2021-12-02T11:38:36Z

In addition perhaps GitHub would be a more appropriate place for storing and tracking software releases to be downloaded? Personal areas tend not to be stable. On Dec 2, 2021 10:23 AM, Malik Shahzad Muzaffar ***@***.***> wrote: @smuzaffar commented on this pull request.

________________________________ In hydjet2.spec<#7483 (comment)>:

@@ -0,0 +1,20 @@

+### RPM external hydjet2 2.x.y + +Source: http://cern.ch/lokhtin/hydjet++/%{n}.tar.gz Note that cmsBuild ( our packaging system) also caches the contents of each source URL. So cmsBuild will always use the previously cached contents if url itself is not changed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#7483 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQZNLIRNVLERSWDYJWLUO43HXANCNFSM5JEWD5NA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

wouf · 2021-12-02T15:02:48Z

I am in contact with the main author of HYDJET++. He prefer to have all items related to the event generator on his personal web-page, but he "doesn't mind if any necessary items would be duplicated via GitHub as well". But I think that stability of personal area does not matter, because of cmsBuild "also caches the contents of each source URL. So cmsBuild will always use the previously cached contents if url itself is not changed".

cmsbuild · 2021-12-02T15:18:42Z

Pull request #7483 was updated.

smuzaffar · 2021-12-05T12:46:16Z

@wouf , looks like hydjet2 also makes a copy of libpyquen.so in hydjet2/lib dirctory [a]. Although we can delete it in our sopec file but I think proper fix should be to remove install(FILES ${PYQUEN_LIBRARIES} DESTINATION lib) line from hydjet2/CMakeLists.txt

>ls -ld hydjet2/2.4.3-f3c1bda44ebbd799b8162526a0512b93/lib/*
-rwxr-xr-x. 1 cvmfs cvmfs  220896 Dec  5 09:22 hydjet2/2.4.3-f3c1bda44ebbd799b8162526a0512b93/lib/libhydjet2.so
-rw-r--r--. 1 cvmfs cvmfs 1076656 Nov 12 05:31 hydjet2/2.4.3-f3c1bda44ebbd799b8162526a0512b93/lib/libpyquen.so

wouf · 2021-12-05T15:27:42Z

install is needed for standalone usage. anyway ot makes a copy only if PYQUEN library absent in the system. I guess i have to adjust FindPYQUEN module for cmssw case. Where pyquen library should be looking for at compilation time?

smuzaffar · 2021-12-06T06:25:44Z

cmake properly found cmssw pyquen

-- Found PYQUEN: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/pyquen/1.5.4-edb141c01d142e7411bce63143948569/lib/libpyquen.so

see the full build log https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cae5ac/20993/external/hydjet2/2.4.3-f3c1bda44ebbd799b8162526a0512b93/log . but because of install(FILES ${PYQUEN_LIBRARIES} DESTINATION lib) line it also forces it to install along with hydjet2 libs.

wouf · 2021-12-06T17:27:58Z

Thank you for this important catch! The problem is more global than it may seem. If someone checks libhydjet2.so , he would have found that its RUNPATH is setted:

`$ objdump -x /afs/cern.ch/work/w/wouf/TMP/2git/lib/slc7_amd64_gcc900/external/hydjet2/2.4.3-5c4bd156a11e3c022693e40677fa8e04/lib/libhydjet2.so | egrep 'R(|UN)PATH'

RPATH /afs/cern.ch/work/w/wouf/TMP/2git/lib/tmp/BUILDROOT/5c4bd156a11e3c022693e40677fa8e04/opt/cmssw/slc7_amd64_gcc900/external/hydjet2/2.4.3-5c4bd156a11e3c022693e40677fa8e04/lib:/afs/cern.ch/work/w/wouf/TMP/2git/lib/slc7_amd64_gcc900/external/pyquen/1.5.4-edb141c01d142e7411bce63143948569/lib:/afs/cern.ch/work/w/wouf/TMP/2git/lib/slc7_amd64_gcc900/external/lhapdf/6.4.0-d2f68e74e7b7722f8a01055c82dc1e31/lib
`
It's not really correct, especially for CMSSW. I will care about it.

wouf · 2021-12-08T11:14:20Z

It's now fixed, please try again.

smuzaffar · 2021-12-10T12:45:20Z

please test

cmsbuild · 2021-12-10T18:36:33Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21169/summary.html
COMMIT: fcac651
CMSSW: CMSSW_12_3_X_2021-12-09-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7483/21169/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 3250704
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3250676
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-01-17T15:10:18Z

please test
let me re-run the tests for all os/archs

smuzaffar · 2022-01-17T15:10:40Z

please test for slc7_aarch64_gcc11

smuzaffar · 2022-01-17T15:11:04Z

please test for slc7_ppc64le_gcc11

smuzaffar · 2022-01-17T15:11:48Z

please test for cs8_amd64_gcc11

cmsbuild · 2022-01-17T19:09:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21755/summary.html
COMMIT: 4175cf6
CMSSW: CMSSW_12_3_X_2022-01-16-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7483/21755/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21755/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21755/git-merge-result

cmsbuild · 2022-01-17T19:51:55Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21753/summary.html
COMMIT: 4175cf6
CMSSW: CMSSW_12_3_X_2022-01-16-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7483/21753/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21753/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21753/git-merge-result

RelVals

4.53

----- Begin Fatal Exception 17-Jan-2022 19:39:09 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 17-Jan-2022 19:50:02 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 17-Jan-2022 20:08:46 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2022-01-17T20:43:10Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21760/summary.html
COMMIT: 4175cf6
CMSSW: CMSSW_12_3_X_2022-01-16-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7483/21760/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21760/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21760/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3464734
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3464704
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 42 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-01-18T00:13:07Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21761/summary.html
COMMIT: 4175cf6
CMSSW: CMSSW_12_3_X_2022-01-16-2300/cs8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7483/21761/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21761/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0ab80e/21761/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 65791 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3464734
DQMHistoTests: Total failures: 614026
DQMHistoTests: Total nulls: 312
DQMHistoTests: Total successes: 2850374
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.342 KiB( 42 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.288 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11834.0 ): 2.191 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.111 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.129 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 7.3 ): -2.009 KiB SiStrip/MechanicalView
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: found differences in 14 / 42 workflows

smuzaffar · 2022-01-21T15:17:44Z

+externals
looks good to go in. cms-sw/cmssw#36316 requires this

SiewYan · 2022-01-24T05:51:52Z

thank you @smuzaffar , are those two errors concerning? I can't view those error in the Details.

smuzaffar · 2022-01-24T08:21:56Z

@SiewYan , no errors for slc7_aarch64_gcc11 are already there in the IBs. So those can be ignored for this PR test

SiewYan · 2022-01-24T11:56:07Z

@smuzaffar , thanks, I will sign this PR.

SiewYan · 2022-01-24T11:56:23Z

+generators

cmsbuild · 2022-01-24T11:56:27Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2022-02-08T10:12:10Z

merging it as it is needed by cms-sw/cmssw#36316

perrotta · 2022-02-08T10:19:36Z

merging it as it is needed by cms-sw/cmssw#36316

Thank you @smuzaffar : I was forgetting it, indeed...

external lib for HYDJET++ cms-sw/cmssw# #36316

6faaf32

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Dec 1, 2021

Update doc.txt

6c26a4d

wouf mentioned this pull request Dec 1, 2021

Hydjet++ 2.4.3 integration cms-sw/cmssw#36316

Merged

cmsbuild added the generators-pending label Dec 2, 2021

smuzaffar reviewed Dec 2, 2021

View reviewed changes

Update hydjet2.spec

fcac651

smuzaffar changed the base branch from IB/CMSSW_12_2_X/master to IB/CMSSW_12_3_X/master December 6, 2021 21:29

cmsbuild added tests-started and removed tests-pending labels Dec 10, 2021

cmsbuild added tests-approved and removed tests-started labels Dec 10, 2021

cmsbuild added tests-started and removed tests-approved labels Jan 17, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 17, 2022

cmsbuild added externals-approved and removed externals-pending labels Jan 21, 2022

cmsbuild added fully-signed generators-approved and removed generators-pending pending-signatures labels Jan 24, 2022

smuzaffar merged commit bbd5012 into cms-sw:IB/CMSSW_12_3_X/master Feb 8, 2022

wouf deleted the IB/CMSSW_12_2_X/master branch February 8, 2022 10:26

This was referenced Feb 8, 2022

Implementation of beam halo tagger algorithm for the endcap photons cms-sw/cmssw#36901

Merged

[geant4_vecgeom] Update Geant4 to 11.0, G4EMLOW to 8.0, G4PARTICLEXS to 4.0 #7611

Merged

		@@ -0,0 +1,20 @@
		### RPM external hydjet2 2.x.y

		Source: http://cern.ch/lokhtin/hydjet++/%{n}.tar.gz

external lib for HYDJET++ #7483

external lib for HYDJET++ #7483

Conversation

wouf commented Dec 1, 2021 • edited Loading

cmsbuild commented Dec 1, 2021

cmsbuild commented Dec 1, 2021

smuzaffar commented Dec 2, 2021

cmsbuild commented Dec 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RPM external hydjet2 version

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smuzaffar Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidlange6 commented Dec 2, 2021 via email

wouf commented Dec 2, 2021

cmsbuild commented Dec 2, 2021

smuzaffar commented Dec 5, 2021

wouf commented Dec 5, 2021

smuzaffar commented Dec 6, 2021

wouf commented Dec 6, 2021 • edited Loading

wouf commented Dec 8, 2021

smuzaffar commented Dec 10, 2021

cmsbuild commented Dec 10, 2021

Comparison Summary

smuzaffar commented Jan 17, 2022

smuzaffar commented Jan 17, 2022

smuzaffar commented Jan 17, 2022

smuzaffar commented Jan 17, 2022

cmsbuild commented Jan 17, 2022

cmsbuild commented Jan 17, 2022

RelVals

cmsbuild commented Jan 17, 2022

Comparison Summary

cmsbuild commented Jan 18, 2022

Comparison Summary

smuzaffar commented Jan 21, 2022

SiewYan commented Jan 24, 2022

smuzaffar commented Jan 24, 2022

SiewYan commented Jan 24, 2022

SiewYan commented Jan 24, 2022

cmsbuild commented Jan 24, 2022

smuzaffar commented Feb 8, 2022

perrotta commented Feb 8, 2022

wouf commented Dec 1, 2021 •

edited

Loading

smuzaffar Dec 2, 2021 •

edited

Loading

wouf commented Dec 6, 2021 •

edited

Loading