Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release with newest LHAPDF for W mass sample production #37528

Open
kdlong opened this issue Apr 11, 2022 · 28 comments
Open

Release with newest LHAPDF for W mass sample production #37528

kdlong opened this issue Apr 11, 2022 · 28 comments

Comments

@kdlong
Copy link
Contributor

kdlong commented Apr 11, 2022

Dear experts,

We are investigated a workflow to manage the production of very large samples (~2 B events) for the W mass analysis. We understand that computing has some resources available to test running the GEN step on HPC resources. The resources are available now and are just waiting on us to have a workflow ready. From our side, we have gridpacks produced and validated, but we would very much like to take advantage of the most modern PDF sets that have just been released. This would require a new CMSSW release. The campaign (UL production) runs in CMSSW_10_6_20, but we're able to use CMSSW_12_X for the gridpack production, provided we work around the constraints of scram (https://cms-talk.web.cern.ch/t/problem-setting-up-cmssw-12-2-after-cmssw-10-6/9117/4)

In this sense, we are not super tied to the CMSSW version. To have the newest sets, we just need a release built with this PR merged (cms-sw/cmsdist#7776) and that is guaranteed to be accessible at all sites and available long term for future production. Since the resources from computing are time sensitive, we would also want whatever be completed the most quickly. Does this make sense for a new 12_3 release, or a 12_4 pre-release? In particular, do pre releases satisfy the availability requirement?

Thanks,

Kenneth

Tagging people already involved in the discussion: @bendavid @mseidel42 @tyjyang @qliphy @jordan-martins

@cmsbuild
Copy link
Contributor

A new Issue was created by @kdlong Kenneth Long.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

assign generators, core

@cmsbuild
Copy link
Contributor

New categories assigned: core,generators

@mkirsano,@Dr15Jones,@smuzaffar,@alberto-sanchez,@SiewYan,@makortel,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks

@qliphy
Copy link
Contributor

qliphy commented Apr 11, 2022

In this sense, we are not super tied to the CMSSW version. To have the newest sets, we just need a release built with this PR merged (cms-sw/cmsdist#7776) and that is guaranteed to be accessible at all sites and available long term for future production. Since the resources from computing are time sensitive, we would also want whatever be completed the most quickly. Does this make sense for a new 12_3 release, or a 12_4 pre-release? In particular, do pre releases satisfy the availability requirement?

If urgently needed, ORP (@perrotta and me) can build 12_4_0_pre3 (originally scheduled on April/19) ahead this week.
It should be distributed to and accessible via cvmfs. @smuzaffar and others may comment more on your requirement, i.e. "...accessible at all sites and available long term for future production"

@smuzaffar
Copy link
Contributor

I think all cms sites get cmssw releases from CVMFS. So as soon as we deploy it under /cvmfs/cms.cern.ch then it should be available on all sites. As we never have deleted any thing from /cvmfs/cms.cern.ch, so it should remain available for long time.

@davidlange6
Copy link
Contributor

davidlange6 commented Apr 11, 2022 via email

@smuzaffar
Copy link
Contributor

right @davidlange6 , we should not use pre-releases for production.

@bendavid
Copy link
Contributor

Well in this case it's only the environment + externals which would be used, since the gridpack+LHE production doesn't actually link or call any CMSSW code...

@smuzaffar
Copy link
Contributor

I would still suggest to backport cms-sw/cmsdist#7776 to 12.3.X and build a new release. I know , in past production system was using https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML to get the valid releases and pre-releases are not shown there by default.

@bendavid
Copy link
Contributor

The use of the alternative environment inside the gridpack is transparent to the production system (for better or for worse).

@davidlange6
Copy link
Contributor

davidlange6 commented Apr 11, 2022 via email

@kdlong
Copy link
Contributor Author

kdlong commented Apr 11, 2022

From our side we're equally happy with CMSSW_12_3_X. Is the timescale to have a new release longer, though?

@qliphy
Copy link
Contributor

qliphy commented Apr 12, 2022

From our side we're equally happy with CMSSW_12_3_X. Is the timescale to have a new release longer, though?

@kdlong CMSSW_12_3_0 was built last weekend. Now if cms-sw/cmsdist#7776 gets merged and backported to 12_3_X, a new release CMSSW_12_3_1 can be built this week (with only difference from 12_3_0 on this LHAPDF change) if there is no objection.

@kdlong
Copy link
Contributor Author

kdlong commented Apr 12, 2022

@qliphy sorry for the delay, it seems that the backport is made and accepted, from our standpoint having the CMSSW_12_3_1 release with it would be great.

@qliphy
Copy link
Contributor

qliphy commented Apr 13, 2022

@kdlong Right, it is almost ready (also needs to wait for the IB test results). Will try to launch the build soon today or tomorrow.

@kdlong
Copy link
Contributor Author

kdlong commented Apr 13, 2022

Thanks Qiang!

@qliphy
Copy link
Contributor

qliphy commented Apr 14, 2022

@kdlong 12_3_1 is now ready.

@kdlong
Copy link
Contributor Author

kdlong commented Jul 4, 2022

Hi all, is it possible to have this back ported to CMSSW_10_2? We would like to have a production for the low pileup which would use slc6.

@mkirsano
Copy link
Contributor

mkirsano commented Jul 7, 2022

Hello, I will look into this

@qliphy
Copy link
Contributor

qliphy commented Jul 8, 2022

@kdlong
Copy link
Contributor Author

kdlong commented Jul 8, 2022

Excellent, thanks a lot @qliphy and @mkirsano!

@kdlong
Copy link
Contributor Author

kdlong commented Jul 12, 2022

Sorry, last request I promise, but if it's not too much trouble, having this back ported to 10_6 would be very useful for the Nano production. Is it possible? Thanks!

@kdlong
Copy link
Contributor Author

kdlong commented Feb 21, 2023

hi all, just realised this never materialised. Is there any plans to have a new 10_6 release? Having the PDF sets updated in the CMSSW version used for the Nano would be very helpful.

@perrotta
Copy link
Contributor

hi all, just realised this never materialised. Is there any plans to have a new 10_6 release? Having the PDF sets updated in the CMSSW version used for the Nano would be very helpful.

@kdlong we plan a 10_6 release kind of tomorrow: too early for another PR not yet submitted-
As usual, I'd suggest you to prepare the backport, if you need it, and try to arrange with @cms-sw/generators-l2 to decide with them about a possible merging in a forthcoming release

@menglu21
Copy link
Contributor

@kdlong is it urgent for ongoing production, but anyway it seems too tight for the new release which will happen today, please prepare the backport and we will work on it before next release plan

@kdlong
Copy link
Contributor Author

kdlong commented Feb 22, 2023

Hi @menglu21 it's not urgent since this is mostly for making our private production much less error prone. But I think it would be good to have, since my understanding is it isn't a lot of effort. It requires a PR to cmsdist. In the past the gen integration experts (Mikhail) took care of this

@lviliani
Copy link
Contributor

lviliani commented Sep 6, 2024

Can we close this one?

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 6, 2024

cms-bot internal usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests