[Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585

jshlee · 2021-07-22T11:07:19Z

PR description:

DAQ dataformat is updated to the current version used in P5
GEM unpacked is changed to only produce digis when there are not DAQ errors
new DAQ status classes that checks for DAQ errors and stores it for OnlineDQM
run3 GE1/1 cannot read multi-BX, so by default non zero BX will not be unpacked for run3
added multi-BX mode for phase2
added plugin that checks if output the packing and unpacking is identical to the original simDigis
Discussed in the minutes of https://indico.cern.ch/event/1056074/#32-dpg
Link to presentation at GEM operations (https://indico.cern.ch/event/1058368/contributions/4448079/subcontributions/345162/attachments/2283335/3881798/gem_unpacker_20210719.pdf)

PR validation:

Validated with wf 11634.0 and 34611.0

Backport

Backport of #34504

updated dataformats to latest version currently running

added GEMPackingTester

fix bug due to smaller vfat position check-headers fixed

cmsbuild · 2021-07-22T11:07:45Z

A new Pull Request was created by @jshlee (Jason Lee) for CMSSW_11_3_X.

It involves the following packages:

CondFormats/GEMObjects (db)
DataFormats/GEMDigi (upgrade, simulation)
EventFilter/GEMRawToDigi (reconstruction)

@perrotta, @civanch, @mdhildreth, @cmsbuild, @jpata, @srimanob, @kpedro88, @slava77, @ggovi can you please review it and eventually sign? Thanks.
@rovere, @Martin-Grunewald, @watson-ij, @dildick, @mmusich, @seemasharmafnal this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

Backported from GEM unpacker - updated DAQ dataformat and new DAQ status collections #34504

qliphy · 2021-07-22T12:05:10Z

backport of #34504

qliphy · 2021-07-22T12:05:42Z

please test

cmsbuild · 2021-07-22T14:47:18Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-121749/17084/summary.html
COMMIT: 1adc2cb
CMSSW: CMSSW_11_3_X_2021-07-22-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34585/17084/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 51 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2878314
DQMHistoTests: Total failures: 46
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2878245
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-07-30T19:11:52Z

Pull request #34585 was updated. @perrotta, @civanch, @mdhildreth, @cmsbuild, @jpata, @srimanob, @kpedro88, @slava77, @ggovi can you please check and sign again.

slava77 · 2021-07-31T01:06:36Z

@cmsbuild please test

cmsbuild · 2021-07-31T03:44:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-121749/17379/summary.html
COMMIT: 147e32f
CMSSW: CMSSW_11_3_X_2021-07-29-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34585/17379/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 7 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2878314
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2878279
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

davidlange6 · 2021-07-31T04:47:55Z

We need a PR to master also to confirm there is no problem created by this change there.

…

On Jul 31, 2021, at 5:44 AM, cmsbuild ***@***.***> wrote: +1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-121749/17379/summary.html COMMIT: 147e32f CMSSW: CMSSW_11_3_X_2021-07-29-2300/slc7_amd64_gcc900 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34585/17379/install.sh to create a dev area with all the needed externals and cmssw changes. Comparison Summary Summary: • No significant changes to the logs found • Reco comparison results: 7 differences found in the comparisons • DQMHistoTests: Total files compared: 38 • DQMHistoTests: Total histograms compared: 2878314 • DQMHistoTests: Total failures: 12 • DQMHistoTests: Total nulls: 1 • DQMHistoTests: Total successes: 2878279 • DQMHistoTests: Total skipped: 22 • DQMHistoTests: Total Missing objects: 0 • DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared) • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings • Checked 160 log files, 37 edm output root files, 38 DQM output files • TriggerResults: no differences found — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

slava77 · 2021-08-03T15:31:32Z

it looks like the updates after the last reco signature were only in DataFormats/GEMDigi/src/classes_def.xml, which is not in the reco category.

jpata · 2021-08-03T15:53:29Z

+reconstruction

signature follows [Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585 (comment)
last changes are outside of reco

boudoul · 2021-08-04T07:52:06Z

Hi @perrotta, @civanch, @mdhildreth, @srimanob, @kpedro88, @ggovi : could you please proceed with this PR ? We need it for CRUZET ... thanks

srimanob · 2021-08-04T07:57:57Z

+Upgrade

The last fix on class version is merged on master 16 hours ago. If it runs fine then I think this PR can be merged.

perrotta · 2021-08-04T14:52:46Z

urgent

boudoul · 2021-08-04T14:52:57Z

hi @civanch, @mdhildreth : could you please proceed ?

@perrotta : you may want to consider bypassing @ggovi , I believe he is on holidays

civanch · 2021-08-04T19:25:30Z

+1

perrotta · 2021-08-04T20:12:51Z

+1
@cms-sw/db-l2 you signed the master version of this PR, I assume you are also fine with this backport

perrotta · 2021-08-04T20:21:33Z

merge

jshlee added 13 commits July 21, 2021 19:42

GEM new DAQ status data collections

15d8e8a

updated dataformats to latest version currently running

added status collections

8a2b9fe

added GEMPackingTester

removed phase2_GEM.toModify

7993d41

code-checks

49afa4b

added multiBX read for unpacking

05e4e62

packing testing

dfb2dee

code-checks

3f06794

uncomment msg

3a5d91b

code-checks

4d5c58d

updated GEMPackingTester

5a18c64

fix bug due to smaller vfat position check-headers fixed

support for vfat ver 2

6f5c5f5

applied @jpata comments

e954467

removed amc13 vs amc crc checks for now

1adc2cb

cmsbuild added this to the CMSSW_11_3_X milestone Jul 22, 2021

cmsbuild added db-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending upgrade-pending labels Jul 22, 2021

cmsbuild added the backport-ok label Jul 22, 2021

cmsbuild added tests-started urgent and removed tests-pending labels Jul 22, 2021

cmsbuild removed the tests-started label Jul 22, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 31, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 31, 2021

This was referenced Jul 31, 2021

GEM dataformat - fix classversion #34713

Merged

[Backport] GEM dataformat - fix classversion #34743

Merged

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Aug 3, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels Aug 4, 2021

cmsbuild added the urgent label Aug 4, 2021

cmsbuild added simulation-approved and removed simulation-pending labels Aug 4, 2021

cmsbuild added orp-approved and removed orp-pending labels Aug 4, 2021

cmsbuild merged commit 23cf19b into cms-sw:CMSSW_11_3_X Aug 4, 2021

quark2 mentioned this pull request Aug 4, 2021

Updates on GEM onlineDQM due to the updates of DAQ dataformat, backport to CMSSW_11_3_X #34782

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585

[Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585

jshlee commented Jul 22, 2021 •

edited by qliphy

Loading

cmsbuild commented Jul 22, 2021 •

edited

Loading

qliphy commented Jul 22, 2021

qliphy commented Jul 22, 2021

cmsbuild commented Jul 22, 2021

cmsbuild commented Jul 30, 2021

slava77 commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

davidlange6 commented Jul 31, 2021 via email

slava77 commented Aug 3, 2021

jpata commented Aug 3, 2021

boudoul commented Aug 4, 2021

srimanob commented Aug 4, 2021

perrotta commented Aug 4, 2021

boudoul commented Aug 4, 2021

civanch commented Aug 4, 2021

perrotta commented Aug 4, 2021

perrotta commented Aug 4, 2021

[Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585

[Backport] - GEM unpacker - updated DAQ dataformat and new DAQ status collections #34585

Conversation

jshlee commented Jul 22, 2021 • edited by qliphy Loading

PR description:

PR validation:

Backport

cmsbuild commented Jul 22, 2021 • edited Loading

qliphy commented Jul 22, 2021

qliphy commented Jul 22, 2021

cmsbuild commented Jul 22, 2021

Comparison Summary

cmsbuild commented Jul 30, 2021

slava77 commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

Comparison Summary

davidlange6 commented Jul 31, 2021 via email

slava77 commented Aug 3, 2021

jpata commented Aug 3, 2021

boudoul commented Aug 4, 2021

srimanob commented Aug 4, 2021

perrotta commented Aug 4, 2021

boudoul commented Aug 4, 2021

civanch commented Aug 4, 2021

perrotta commented Aug 4, 2021

perrotta commented Aug 4, 2021

jshlee commented Jul 22, 2021 •

edited by qliphy

Loading

cmsbuild commented Jul 22, 2021 •

edited

Loading