DAQ: support for FRD file header with file locking

[smorovic]

PR description:

• raw (FRD) file format for DAQ3 has recently been modified to include metadata in the header, so json files can be removed (due to 2 MB block constraint in huge-page ramdisk). Initially support was added with file broker mode. Now it is added to the file-locking mode which is used in testing and in Hilton.
• Support for the file header is also added to the single-buffer mode.
• Error handling in file locking mode is improved. It also allows unit test to catch problem in case JSON (raw) files can not be found. Since throwing exception is an issue in separate file discovery thread used by input source, error condition is propagated to the main thread instead of throwing directly.

PR includes also two bug fixes:

• unique_ptr release() calls do not execute destructor of allocated object wrapped with unique_ptr. This created a (small) memory leak and also caused processed files to not be deleted in single-threaded input source mode (which is available, but not used by default in production setup).

• in the check of non-opened fd value comparision was with 0, it should be compared with -1.

PR validation:

Verified to work correctly in DAQ test system in single-buffered/multi-buffered running modes, with file broker/locking, and with/without file header (and combinations).

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057

• This PR adds an extra 48KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File EventFilter/Utilities/interface/EvFDaqDirector.h modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933
  • File EventFilter/Utilities/src/EvFDaqDirector.cc modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks performance-for-range-copy #29927
  • File EventFilter/Utilities/src/FedRawDataInputSource.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-make-unique #29934, [RFC] Test clang-tidy --checks modernize-use-auto #29936, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-emplace #29937

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057/code-format.patch
  e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16060

• This PR adds an extra 48KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File EventFilter/Utilities/interface/EvFDaqDirector.h modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933
  • File EventFilter/Utilities/src/EvFDaqDirector.cc modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks performance-for-range-copy #29927
  • File EventFilter/Utilities/src/FedRawDataInputSource.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-make-unique #29934, [RFC] Test clang-tidy --checks modernize-use-auto #29936, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-emplace #29937

[cmsbuild]

A new Pull Request was created by @smorovic (Srecko Morovic) for master.

It involves the following packages:

EventFilter/Utilities

@perrotta, @mommsen, @cmsbuild, @emeschi, @slava77 can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6990/console Started: 2020/06/12 15:58

[cmsbuild]

+1
Tested at: 8c55286
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-92bfae/6990/summary.html
CMSSW: CMSSW_11_2_X_2020-06-12-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[mommsen]

+1

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-92bfae/6990/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 36
• DQMHistoTests: Total histograms compared: 2778811
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2778760
• DQMHistoTests: Total skipped: 50
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
• Checked 152 log files, 16 edm output root files, 36 DQM output files

[perrotta]

+1

• Updates relevant for DAQ
• Jenkins tests pass

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1