MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag #332

jpata · 2024-06-17T10:49:16Z

Regenerated CMS and CLIC datasets as v2.0.0:

CMS: new CMSSW generation with CMSSW_14->postprocessing->tfds
- add CMS ttbar no-pileup sample as a cross-check
- improve and cross-check ground truth
CLIC: new Key4HEP generation->postprocessing->tfds, updated with the features below, crucially a corrected genStatus definition (introduced a while ago but never regenerated)
CLD: first generation, consistent setup with CLIC
Delphes: deprecated and removed

New features and fixes in v2.0.0 in postprocessing:

for CMS postprocessing2.py, use only CaloParticles (add track links from TrackingParticles)
track pythia-level genjets, genmet in datasets as the ultimate jet/MET reconstruction target
- for CMS: generate v3_1 with updated PFAnalysisNtuplizer, added pythia genjets and genmet: jpata/cmssw@eac6192
- for Key4HEP: compute genjets, genmet on postprocessing using visible stable particle from .hepmc input events
propagate per-particle ispu flag
- for CMS postprocessing2.py, implemented and cross-checked
- for Key4HEP: currently placeholder

Misc other things:

CMSSW validation also generates JME nano

still TODO:

revert T2_EE specific conf from cmssw script

…to ttbar_nopu

…ticles

…_nopu

jpata · 2024-06-28T12:17:36Z

We find perfect alignment between genMetTrue and CaloParticleMET in single-particle guns, PF ntuple ROOT level:

and at the MLPF truth level:

… add per-particle ispu flag (jpata#332) * generate ttbar nopu events * up * update postprocessing * small sample generation * v3_1 run * updates for CMSSE 14 generation * [skip ci] cleanup postprocessing * [skip ci] update pu gen * update postprocessing with new truth definition based only on caloparticles * remove pdb, switch genjet to energy * [skip ci] prepare for v3_3 * [skip ci] fix flag * added time and mem limits * pu files from scratch * 20240702_cptruthdef submission * ttbar nopu v2 * up * added genjet, genmet to clic postprocessing * remove delphes * update tests * add postprocessing jobs * update torch * update dataset version * propagate genjets, genmet * shared memory error * training on v2.0.0 for cms * fix occasional root file load bug * add jmenano * fix qq * clic training * up

* chore: update raytune search space, utils and startscript * fix: raytune deprecated env var for storage_path Also add num samples to draw in HPO as cmd line arg * chore: update clic config file for jureap57 * feat: script to build python env from scratch * chore: update startscripts for raytrain and raytune * fix CMS model path for ACAT2022 * MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag (#332) * generate ttbar nopu events * up * update postprocessing * small sample generation * v3_1 run * updates for CMSSE 14 generation * [skip ci] cleanup postprocessing * [skip ci] update pu gen * update postprocessing with new truth definition based only on caloparticles * remove pdb, switch genjet to energy * [skip ci] prepare for v3_3 * [skip ci] fix flag * added time and mem limits * pu files from scratch * 20240702_cptruthdef submission * ttbar nopu v2 * up * added genjet, genmet to clic postprocessing * remove delphes * update tests * add postprocessing jobs * update torch * update dataset version * propagate genjets, genmet * shared memory error * training on v2.0.0 for cms * fix occasional root file load bug * add jmenano * fix qq * clic training * up * CMS training instructions (#336) * CMS training instructions * Update pyg-clic.yaml * Update pyg-clic.yaml * fix: black formatting * Enable CI/CD test of HPO workflow * fix: typo in test script --------- Co-authored-by: Joosep Pata <joosep.pata@gmail.com>

… add per-particle ispu flag (jpata#332) * generate ttbar nopu events * up * update postprocessing * small sample generation * v3_1 run * updates for CMSSE 14 generation * [skip ci] cleanup postprocessing * [skip ci] update pu gen * update postprocessing with new truth definition based only on caloparticles * remove pdb, switch genjet to energy * [skip ci] prepare for v3_3 * [skip ci] fix flag * added time and mem limits * pu files from scratch * 20240702_cptruthdef submission * ttbar nopu v2 * up * added genjet, genmet to clic postprocessing * remove delphes * update tests * add postprocessing jobs * update torch * update dataset version * propagate genjets, genmet * shared memory error * training on v2.0.0 for cms * fix occasional root file load bug * add jmenano * fix qq * clic training * up

* chore: update raytune search space, utils and startscript * fix: raytune deprecated env var for storage_path Also add num samples to draw in HPO as cmd line arg * chore: update clic config file for jureap57 * feat: script to build python env from scratch * chore: update startscripts for raytrain and raytune * fix CMS model path for ACAT2022 * MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag (jpata#332) * generate ttbar nopu events * up * update postprocessing * small sample generation * v3_1 run * updates for CMSSE 14 generation * [skip ci] cleanup postprocessing * [skip ci] update pu gen * update postprocessing with new truth definition based only on caloparticles * remove pdb, switch genjet to energy * [skip ci] prepare for v3_3 * [skip ci] fix flag * added time and mem limits * pu files from scratch * 20240702_cptruthdef submission * ttbar nopu v2 * up * added genjet, genmet to clic postprocessing * remove delphes * update tests * add postprocessing jobs * update torch * update dataset version * propagate genjets, genmet * shared memory error * training on v2.0.0 for cms * fix occasional root file load bug * add jmenano * fix qq * clic training * up * CMS training instructions (jpata#336) * CMS training instructions * Update pyg-clic.yaml * Update pyg-clic.yaml * fix: black formatting * Enable CI/CD test of HPO workflow * fix: typo in test script --------- Co-authored-by: Joosep Pata <joosep.pata@gmail.com>

jpata and others added 5 commits June 15, 2024 09:59

generate ttbar nopu events

f96d567

up

281ccbd

update postprocessing

b28e893

small sample generation

6580995

v3_1 run

3bcc1bc

jpata changed the title ~~generate ttbar nopu events~~ track pythia-level genjets, genmet in datasets; add per-particle ispu flag; generate CMS ttbar+noPU Jun 21, 2024

jpata changed the title ~~track pythia-level genjets, genmet in datasets; add per-particle ispu flag; generate CMS ttbar+noPU~~ MLPF datasets v2.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag; generate CMS ttbar+noPU [WIP] Jun 21, 2024

jpata marked this pull request as draft June 21, 2024 09:04

jpata and others added 9 commits June 25, 2024 16:49

updates for CMSSE 14 generation

fc7b65f

[skip ci] cleanup postprocessing

0912959

Merge branch 'ttbar_nopu' of https://github.com/jpata/particleflow in…

046cfac

…to ttbar_nopu

[skip ci] update pu gen

bfb69df

update postprocessing with new truth definition based only on calopar…

51912d6

…ticles

remove pdb, switch genjet to energy

39cd09d

Merge branch 'ttbar_nopu' of github.com:jpata/particleflow into ttbar…

ec8954d

…_nopu

[skip ci] prepare for v3_3

f913bc8

[skip ci] fix flag

02422e1

jpata added 12 commits June 29, 2024 00:45

added time and mem limits

12d1612

pu files from scratch

cce532f

20240702_cptruthdef submission

dfbed49

ttbar nopu v2

2a8d5b6

up

f7f0134

added genjet, genmet to clic postprocessing

6d977ee

remove delphes

b3f2b10

update tests

889e22c

add postprocessing jobs

c4317b0

update torch

28296ef

update dataset version

10fba64

propagate genjets, genmet

0158400

jpata and others added 6 commits July 5, 2024 16:14

shared memory error

6b4ebd0

training on v2.0.0 for cms

1812cae

Merge branch 'main' into ttbar_nopu

4f16122

fix occasional root file load bug

6ca6ae8

add jmenano

d52e58b

fix

092124d

jpata changed the title ~~MLPF datasets v2.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag; generate CMS ttbar+noPU [WIP]~~ MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag Jul 15, 2024

fix qq

c3e7f15

jpata marked this pull request as ready for review July 16, 2024 10:37

jpata added 2 commits July 16, 2024 16:04

clic training

0674e46

up

2b28af4

jpata merged commit a0f4428 into main Jul 17, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag #332

MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag #332

jpata commented Jun 17, 2024 •

edited

Loading

jpata commented Jun 28, 2024 •

edited

Loading

MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag #332

MLPF datasets v2.0.0: track pythia-level genjets, genmet in datasets; add per-particle ispu flag #332

Conversation

jpata commented Jun 17, 2024 • edited Loading

jpata commented Jun 28, 2024 • edited Loading

jpata commented Jun 17, 2024 •

edited

Loading

jpata commented Jun 28, 2024 •

edited

Loading