Full CMS pytorch training in May 2024

[jpata]

major:

• training on the full PU ttbar, QCD, Ztautau (500k total for each sample)
• pytorch transformer with 6 layers, head_dim 16, num_heads 32, relu, bf16, FlashAttentionV2
• training was done with pytorch 2.3
• using 1xA100 80GB
• dashboard in https://www.comet.com/jpata/particleflow-pt/reports/cms-training-may-2024
• weights in https://huggingface.co/jpata/particleflow/tree/main/cms/2024_05_16_attn_model21M
• slides with results in https://indico.cern.ch/event/1415765/#2-cms-status-and-plans

minor:

• moved scripts/fcc -> scripts/clic (more accurate, making space for fccee_cld)