increasing num_generation_jobs fails

[irinaespejo]

Context

The workflow fails when increasing the top-level parameter num_generation_jobs. The workflow first fails at the delphes
stage. Usually delphes_0 is successful and the error appears after delphes_1. Once the error appears all the delphes stages after that one also fail.

Process

1. Change "num_generation_jobs" in reana/reana.yaml for a higher number than 6.
   

2. run the workflow make reana-run

Expected result

changing the num_generation_jobs should not make the workflow fail. It should branch the generation stage accordingly and each branch should succesfully pass pythia and delphes stages.

Current result

See screenshot.

The Logs tab gives more information about delphes_1. The error is a KeyError in madminer/code/delphes.py
'''
Traceback (most recent call last):
File "/madminer/code/delphes.py", line 96, in
reader.analyse_delphes_samples()
File "/usr/local/lib/python3.8/dist-packages/madminer/delphes/delphes_reader.py", line 603, in analyse_delphes_samples
this_observations, this_weights, this_n_events = self._analyse_delphes_sample(
File "/usr/local/lib/python3.8/dist-packages/madminer/delphes/delphes_reader.py", line 772, in _analyse_delphes_sample
_, this_weights = parse_lhe_file(
File "/usr/local/lib/python3.8/dist-packages/madminer/utils/interfaces/lhe.py", line 262, in parse_lhe_file
output_weights[benchmark_name] = weights_all_events[benchmark_name]
KeyError: 'sm'

'''

The rest of the delphes_2,3,4,5,6 give similar logs but with different keys causing the error:

• delphes_2 : KeyError: 'w'
• delphes_3 : KeyError: 'morphing_basis_vector_2'
• delphes_4 : KeyError: 'morphing_basis_vector_3'
• delphes_5 : KeyError: 'morphing_basis_vector_4'
• delphes_6 : KeyError: 'morphing_basis_vector_5'

Possible Fix

Go to debug madminer/code/delphes.py there must be a problem with the circular generation method: that is a modulo % 6 which has to be modified.

[irinaespejo]

created new branch increase_jobs_#43 in madminer-workflow-ph/

[irinaespejo]

UPDATE

The workflow fails if num_generation_jobs > 6 and num_generation_jobs % 6 != 0. Basically, if we want to increase the number of jobs, we have to set num_generation_jobs to be a multiple of 6. My best guess is that this is closely related to the fact that the morphing basis is 6-dimensional. Here is a summary of what I tried with screenshots:

• n = 6 ✔️
• n = 7 ❌
• n = 8 ❌
• n = 10 ❌
• n = 11 ❌
• n = 12 ✔️
• n = 18 ✔️
• n = 59 ❌
• n = 60 ✔️

Screenshots

[Sinclert]

Hi @irinaespejo ,

My understanding of the morphing basis is very limited, so it may be the case that, as you said, it is a 6 dimensional basis that can only scale on a factor of 6. That is something a Physics-background person should clarify.

On the other hand, the piece of code you are referring to (on the generate.py module), comes from this one line Python code that was present in the madminer-workflow repository before I took over the project. I respected it because I understood it was there for a reason, but, if it is not the case, feel free to tune it as desire.

https://github.com/scailfin/madminer-workflow/blob/bbc0ff6a94628eb3ab9feadfc4dff0d8454f2ab5/docker-images/docker-madminer-physics/code/generate.py#L66

[Sinclert]

Update:

The scaling via increasing the number of jobs is being considered and tracked in this madminer-workflow-ph issue (as it only affects the Physics part of the workflow).

Please refer to that issue for future discussion.