Combining plots to or creating new plots

[coolalexzb]

This goal needs more discussion. In run.py, each time the user runs the program, the results will be stored into output file with filename randomly generated. It is difficult for users to combine plots with the random filename. I can add function to rename the filenames but it is highly possible for the users to run the program with the same dataset over one time. In this case, the file will be overwritten. My current thought is the following:

Add option in the command line. For example, when users input command line
python run.py -b [numpy] -o [cpu] -p 1Lbb-likelihoods-hepdata -m [750,100]
Users will only get the result run by numpy + cpu

while with the following command lines:

python run.py -b [numpy, jax] -o [cpu, gpu] -p 1Lbb-likelihoods-hepdata -m [750,100]
In the output file, run.py will run result for cpu+numpy, cpu+jax, gpu+numpy and gpu+jax and all the results will be plotted in the same graph.

I looked through the document and noticed that the graph example shown on this page is different from what we have now. The x-axis is the number of total bins. To plot a graph like the page shows, I think I need more datasets.

Look forward to more discussion to solve this problem.

[matthewfeickert]

python run.py -b [numpy, jax] -o [cpu, gpu] -p 1Lbb-likelihoods-hepdata -m [750,100]

This API seems fine, but if a user wanted to run with just a single backend (say JAX) could they do this

python run.py -b jax -o [cpu, gpu] -p 1Lbb-likelihoods-hepdata -m [750,100]

or would they need to do this (always use brackets [ ])

python run.py -b [jax] -o [cpu, gpu] -p 1Lbb-likelihoods-hepdata -m [750,100]

?

Note also that once we have things packaged up then this will be a real CLI and we won't need to prefix with python. :)

[coolalexzb]

@matthewfeickert Since pyhf doesn't contain cpu and gpu option when making computation (see #PR899), the python script in pyhf-benchmark also doesn't have cpu and gpu option. How to set cpu/gpu option in our python script and how can the python script know what machine it makes use of?

[matthewfeickert]

@matthewfeickert How to set cpu/gpu option in our python script and how can the python script know what machine it makes use of?

This isn't really a problem at the moment as this is a minor thing. The important thing is that there is a working package, and adding the ability to switch modes later isn't a big deal. At the moment, all the installs can be assumed to be GPU (as you'd need to specify a cpu version of TensorFlow) that will run in CPU mode if not appropriate GPU device is found. So just move forward with the assumption that things are running in the GPU mode and we can worry about the CPU mode later.