[Profiler] eDSL support for source info table + eDSL-level flame graphs

[ayakayorihiro]

Continuing fun source-info work from PR #2396 , this PR contains an application of the metadata to the profiler. We have:

• eDSL: Updates to builder.py and py_ast.py to produce @pos attributes and the sourceinfo metadata block
• tools/fileinfo_emitter: a tool that takes a Calyx with a source-info metadata block and emits ADL-to-Calyx mapping information on components, cells, and groups (which is what the profiler gathers information about right now)
• Updates to the profiling script to generate ADL-level flame graphs
• fud2 support for the Python eDSL → profiling results pipeline
  • Arguments to the input Python eDSL program can be provided using the profiler.edsl-args option.

The result is that a user can now obtain flame graphs that contain eDSL-level profiling information! Now <fud2-directory>/profiler-out contains two new flame graph files if an eDSL program was provided to fud2:

• mixed-scaled-flame.svg: A version of scaled-flame.svg that contains the Calyx construct (component/cell/group) name along with the associated eDSL file name and line number
• adl-scaled-flame.svg: A version of scaled-flame.svg that uses eDSL variable names for Calyx constructs wherever possible. If we couldn't find an appropriate variable name, the Calyx construct name will be given in single quotes.

Usage

To obtain eDSL-level flame graphs,

0. Set the below environment variable:

export CALYX_PY_SOURCELOC=true

NOTE: To disable source-info metadata, set the environment variable to "" (the empty string).

1. Run fud2 with the arg --through calyx-py-profiler, and write the fud2 output to a directory. If the eDSL program requires arguments, use the profiler.edsl-args option.
   ex)

fud2 frontends/queues/tests/strict/strict_6flow_test.py -o strict_6flow_test.svg -s sim.data=frontends/queues/tests/strict/strict_6flow_test.data --dir fud2-runs/strict_6flow_test/ -s profiler.edsl-args="20000 --keepgoing"

The reason why we need to use --through is that fud2 would choose to simply compile the eDSL source file to Calyx, and run the profiler on that instead, which wouldn't yield any eDSL-level profiling information.

2. The eDSL-level flame graphs should be available in the fud2 output directory that you provided under profiler-out. So in the example above, there should be fud2-runs/strict_6flow_test/adl-scaled-flame.svg and fud2-runs/strict_6flow_test/mixed-scaled-flame.svg.

Issues

Solved! Thanks @EclecticGriffin for the pro-tip of using environment variables. Step 0 above has been updated!

I'd like to be able to make the eDSL source-info metadata configurable, without having to manually change tests as much as possible. (This is partially because I didn't want to mess with all of the tests that use calyx-py for this PR). I've added a new function in builder.py that toggles a global variable in py_ast.py which controls the metadata emitting, but ideally I wouldn't need the user to change their source file (as in step 0 above). Is there a nice elegant way I can go around this? Sorry for the weird question.

[ekiwi]

Looks good. I have a couple of questions.

Please also add a test where you generate a source info block and pos annotations.