Add size tracking to transpilation benchmarks

[jakelishman]

Summary

This is in addition to the depth trackers, which are generally a bit
more important.

Details and comments

These are the benchmarks that go along with Qiskit/qiskit#7542. That said, there doesn't actually appear to be any improvement in the benchmarks between that commit (Qiskit/qiskit@37de1254) and its parent (Qiskit/qiskit@6e412ab):

Benchmarks that have stayed the same:

       before           after         ratio
     [6e412abc]       [37de1245]
     <main~4>         <main~3>  
              441              441     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(0, 'sabre')
              618              618     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(0, None)
              269              269     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(1, 'sabre')
              101              101     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(1, None)
              237              237     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(2, 'sabre')
               87               87     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(2, None)
              307              307     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(3, 'sabre')
              176              176     1.00  queko.QUEKOTranspilerBench.track_size_bigd_optimal_depth_45(3, None)
            10124            10124     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(0, 'sabre')
            13226            13226     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(0, None)
             4104             4104     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(1, 'sabre')
              438              438     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(1, None)
             4010             4010     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(2, 'sabre')
              370              370     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(2, None)
             3323             3323     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(3, 'sabre')
              441              441     1.00  queko.QUEKOTranspilerBench.track_size_bntf_optimal_depth_25(3, None)
             6782             6782     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(0, 'sabre')
            12890            12890     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(0, None)
             3313             3313     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(1, 'sabre')
             1276             1276     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(1, None)
             2957             2957     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(2, 'sabre')
              868              868     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(2, None)
             3104             3104     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(3, 'sabre')
              970              970     1.00  queko.QUEKOTranspilerBench.track_size_bss_optimal_depth_100(3, None)
              306              306     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(10)
              440              440     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(12)
             2976             2976     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(14)
               50               50     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(4)
              112              112     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(6)
              198              198     1.00  random_circuit_hex.BenchRandomCircuitHex.track_size_ibmq_backend_transpile(8)
              914              914     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(10, 0)
             1021             1021     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(10, 1)
             1007             1007     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(10, 2)
              686              686     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(10, 3)
             2365             2365     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(20, 0)
             2418             2418     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(20, 1)
             2397             2397     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(20, 2)
             1470             1470     1.00  ripple_adder.RippleAdderTranspile.track_size_transpile_square_grid_ripple_adder(20, 3)
            16105            16105     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_quantum_volume_transpile_50_x_20(0)
            15891            15891     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_quantum_volume_transpile_50_x_20(1)
            15891            15891     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_quantum_volume_transpile_50_x_20(2)
            10757            10757     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_quantum_volume_transpile_50_x_20(3)
             3505             3505     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm(0)
             2305             2305     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm(1)
             2305             2305     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm(2)
               11               11     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm(3)
             7505             7505     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm_backend_with_prop(0)
             2306             2306     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm_backend_with_prop(1)
             2307             2307     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm_backend_with_prop(2)
               11               11     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_from_large_qasm_backend_with_prop(3)
             2793             2793     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_qv_14_x_14(0)
             2793             2793     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_qv_14_x_14(1)
             2793             2793     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_qv_14_x_14(2)
             1742             1742     1.00  transpiler_levels.TranspilerLevelBenchmarks.track_size_transpile_qv_14_x_14(3)
              373              373     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'sabre', 'dense')
              370              370     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'sabre', 'noise_adaptive')
              313              313     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'sabre', 'sabre')
              331              331     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'stochastic', 'dense')
              355              355     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'stochastic', 'noise_adaptive')
              355              355     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(0, 'stochastic', 'sabre')
              280              280     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'sabre', 'dense')
              274              274     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'sabre', 'noise_adaptive')
              264              264     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'sabre', 'sabre')
              295              295     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'stochastic', 'dense')
              318              318     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'stochastic', 'noise_adaptive')
              306              306     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(1, 'stochastic', 'sabre')
              274              274     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'sabre', 'dense')
              268              268     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'sabre', 'noise_adaptive')
              258              258     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'sabre', 'sabre')
              291              291     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'stochastic', 'dense')
              298              298     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'stochastic', 'noise_adaptive')
              294              294     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(2, 'stochastic', 'sabre')
              432              432     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'sabre', 'dense')
              423              423     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'sabre', 'noise_adaptive')
              398              398     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'sabre', 'sabre')
              491              491     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'stochastic', 'dense')
              516              516     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'stochastic', 'noise_adaptive')
              464              464     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4gt10_v1_81(3, 'stochastic', 'sabre')
               94               94     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'sabre', 'dense')
               76               76     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'sabre', 'noise_adaptive')
               73               73     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'sabre', 'sabre')
              103              103     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'stochastic', 'dense')
               73               73     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'stochastic', 'noise_adaptive')
               73               73     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(0, 'stochastic', 'sabre')
               75               75     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'sabre', 'dense')
               69               69     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'sabre', 'noise_adaptive')
               60               60     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'sabre', 'sabre')
               90               90     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'stochastic', 'dense')
               62               62     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'stochastic', 'noise_adaptive')
               61               61     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(1, 'stochastic', 'sabre')
               75               75     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'sabre', 'dense')
               69               69     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'sabre', 'noise_adaptive')
               60               60     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'sabre', 'sabre')
               82               82     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'stochastic', 'dense')
               54               54     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'stochastic', 'noise_adaptive')
               57               57     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(2, 'stochastic', 'sabre')
              104              104     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'sabre', 'dense')
              105              105     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'sabre', 'noise_adaptive')
               99               99     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'sabre', 'sabre')
              149              149     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'stochastic', 'dense')
              100              100     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'stochastic', 'noise_adaptive')
               95               95     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_4mod5_v0_19(3, 'stochastic', 'sabre')
              852              852     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'sabre', 'dense')
              816              816     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'sabre', 'noise_adaptive')
              837              837     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'sabre', 'sabre')
              828              828     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'stochastic', 'dense')
              870              870     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'stochastic', 'noise_adaptive')
              840              840     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(0, 'stochastic', 'sabre')
              668              668     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'sabre', 'dense')
              665              665     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'sabre', 'noise_adaptive')
              644              644     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'sabre', 'sabre')
              737              737     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'stochastic', 'dense')
              766              766     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'stochastic', 'noise_adaptive')
              754              754     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(1, 'stochastic', 'sabre')
              658              658     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'sabre', 'dense')
              647              647     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'sabre', 'noise_adaptive')
              624              624     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'sabre', 'sabre')
              731              731     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'stochastic', 'dense')
              722              722     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'stochastic', 'noise_adaptive')
              734              734     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(2, 'stochastic', 'sabre')
              996              996     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'sabre', 'dense')
             1073             1073     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'sabre', 'noise_adaptive')
              982              982     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'sabre', 'sabre')
             1243             1243     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'stochastic', 'dense')
             1288             1288     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'stochastic', 'noise_adaptive')
             1247             1247     1.00  transpiler_qualitative.TranspilerQualitativeBench.track_size_transpile_mod8_10_178(3, 'stochastic', 'sabre')

[mtreinish]

Do you have any ideas on the runtime associated with this? We're basically duplicating our already duplicated transpile depth tracking runs (which are already duplicated with the time benchmarks). I'm a bit worried about the runtime cost vs the value on doing this, since I think the case Qiskit/qiskit#7542 was trying to fix is pretty rare. If the runtime here is large maybe we should look at transitioning some of the depth to size or add two new benchmarks to track the size and depth of the circuit from Qiskit/qiskit#7386

[jakelishman]

Yeah, I wasn't feeling particularly good about duplicating a bunch of run-time cost here. An ideal solution would be for asv to allow tracking more than one parameter from a single run (possibly even let it be folded into the timing benchmarks!), but as far as I know, that's not possible. On my laptop, these new benchmarks take ~10 minutes.

It's probably best to not add these at all - Qiskit/qiskit#7542 didn't show any progressions on this particular set, which strongly suggests we're not going to get any useful data from them. I suppose the particular ones we've got are tailored towards testing optimisation of depth, not size (QV and Queko in particular).

[jakelishman]

We decided not to go this way for now to avoid exploding the size of the benchmark suite. The most complete path forwards if we want to do things like this is to work with asv to introduce some concept of a "track" benchmark that tracks multiple values (depth and size, in this case) from the same code.