Add support for custom backend transpiler stages

[mtreinish]

Summary

This commit adds initial support to the transpiler and backend
interfaces for backends to specify custom transpiler stages. There are
often specific hardware compilation requirements that a general purpose
transpiler's preset pass manager can't account for. While we strive to
provide interfaces to outline all the hardware constraints via the
Target class and general purposes passes to fit and optimize an input
circuit to the target backend there are some constraints that aren't
easily addressed in a general purpose way. For such cases having an
interface for a specific backend which has the necessary context of its
own constraints to provide custom compilation steps to integrate into
the pipeline is a necessary feature. The two initial examples of this
are custom basis gate translation and custom scheduling. This commit
adds two new hook point methods for BackendV2 objects,
get_translation_stage_method() and get_scheduling_stage_method(). These allow
for backends to specify custom default methods to use for these stages,
typically via the plugin interface. This should enable backends with
custom hardware specific requirements to influence the compilation
process so that any required custom steps to ensure the output circuit
is executable. In the future we may add additional hook points in a
similar manner to enable backends to assert more hardware-specific
compilation where the need arises.

Details and comments

Closes #8329

[coveralls]

Pull Request Test Coverage Report for Build 3178304457

• 11 of 12 (91.67%) changed or added relevant lines in 3 files are covered.
• 77 unchanged lines in 6 files lost coverage.
• Overall coverage decreased (-0.09%) to 84.611%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
qiskit/transpiler/preset_passmanagers/level3.py	1	2	50.0%

Files with Coverage Reduction	New Missed Lines	%
qiskit/extensions/quantum_initializer/squ.py	2	79.78%
qiskit/pulse/library/waveform.py	3	91.49%
src/dense_layout.rs	4	91.89%
src/sabre_swap/neighbor_table.rs	7	69.23%
src/sampled_exp_val.rs	14	63.93%
src/results/marginalization.rs	47	61.49%

Totals
Change from base Build 3173846729:	-0.09%
Covered Lines:	61595
Relevant Lines:	72798

---

💛 - Coveralls

[jakelishman]

I can't comment on whether this interface is sufficient for what custom backends need, but here's some minor code comments.

The removal of optimization_level from the PM configs looks very sensible - I'm not 100% sure how it ended up there anyway - since that's for selecting the PM builder rather than wanting to be passed to the builder.

[jakelishman]

Are we planning to add a different interface for passes that are suggested but not required? For example, ResetAfterMeasureSimplification is (presumably) never required, but for certain backends, it's a generally good idea.

[mtreinish]

I guess required is probably too strong a word. I was trying to reinforce here that these passes will be run by default for every transpile() call so only use it for things you want to be run all the time. My intent was for something like ResetAfterMeasureSimplification to be usable in post-translation which is why I put it in the example. For the appropriate backend it's an easy optimization that improves the fidelity, but that is a backend specific thing and only the backend object will have sufficient context to know whether it's a good idea or not. It's not something we can cleanly express via the target (which is the point of having this interface).

But, writing this I just realize that I think we should add an optimization level kwarg to the hook method. Because, I realize to really support using ResetAfterMeasureSimplification we want to not inject that into the pipeline for level 0. So giving backends the context of which optimization level the pass will enable doing this and potential tweak the custom stages in a similar way for higher optimization levels.

[kdk]

At a high level, I think #8329 would be better served by leveraging the plugin structure introduced in #8305 than by exposing specific hook points for backends to inject PassManagers.

This would make for a more natural integration with other tools that allow users to enable/disable specific plugins, and to know which plugins are active during a given call to transpile. It should hopefully also facilitate these passes being written in a reusable and composable way.

[mtreinish]

I pivoted the PR to have the hook point methods to return a method name string to be used as the default method when calling transpile on the backend in: cf6a7ec

[nkanazawa1989]

This is really a high-level comment, but this new interface makes Target more practical. With this every backend can provide hardware (low-level, not necessary architecture agnostic) constraints as a part of provider-specific target subclass, along with such pass plugin to digest them in transpile. I think this is good idea.

[nkanazawa1989]

This seems to me like restricting the flexibility of StagedPassManager where we can define transpiration with arbitrary combination of passes, i.e. the class can take stages argument. I have no idea how current configuration of passes is generic, but alternatively backend can provide full set of staged passes rather than providing a particular subset passes that Qiskit transpiler defines. For example, some ecosystem software may offer a service that overrides an IBM backend with fancy pass manager that makes circuit more robust to noise -- I also think we no longer say this is a plugin. If this means to be a plugin, perhaps current implementation is fine.

[mtreinish]

I think there are 2 pieces here, for the preset passmanagers/transpile() vs a more generic interface. I think for transpile() we still want to have a more structured set of default stages. This was actually a big part of defining the StagedPassManager for me was being able to formally define the stages for the default passmanagers to enable hooking into it like this. For this interface I was trying to enable backends to hook into the defaults stages with custom implementations (via plugins) that are specific for the needs of a backend. This is more about influencing the defaults to be better for the custom constraints of a particular backend. A user or a backend provider can still implement a fully custom StagedPassManager with additional or just different stages, but that is a lower level interface than what this PR is for. The idea for this is to basically enable users who run transpile(circuit, backend) where backend has custom constraints of some kind that the default preset passmanagers aren't able to handle. Having this interface enables the backend to easily keep transpile() working so those custom constraints can be handled cleanly without having to reimplement the entire pass manager and keep the rest of the user experience the same. For this PR I started with just translation and scheduling as I figured these stages were the most likely to be the ones which would have a need which was backend specific. The other thing is the default stages in the preset passmanager can (and probably will) be expanded in the future. So if we have a change in the global needs for the stages we use in the default pipeline we can add stages in the future (we've documented stability expectations around this in the class docs: https://qiskit.org/documentation/stubs/qiskit.transpiler.StagedPassManager.html?highlight=stagedpassmanager#qiskit.transpiler.StagedPassManager and basically said that additions are expected in the future).

The second part is what the plugin usage. If you look at earlier iterations of this PR, I had this interface be a bit more generic similar to what you're describing here. Instead of working with plugins backends would return PassManagers for each stage implementation (see cf6a7ec where this changed). The consensus on this interface in discussions about this PR was that because we have a plugin interface for providing alternative implementations of the default stages (which was added in #8305) was that for this backend interface to leverage that new plugin interface would be ideal. This has 2 effects, it simplifies the code since we have a single entrypoint for adding custom methods for stages and also for custom stage implementations for a backend it advertises these over the standard plugin interface which hopefully makes them globally discoverable outside of just the default use when targeting a backend.

[nkanazawa1989]

Okey, so we think current default stages of qiskit transpiler is sort of stable in future. My only concern was the stability of new API. To me current mechanism based off of #8305 seems reasonable.

[kdk]

Should these be renamed to get_translation_stage_plugin?

[mtreinish]

Good catch, I'll update this now

[mtreinish]

Actually I misread what you wrote here, I'm not sure we want to rename this. I had standardized on method because it's not actually restricted to plugins. It basically is the return of these methods are for changing the default for translation_method and scheduling_method values on transpile(). They don't necessarily have to be a plugin (well at least until we make everything a plugin per #8661). But I can change the name if you'd prefer using plugin

[kdk]

Agree, I think this is clearer post- #8661 . I was trying to avoid the confusion between method as in {layout,routing,scheduling}_method and method as in "python function".

I think either get_translation_method (matching the language from transpile(..., layout_method='foo') or get_translation_stage_plugin (matching the language from #8661 and #8305) would be okay.

[mtreinish]

I renamed it get_$FOO_stage_plugin in 214d9f7 (my shell messed up the commit message on it).

[kdk]

Is there a more descriptive name that we can use here? I'm not sure that "backend default methods" will be clear to a user without context. Maybe something like ignore_backend_defined_plugins or ignore_backend_supplied_plugins?

[mtreinish]

I'm open to changing the name, what about ignore_backend_supplied_default_methods? I'm hesitant about the two suggestions because the plugin could be defined or supplied from somewhere else. For example, I could see a backend using the synthesis translation method (assuming it actually worked reliably which I'm skeptical of) which is supplied and defined in terra itself. Or more realistically a plugin defined in a different package that's not terra or the provider.

[jlapeyre]

Also see my comment above trying to make the text a bit more clear.... I didn't realize I had not done "finish your review".
It's also not clear to me whether "method" means instance method or has the more generic, non-PL meaning. This is a recurrent problem in documentation.
EDIT: Re-reading the doc string, it's clear that this refers to an instance method.

[mtreinish]

In the context of transpile() and generate_preset_pass_manager() the "method" I'm referring to is the *method kwargs (init_method, layout_method, routing_method, translation_method, optimization_method, scheduling_method). This new option is to disable changing the default values that a BackendV2 object (from the backend kwarg) can optionally set for these kwargs. I'm open to changing the word method if people have suggestions, but I was using it try and be consistent with the rest of the transpile() api.

[jlapeyre]

Ok. I see that it's also clear in the doc that it means "instance method". It makes sense to me that whatever the name of the method in question here (eg gnore_backend_supplied_default_methods) , including the word "method" makes sense because it controls which methods will be called.
(I don't know why my comment appears twice in my view of this page. Probably due to the way I finished the review.)

[mtreinish]

I renamed it to ignore_backend_supplied_default_methods in d004080

[jlapeyre]

Also see my comment above trying to make the text a bit more clear.... I didn't realize I had not done "finish your review".
It's also not clear to me whether "method" means instance method or has the more generic, non-PL meaning. This is a recurrent problem in documentation.
EDIT: Re-reading the doc string, it's clear that this refers to an instance method.