Simplified experience with the transpiler using estimator.

[jaygambetta]

What should we add?

This is an issues beyond the bug I reported in #10818. As we start to work with the transpiler and use the estimator more the current flow is rather complicated. While most users can just send abstract quantum circuits to the system I want to be able to send logical circuits to the system using the gate set and coupling map I have. Given I know what I want to do I can in general make a better quantum circuit than the default optimizations and/or I can work on a better definition of the defaults.

In my view we should remove layout from being bound to the transpile circuit and have a layout object and meta data returned to the user after they run the transpiler. I think the workflow should be

qc_transpile, layout, metadata = transpile(qc_example, options=)

personally I find the layout object over complex and should just be a list (I think @mtreinish shares my views here) and in the case that I need metadata from this transpliation I think we can put it all in a new object. I don't have strong opinions on it but some things that are interesting is to know how the layout was changed. Today we kinda get this as it is all put into a tranpilelayout object but it is not clear what pass did what (as I tried to explain in the bug #10818). My suggestion is that the meta data should contain

1. a ordered list of passes that are used on the circuit (this I would find helpful as I can then think about which ones I could improve)
2. It would be useful to know which passes changed the layout so someway of correlating that to the pass would be good. As @mtreinish points out all passes don't change the layout so having one for each pass is a waste
3. if approximations where used rather then exact some way of knowing which passes introduced errors would be good.
4. Timing of each pass and total time used by the transpiler.

These are the ones I can think of for now

Finally we need a new function in the compiler and when I am using the estimator I want to be able to run using the layout from the transpiler.

opts_transpile = permute_sparse_pauli_ops(opts, layout)

that way I can send both qc_transpile and opts_transpile to the estimator, while it is simple to write a few lines of code that is not the nicest user experience.

This is very much related to Qiskit/qiskit-ibm-runtime#338 and solution #9988 but it is fundamental for the workflows using the estimator not only the ability to add ancillas if the transpiler enlarges the system but if the layout changes the layout object needs to be used to redefine the observables and even used in the post-processing if I want the data to be changed back to my original circuit order. We need to make layout a variable that people want to work with.

[mtreinish]

I'm a bit conflicted on the interface here, I agree with the goals of simplifying the interface around working with layout. But, I'm not a fan of splitting out the returns like that, just from a backwards compatibility standpoint it's a non-starter imo. It'd be too drastic a change to have multiple returns from transpile() like that as it's probably the most commonly used function in qiskit and it would require everyone to update their code (which has likely worked for 4+ years at this point). I personally don't see an issue if we embed the necessary transpilation metadata in the output circuit (right now that's only layout and some timing information if scheduling ran).

The thing I was planning after #10818 is that the easy short term win here is to add some helper methods to TranspileLayout get the representations that people actually want. For example, I think the view everyone really wants for the layout is something like this:

 def get_index_layout(transpiled_circuit: QuantumCircuit, num_source_qubits: int):
    """Get a list of final positions from a transpiled circuit and the input circuit's width
    
    transpiled_circuit: The output circuit from the transpiler.
    num_source_qubits: The number of qubits that were in the original circuit
    """
    
    layout = transpiled_circuit.layout
    if layout is None:
        return list(range(num_source_qubits))

    pos_to_virt = {v: k for k, v in layout.input_qubit_mapping.items()}
    qubit_indices = []
    for index in range(num_source_qubits):
        qubit_idx = layout.initial_layout[pos_to_virt[index]]
        if layout.final_layout is not None:
            qubit_idx = layout.final_layout[transpiled_circuit.qubits[qubit_idx]]
        qubit_indices.append(qubit_idx)
    return qubit_indices

which for the estimator gives you the positional mapping for each of the input circuit's qubits as a list you can use to permute the operator like what #9988 is trying to do. We could add similar easier to use alternatives for all the pieces about layout we want to expose so it's much easier to work with, but leave the current attributes in place to satisfy existing users of the API (which there are several out there because the current interface is all that is available for this). Then we can migrate to the newer better interface over time.

The piece I'm not clear on is why the per pass layout transformation is important in the output. Having a log of how the transpiler updates the layout I don't think would be really helpful in most cases because we transform the layout at different points sometimes multiple times. The best example of the is VF2PostLayout (i.e. mapomatic) which after we've already selected an initial qubit mapping and run routing to insert swaps we potentially find an alternative better qubit mapping and update everything to use that instead. We only run this once now but we could conceivably run it > 1 time too. Similarly another pass I'm working on (very slowly) is #92523 to push through permutations and swaps in the circuit, this has a similar layout effect as routing and looking at layout per pass would probably be more confusing than helpful (aside from doing step by step inspection/debugging).

[mtreinish]

As for the lower level details in the numbered list a lot of that already exists (the only thing which I don't remember off the top of my head is the approximation and error reporting), just in a different format or over different mechanisms. For example, we log a lot of the details already, so if you do something like:

import logging
logging.basicConfig(level="INFO")

you get the execution order and all the timing info as the passes run. For deeper inspection you can leverage the callback argument to get a handle after each pass executes and do whatever you need to.

This lets you pretty much debug every step as it goes by and provides an interface for interacting with this data. For example: https://github.com/TheGupta2012/qiskit-timeline-debugger/tree/main is a tool that gives you a jupyter widget or a CLI tool to see the transforms each pass does as it executes and explore all the transforms and their effect on the circuit.

The reason we do this via other mechanisms is that adding the details by default increases the complexity for the non-power user and also some of the introspection has noticeable runtime performance and memory overhead. So we're optimizing for the common path in the current interface design; which does add a couple of extra steps if you know what you're doing and want more detail. But, I think that's a reasonable tradeoff here, as the vast majority of our users aren't going to want to know the inner details of the transpiler.

[jaygambetta]

personally I don't use the other layouts but I can see how I might if I was optimizing for the swapping. The most important is a simple way to get the final layout

[jaygambetta]

as for working for 4 years it is because the default when including measurements is to transpile the measurements as well so backend.run works fine as an executor but as we move to estimator this becomes a huge problem. Unless we fix this estimator is essentially useless unless and here what we need is a pair circuits, observables that meet a common layout.