Feature/base qcvv framework

[cdbf1]

In preparation for the XEB/IRB benchmarking routines, this PR implements some common functionality that can form the basis of a wide QCVV within superstaq.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[cdbf1]

Moved qcvv folder to supermarq following discussion at dev meeting

[richrines1]

(adding a few comments/questions)

[richrines1]

looks great, just a couple more questions

not necessarily in this pr, but we'll probably also want to add a third path for experiments on local hardware: in this case we would still use superstaq to compile (but not submit) all the circuits, and then lets the user run the compiled circuits locally and pass in their counts back in to the experiment for analysis

[cdbf1]

not necessarily in this pr, but we'll probably also want to add a third path for experiments on local hardware: in this case we would still use superstaq to compile (but not submit) all the circuits, and then lets the user run the compiled circuits locally and pass in their counts back in to the experiment for analysis

Definitely a good shout. A hacky way to do this at the moment would be for the user to use the _build_circuits() method to build the samples and then process them separately. Adding the option to compile them would be cool too, and I guess ideally a slick handle for the users to pass in connection to their own hardware would be the dream - definitely a separate PR though!

[richrines1]

^ sounds good! we'll definitely need something like this for testbed users (e.g. aqt and qscout) who use superstaq to compile but don't have cloud-accessible machines. having a slick interface for this would be really nice

[cdbf1]

^ sounds good! we'll definitely need something like this for testbed users (e.g. aqt and qscout) who use superstaq to compile but don't have cloud-accessible machines. having a slick interface for this would be really nice

I've implemented this by separating out the prepare_experiment function from the run function and added a compile method which will compile the circuits to the desired target (with additional optimization options). Then I've added an additional run_with_callable method which allows a custom callable to be used to evaluate each circuit which could in principal be used to run circuits on some local device

[richrines1]

can we just use

Suggested change

	probabilities = {
	format(idx, f"0{self.num_qubits}b"): 0.0 for idx in range(2**self.num_qubits)
	}
	for key, count in counts.items():
	probabilities[key] = count / total
	probabilities = {key: count / total for key, count in counts.items()}

?

we shouldn't need to include all the keys with no counts in the dictionary (/if we do we should just use an array)

[cdbf1]

Because of they way these probability dictionaries are treated as records in a dataframe in the various implementations of actual experiments (see for example the qcvv_css.ipynb), to avoid NaNs I prefer enforcing that all the bitstrings are present in the dictionary.

[richrines1]

ok sounds good. in that case i think we also need to fill in the missing values in the simulator output?

[richrines1]

similarly here: can we remove this restriction? possibly with some validation, e.g.

Suggested change

	if sorted(probability.keys()) != sorted(
	f"{i:0{self.num_qubits}b}" for i in range(2**self.num_qubits)
	):
	raise RuntimeError("Returned probabilities are missing bitstrings.")
	probability = circuit_eval_func(sample.circuit, **kwargs)
	if not all(len(key) == self.num_qubits for key in probability.keys()):
	raise RuntimeError("Returned probabilities include an incorrect number of bits.")

[cdbf1]

I like this way of checking that the bit strings are correct, and rather than enforcing that all probabilities are included I've manually set any missing bitstrings to have zero probability. However, as above (for purposes of making into a dataframe nicely, I prefer to make sure that the probabilities are stored in a dictionary with all the possible bitstrings as keys

[richrines1]

why don't we make this

Suggested change

	raw_circuit: cirq.Circuit = field(init=False)
	compiled_circuit: cirq.Circuit = field(init=False)

so we're not redefining circuit in some cases (and to make it more consistent with css.Job)

we can also get the compiled circuits for jobs submitted to hardware via job.compiled_circuits()

[cdbf1]

I've gone for a mix of both here. The Sample object now has explicitly as raw_circuit (used in the init) and a compiled_circuit attribute. By default we want to use the compiled circuit if it exists, so the circuit property returns exactly this.

In the run_on_device() method, I've also now added a line that updates the compiled circuit for each sample by retrieving it from the css.Job.

[richrines1]

cool, makes sense to me

[richrines1]

awesome, this looks great to me % that simulator output fix (if necessary) :)

[cdbf1]

Thanks for all your help @richrines1 - I think we've got something really nice here!