How should different quantum computing devices be compared?

Question

In the last years, there has been a spur of demonstrations of devices able to perform proof of principle, small-scale, non-fault-tolerant quantum computation (or Noisy Intermediate-Scale Quantum technologies, how they have been referred to).

With this I'm mostly referring to the superconducting and ion trap devices demonstrated by groups such as Google, Microsoft, Rigetti Computing, Blatt's group (and probably others that I'm forgetting now).

These devices, as well as the ones that will follow them, are often radically different from each other (in terms of architecture, gates that are easier/harder to implement, number of qubit, connectivity between the qubits, coherence and gate times, generation and readout capabilities, gate fidelities, to name the most obvious factors).

On the other hand, it is very common in press releases and non-technical news to just say "the new X device has Y more qubits than the one before, therefore it is so much more powerful".

Is the number of qubits really such an important factor to assess these devices? Or should we instead use different metrics? More generally, are there "simple" metrics that can be used to qualitatively, but meaningfully, compare different devices?

Answer 1

I think the answer depends on why you are comparing them. Things like the quantum volume, are perhaps better suited to defining progress in the development of devices rather than fully informing end users.

For example, you are buying a new laptop, you probably use more than just a single number when comparing them. The same should be true for quantum processors. There are many different aspects to a device: number of qubits, connectivity, all the different types of noise, time for measurement (and so whether feedback from measurement results is feasible), gate operation times, etc. All these need to be combined to tell you the one thing you actually need to know: can it run the program that you want to run? That is, I think, always going to be the most pertinent comparison. But it is also the trickiest.

Answer 2

This is a greatly debated topic, and I'm not sure there is an answer to your question at the current time. However, the IEEE (Institute of Electrical and Electronics Engineers) has proposed PAR 7131 - Standard for Quantum Computing Performance Metrics & Performance Benchmarking:

The purpose of this project is to provide a standardized set of performance metrics and a standardized methodology of benchmarking the speed/performance of various types of quantum computing hardware and software as well as comparing these performance metrics to identical metrics in classical computers such that users of this document may determine the speed of a quantum computer for a specific application can easily, and reliably, compare computer performance.

Full disclosure I am the current Chair of the Quantum Computing Standards Workgroup and the reason this PAR was originally proposed was because of a lack of documentation/standards on testing the various quantum computing architectures against classical architectures and each other. The factors you sighted above

number of qubit, connectivity between the qubits, coherence and gate times, generation and readout capabilities, gate fidelities

are all included as are several other factors. As importantly we've also been working on a way to standardize solvers; an often overlooked component in benchmarking. Non-optimized solvers all too often benefit a quantum machine when comparing quantum architectures to classical architectures. That is, the solver running on the quantum architecture is always optimized where the solver running on the classical architecture is not. This creates an inherent bias in favor of the quantum architecture.

If you're interested in participating in the development of this standard please let me know, the more people involved from both the quantum and classical sides of the argument the better imho. In the meantime the PAR will start work shortly, and will be coordinating their efforts with other standards organizations so that a single common standard with no bias can emerge to help address performance and benchmarking in the future.

Answer 3

While number of qubits should be part of such a metric, as you say, it's far from everything.

However, comparing two different completely different devices (e.g. superconducting and linear optics) is not the most straightforward task¹.

Factors

Asking about coherence and gate times is equivalent to asking about fidelity and gate times¹. Gates being harder or easier to implement just affects the fidelity again.

Initialisation rate, qubit/entanglement generation and readout capabilities (etc.) are going to affect overall fidelities as well as something akin to 'how frequently (on average) can we perform a computation (while getting a high-enough fidelity result, for some idea of 'high-enough fidelity')'.

In terms of architecture, the more macro-architecture (e.g. qRAM) will have its own standards and benchmarks, such as readout time, 'is readout on demand?' and of course, fidelity.

The more microarchitecture can be described under the same notions of connectivity.

Another, often ignored, metric is the power/resources used.

Overall, this may have narrowed this list down slightly, but it's still a list that involves a fair amount of comparison. Comparing different devices that use the same method isn't even that straightforward as (at current levels of technology), the processors with higher numbers of qubits often have lower fidelities².

Quantum volume

Thankfully, a few people at IBM have taken the above (except for power used and the architecture) and defined something a bit more useful than 'number of qubits' and called it quantum volume. In this, for a random pair of $2$ qubits, they first define an effective error rate, $\epsilon_{eff}$, by considering what gate errors would be required in an otherwise perfect system to give the same error as the device. This may require the use of SWAP for low connectivity and Solovay-Kitaev-esque methods for low numbers of implementable gates. This is countered by using teleportation if the system has "fast measurements and feedback" and any other appropriate method.

For a total number of qubits $n$ and maximising over the number of 'active qubits', $n'$, the quantum volume is $$V_Q = \max_{n'\leq n}\min\left[n', \frac{1}{\epsilon_{eff}\left(n'\right)}\right]^2.$$

Of course, we want to move beyond the point of science and into engineering. For that we need a standard³. This is currently being planned, as detailed in Whurley's answer.

However, as any comparison between such lists isn't going to be straightforward, there's always the more subjective way, such as Quantum Awesomeness, where the enjoyment of the game depends on how good the processor is⁴.

---

^{1 In this particular case, one example is that as photons don't decohere, so this has to be adapted to asking about the length of time or number of gates before the realised state is no longer a good approximation to the ideal state, which is just asking for the fidelity, or fidelity and gate times}

^{2 I've tried this much at least and even this isn't exactly the most fun task}

^{3 The first, unlike in XKCD 927}

^{4 The author's opinion is that, while an awesome idea and helpful for getting an idea of how good a processor is, saying that one processor is better than another at such a game is a bit too subjective to tell if one processor is actually better than another}

Answer 4

IBM is promoting their quantum volume (see also this) idea to quantify the power of a gate model machine with a single number. Before IBM, there was an attempt from Rigetti to define a total quantum factor. Unclear if it captures what we want in terms of usefulness of devices for applications. Things such as quantum volume are be designed with supremacy experiments in mind, it seems to me. I am leaning to think that a metric should be really application specific. For sampling, this work suggested to use the qBAS score.

For quantum annealing and similar analog approaches, it seems the community is agreeing on time-to-solution and variants; once again quite application specifics.

The community is working on defining metrics, and I expect in 2018 to see actual runs of the same problem on different devices (empirical comparison).