Could a new benchmark of quantum processors Q-Score by Atos be more useful than quantum volume?

Question

A few days ago, Atos company published new benchmark for quantum computers. The benchmark is called Q-Score and it is defined as follows:

To provide a frame of reference for comparing performance scores and maintain uniformity, Q-score relies on a standard combinatorial optimization problem, the same for all assessments (the Max-Cut Problem, similar to the well-known TSP – Travelling Salesman Problem, see below). The score is calculated based on the maximum number of variables within such a problem that a quantum technology can optimize (ex: 23 variables = 23 Q-score or Qs)

According to Atos, one of main advantages of the new benchmark is its link to real-life problem (Max-Cut or TSP). Moreover, the new benchmark is not purely technical as quantum volume is.

Atos also said that

While the most powerful High Performance Computers (HPC) worldwide to come in the near term (so called “exascale”) would reach an equivalent Q-score close to 60, today we estimate, according to public data, that the best Quantum Processing Unit (QPU) yields a Q-score around 15 Qs. With recent progress, we expect quantum performance to reach Q-scores above 20 Qs in the coming year. [....] As per the above, Atos estimates quantum superiority in the context of optimization problems to be reached above 60 Qs.

Source of quotes: Atos press release

Based on the quotes above, I have these objections against the Atos benchmark:

1. The bechmark measures only one aspect of a quantum processor (QUBO problems performance) while quantum volume takes into cosideration general properties of a quantum processor and hence it describes its average behavior. This seems more useful as QUBO problems are a narrow part of tasks which can be solved on a quantum computer.

2. It has not been proven yet that a gate-based quantum computer or a quantum annealer provides advantage in solving QUBO task in comparison with classical computer. It seems that complexity is still exponential and there is a difference in constants only. Therefore, I would choose Shor, Grover or HHL algorithm as benchmark preferably as these algorithms reduces complexity in comparison with classical equivalents. If Atos is interested in easy-to-explain benchmark, lenght of RSA key which a quantum processor is able to factor (break) would be ilustrative as well.

3. Proclamation that Q-Score equal to 60 means quantum supremacy is not justified. Solving QUBO problems with thousands of variables is possible on current classical (super)computers.

My question is whether my objections are justified and make sense.

Answer 1

(3) is a solid objection. I'm not sure what specifically is meant by "number of variables within a problem that a quantum technology can optimize" but we can use the results of [1] (Fig 4) test some interpretations. In that experiment, QAOA was implemented on a nearest-neighbor connectivity superconducting processor to solve 3-regular MaxCut.

If the benchmark is meant as "the quantum computer performed better than random guessing" then that paper's results seems to indicate a Q-score of 23 or so. But performing marginally better than random guessing on a 60 qubit processor is a weak indicator for quantum advantage, especially since the algorithm is supposed to approach optimal performance with large $p$. Alternatively, if the benchmark is to be interpreted as "the quantum computer performed better than classical problems of the same size" it seems like the Q-score threshold of 60 contradicts claims from [2], which suggests hundreds to thousands of qubits required for demonstrating advantage.

Repeating this exercise with other MaxCut results like [3] it seems that the Q-score would also depend strongly on the connectivity of the graph considered; this dependence suggests that the benchmark would be biased towards quantum processors with connectivity matching the fixed benchmark problem. Practically that would probably provide an advantage to trapped ion computers compared to superconducting quantum computers.

Objection (1) will come up with any kind of single-number benchmark you can propose, e.g. Quantum volume is more general but still appropriate mainly for algorithms that are roughly "square" in depth/width. The appropriateness of the benchmark really depends on how they intend to generalize performance from it. (2) is reasonable but perhaps not very useful for a near term device.

---

[1] Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., ... & Burkett, B. (2020). Quantum approximate optimization of non-planar graph problems on a planar superconducting processor. https://arxiv.org/abs/2004.04197

[2] Guerreschi, G.G., Matsuura, A.Y. (2019). QAOA for Max-Cut requires hundreds of qubits for quantum speed-up. Sci Rep 9, 6903 https://www.nature.com/articles/s41598-019-43176-9

[3] Hamerly, R., Inagaki, T., McMahon, P. L., Venturelli, D., Marandi, A., Onodera, T., ... & Enbutsu, K. (2019). Experimental investigation of performance differences between coherent Ising machines and a quantum annealer. Science advances, 5(5) . https://arxiv.org/abs/1805.05217