A few days ago, Atos company published new benchmark for quantum computers. The benchmark is called Q-Score and it is defined as follows:
To provide a frame of reference for comparing performance scores and maintain uniformity, Q-score relies on a standard combinatorial optimization problem, the same for all assessments (the Max-Cut Problem, similar to the well-known TSP – Travelling Salesman Problem, see below). The score is calculated based on the maximum number of variables within such a problem that a quantum technology can optimize (ex: 23 variables = 23 Q-score or Qs)
According to Atos, one of main advantages of the new benchmark is its link to real-life problem (Max-Cut or TSP). Moreover, the new benchmark is not purely technical as quantum volume is.
Atos also said that
While the most powerful High Performance Computers (HPC) worldwide to come in the near term (so called “exascale”) would reach an equivalent Q-score close to 60, today we estimate, according to public data, that the best Quantum Processing Unit (QPU) yields a Q-score around 15 Qs. With recent progress, we expect quantum performance to reach Q-scores above 20 Qs in the coming year. [....] As per the above, Atos estimates quantum superiority in the context of optimization problems to be reached above 60 Qs.
Source of quotes: Atos press release
Based on the quotes above, I have these objections against the Atos benchmark:
The bechmark measures only one aspect of a quantum processor (QUBO problems performance) while quantum volume takes into cosideration general properties of a quantum processor and hence it describes its average behavior. This seems more useful as QUBO problems are a narrow part of tasks which can be solved on a quantum computer.
It has not been proven yet that a gate-based quantum computer or a quantum annealer provides advantage in solving QUBO task in comparison with classical computer. It seems that complexity is still exponential and there is a difference in constants only. Therefore, I would choose Shor, Grover or HHL algorithm as benchmark preferably as these algorithms reduces complexity in comparison with classical equivalents. If Atos is interested in easy-to-explain benchmark, lenght of RSA key which a quantum processor is able to factor (break) would be ilustrative as well.
Proclamation that Q-Score equal to 60 means quantum supremacy is not justified. Solving QUBO problems with thousands of variables is possible on current classical (super)computers.
My question is whether my objections are justified and make sense.