Using a simple puzzle game to benchmark quantum computers is the most clever approach I have seen so far.
The author of the aforementioned article, James, makes a nice analogy to buying a laptop ("more than just a single number when comparing") in an answer to How should different quantum computing devices be compared?.
Another answer by whurley to the same question mentions the IEEE Standard for Quantum Computing Performance Metrics & Performance Benchmarking.
In When will we know that quantum supremacy has been reached? there is an answer by Niel which includes this snippet:
But the bottom line is that, yes, "quantum supremacy" is precisely about "not being able to simulate quantum computers of certain sizes", or at least not being able to simulate certain specific processes that you can have them perform, and this benchmark depends not only on quantum technology but on the best available classical technology and the best available classical techniques. It is a blurry boundary which, if we are being serious about things, we will only be confident that we have passed a year or two after the fact. But it is an important boundary to cross.
My question is bipartite:
What will be the primary metrics for quantum computers? (Clasical examples include processor speed, RAM & hard drive space)
Best tools / strategies for benchmarking above metrics? (Existing or proposed)