In the last years, there has been a spur of demonstrations of devices able to perform proof of principle, small-scale, non-fault-tolerant quantum computation (or Noisy Intermediate-Scale Quantum technologies, how they have been referred to).

With this I'm mostly referring to the superconducting and ion trap devices demonstrated by groups such as Google, Microsoft, Rigetti Computing, Blatt's group (and probably others that I'm forgetting now).

These devices, as well as the ones that will follow them, are often radically different from each other (in terms of architecture, gates that are easier/harder to implement, number of qubit, connectivity between the qubits, coherence and gate times, generation and readout capabilities, gate fidelities, to name the most obvious factors).

On the other hand, it is very common in press releases and non-technical news to just say "the new X device has Y more qubits than the one before, therefore it is so much more powerful".

Is the number of qubits really such an important factor to assess these devices? Or should we instead use different metrics? More generally, are there "simple" metrics that can be used to qualitatively, but meaningfully, compare different devices?