Quantum Computing Stack Exchange is a question and answer site for engineers, scientists, programmers, and computing professionals interested in quantum computing. It only takes a minute to sign up.
Sign up to join this community
Would there be any disadvantages of using inner product, that is, $\mathrm{Tr}(A^{\dagger}B)$ (say making it, $\mathrm{Tr}(\sqrt A \sqrt B)$ to normalise) to quantify how far two quantum states are instead of fidelity, which is $\mathrm{Tr}(\sqrt{\sqrt A B \sqrt A})$?
Apart from losing the relationship with trace distance.
The quantity $\text{Tr}(\sqrt{A}\sqrt{B})$ that you defined there is actually referred to as the "just-as-good fidelity" (see 1801.02800) because it does have a relationship with the trace distance very similar to the standard fidelity and is therefore "just as good" for quantifying the distinguishability of states. There is no intrinsic reason to prefer the standard fidelity, but the latter gets used a lot because it is often useful to use Uhlmann's theorem, as glS points out in the other answer.
A few thoughts:
It mostly depends on what you are trying to quantify.
The inner product of states, $\text{Tr}(\rho\sigma)$, is used to quantify the distance in state space. More precisely, the squared distance between two states is commonly defined as $$D(\rho,\sigma)^2\equiv \|\rho-\sigma\|_2^2=1-\text{Tr}(\rho\sigma).$$ This is useful and used for example to understand the geometrical features of the space of states (think Bloch representations of states). For pure states, the definitions coincide, as you have $\text{Tr}(\rho\sigma)=\lvert\langle\psi_\rho\rvert\psi_\sigma\rangle\rvert^2$ if $\rho=\lvert\psi_\rho\rangle\!\langle\psi_\rho\rvert$ and same for $\sigma$. For mixed states, however, I don't know if there is a direct physical interpretation of $\text{Tr}(\rho\sigma)$.
On the other hand, the fidelity quantifies the indistinguishability between states (equivalently, the probability of finding one state to equal the other, in an appropriate measurement choice). In the case of mixed states, you have to decide what is meant by this, as there are potentially many possible pure states that are compatible with any given mixed state, and comparing the pure states corresponding to $\rho$ and $\sigma$ might give different results depending on such choice. The state fidelity considers the worst-case scenario in terms of distinguishability: as shown by Uhlmann's theorem (see also this question and links therein), $\text{Tr}(\lvert\sqrt\rho\sqrt\sigma\rvert)$ equals the maximum fidelity among the purifications of $\rho$ and $\sigma$: $$\text{Tr}(\lvert\sqrt\rho\sqrt\sigma\rvert)=\max_{\psi_\rho,\psi_\sigma}\lvert\langle\psi_\rho\rvert\psi_\sigma\rangle\rvert,$$ where the maximisation is performed over all possible purifications $\psi_\rho,\psi_\sigma$ of $\rho,\sigma$, respectively.
