Often when considering such questions it may be beneficial to ask about the same thing, but from a classical perspective. For example, we can ask why do we privilege bits over, say, trits, and ask "are there any classical, ternary systems, and why would they not be as popular?"
I found out when researching this answer that a ternary computer was built by Thomas Fowler in the 1840's - Fowler was prior to or contemporaneous with Boole, from which we get boolean logic to describe two-bit operators such as AND and NOR. Indeed, the alphabet in Turing's original machines did not have only two symbols; I've read that Shannon (or maybe von Neumann?) was the first to reduce his alphabet to $0$ and $1$.
Although nothing seems to have precluded basing computation on some other arbitrary radix, other than $2$, for convenience and probably for historical reasons, bits won out over trits. There certainly has to be some path-dependence or QWERTY-like hysteresis. It might help that now, classically, bits are cheap (as opposed to quantumly, where qubits are expensive). Bits were expensive in the past, so it might have made more sense to explore ternary or mixed-radix computation, but we are long past that now and we can always pad our problem input to the next-largest power of two. See, e.g., the approach to binary-coded decimals, where we use for bits to store a decimal digit but ignore or don't-care states such as
Take note that for any radix that is not prime, for example a qudit-based system with $d=4$, we can always factor the radix into its prime decomposition, and make a similar statement. However, controlling such a qudit with $d=4$ is no easier than controlling two qubits, which, as @ChrisE mentioned, is hard enough already.
Interestingly, in certain kinds of FLASH memory, individual cells or transistors are capable of storing more than one bit of memory by discretizing the threshold voltages. Nonetheless in each such example, the number of such voltage levels is a power of two.
Similarly if we had perfect control of artificial atoms with $d=4$ or $d=8$ energy levels I would suspect that we would simply call our atom a "multi-level qubit", and still intuit our system in binary. However, if we had perfect control of an artificial atom with $d=5$ energy levels, at this time I don't think we'd want to easily give up and ignore the extra control we have of the fifth state!
See also, this very nice (although now closed) question on Stack Overflow about the classical case, with considerations similar to those in this thread.