Adding to @Dripto's answer.
For example, we often see a bit compared to a qubit and are told that
an "old fashion digital bit" can only be in one of two states, 0 or 1.
But qubits on the other hand can be in a state of 0, 1, or
"superposition", presumably a third state that is both 0 AND 1. But
how does being in a state of "both 0 and 1" deliver any value in
computing as it is simply the same as "I don't know"? So for example,
if a qubit represents the last binary digit of your checking account
balance, how does a superimposed 0 and 1 deliver any useful
information for that?
In this context, 0 and 1 refer to orthogonal basis states of a Hilbert space (a complex vector space) representation of the states of physical objects like electrons (say spin states of an electron - "up" and "down"). It is more appropriate to denote them as $|0\rangle$ and $|1\rangle$, according to the Dirac notation. I've written about this previously, here. Just saying "superposition of state 0 and state 1" doesn't convey any useful information, yes. However specifying the superposition state like $\alpha|0\rangle+\beta|1\rangle$, where $\alpha,\beta\in \Bbb C$ and $|\alpha|^2+|\beta|^2=1$, makes complete sense mathematically and conveys useful information. By the way, $|\alpha|^2$ is the probability of the qubit collapsing to state $|0\rangle$ upon measurement and $|\beta|^2$ is the probability of it collapsing to state $|1\rangle$, upon measurement. You might say "superposition of state 0 and state 1" doesn't make physical or intuitive sense. Sure, quantum mechanics is simply a mathematical model that happens to give correct predictions about real world phenomena. It doesn't need to make physical or intuitive sense. It just needs to work.
Also, we would never use a qubit to represent the last binary digit of your account balance, in the first place. That would be silly. And even if we do, the qubit should be restricted to the computational basis states $|0\rangle$ or $|1\rangle$, and not their superposition states.
Worse you see these same articles say things like "and two qubits can
be put together for a total of 4 states, three can make a total of 8
states" -- ok that's just $2^n$, the same capability that
"old-fashioned bits" have.
Yes, that is nonsense. A single qubit can exist in any state one out of uncountably infinite number of states like $\alpha|0\rangle+\beta|1\rangle$ where $\alpha,\beta \in \Bbb C$ (simply because there are uncountably many complex numbers and $\alpha,\beta$ can take up any of those uncountably many values). And, by extension, any $n$-qubit system can exist in an uncountably infinite number of states. I guess by $2^n$ they just meant the number of basis of the Hilbert space in which the composite qubit system belongs, for instance, $|00\rangle, |01\rangle, |10\rangle$ and $|11\rangle$ form the computational basis set for a 2 qubit system. In general, a two qubit system can take up any state like $(\alpha|0\rangle+\beta|1\rangle)\otimes (\gamma |0\rangle + \delta |1\rangle)$ (a tensor product of the state of the two individual qubits which constitute the $2$-qubit system), where $\alpha,\beta, \gamma, \delta \in \Bbb C$ and $|\alpha|^2+|\beta|^2=1$ and $|\gamma|^2+|\delta|^2=1$. $$(\alpha|0\rangle+\beta|1\rangle)\otimes (\gamma |0\rangle + \delta |1\rangle)$$ can be expressed as a linear combination of the computational basis elements $|00\rangle, |01\rangle, |10\rangle$ and $|11\rangle$ i.e. $$\alpha\gamma |0\rangle \otimes |0\rangle + \alpha\delta |0\rangle \otimes |1\rangle + \beta\gamma |1\rangle \otimes |0\rangle + \beta\delta|1\rangle \otimes |1\rangle,$$ alternatively denoted as:
$$\alpha\gamma |00\rangle + \alpha\delta |01\rangle + \beta\gamma |10\rangle + \beta\delta|11\rangle.$$
From here you can say that a $n$-qubit system can store $2^n$ values(coefficients) in parallel, although there's always the restriction that the squared sum of the moduli of the coefficients of the computational basis states must add up to $1$.
Obviously there is great promise in quantum computing so I think its
more a problem of messaging. Any suggestions for intro/primer
literature that doesn't present the quantum computing basics in this
See my previous answers: this and this, for resource recommendations. As mentioned there, I'd recommend starting with Vazirani's lectures and then moving on to Nielsen and Chuang. I recently found the 2006 lecture notes by John Watrous which are also pretty great for beginners. It helps a lot to have a thorough grounding in linear algebra, while learning quantum computing, but I suspect you already have that, being a mathematics and computer science graduate.
As for how computation using qubits can be faster, in some cases, than using classical bits, I recommend carefully thumbing through the standard quantum algorithms like Deutsch-Jozsa, Shor's, Grover's among others. Here is a simple explanation of the Deutsch-Jozsa algorithm. It would be a bit difficult to summarize that in one answer. Please keep in mind that quantum computing cannot speed up all type of computations. It's applicable to only very specific problems.