A qbit is a two-element vector:
$|\psi\rangle = \begin{bmatrix} \alpha \\ \beta \end{bmatrix}$
where $\alpha, \beta \in \mathbb{C}$ and $|\alpha|^2 + |\beta|^2 = 1$, a property called the 2-norm.
We have two important qbit values which we associate with the classical bits 0 and 1:
$|0\rangle = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $|1\rangle = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$
A qbit is in superposition when it is in any state other than $|0\rangle$ or $|1\rangle$. When people say superposition means the qbit "is both 0 and 1 at the same time", what they mean is it is a linear combination of 0 and 1:
$|\psi\rangle = \begin{bmatrix} \alpha \\ \beta \end{bmatrix} = \alpha\begin{bmatrix} 1 \\ 0 \end{bmatrix} + \beta\begin{bmatrix} 0 \\ 1 \end{bmatrix} = \alpha|0\rangle + \beta|1\rangle$
We operate on qbits with quantum logic gates, which must always be equivalent in action to unitary matrices. For example, here's the bitflip or "not" operator:
$X|0\rangle = \begin{bmatrix} 0 & 1 \\ 1 & 0\end{bmatrix}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} = |1\rangle$
We represent the state of multiple qbits through their tensor product:
$\begin{bmatrix} a \\ b \end{bmatrix} \otimes \begin{bmatrix} c \\ d \end{bmatrix} = \begin{bmatrix} ac \\ ad \\ bc \\ bd \end{bmatrix}$
Note that the size of the product state vector grows exponentially with the number of qbits:
$\begin{bmatrix} a \\ b \end{bmatrix} \otimes \begin{bmatrix} c \\ d \end{bmatrix} \otimes \begin{bmatrix} e \\ f \end{bmatrix} = \begin{bmatrix} ace \\ acf \\ ade \\ adf \\ bce \\ bcf \\ bde \\ bdf \end{bmatrix}$
Ordinarily this doesn't matter very much, but sometimes qbits become entangled, which means their product state cannot be factored into the tensor product of individual qbits:
$\begin{bmatrix} \frac{1}{\sqrt{2}} \\ 0 \\ 0 \\ \frac{1}{\sqrt{2}} \end{bmatrix} \ne \begin{bmatrix} a \\ b \end{bmatrix} \otimes \begin{bmatrix} c \\ d \end{bmatrix}$
When this happens, applying a quantum logic gate to $n$ qbits (which takes the form of a $2^n \times 2^n$ unitary matrix) can manipulate $2^n$ pieces of information (the elements of the product state, called amplitudes) in one shot. However, there's a bottleneck: you can only ever get $n$ bits of information out of an $n$-qbit system (through measurement), even though the $n$-qbit system contains $2^n$ pieces of information. This bottleneck means we cannot "solve problems by trying all possible solutions simultaneously" or some other such nonsense you might have read, and quantum computers can only provide speedup in very specific problem domains.
I gave a lecture where I expand on these concepts here.