We want to compare an output state with some ideal state, so normally, fidelity, $F\left(\left|\psi\right>, \rho\right)$ is used as this is a good way to tell how well the possible measurement outcomes of $\rho$ compare with the possible measurement outcomes of $\left|\psi\right>$, where $\left|\psi\right>$ is the ideal output state and $\rho$ is the achieved (potentially mixed) state after some noise process. As we're comparing states, this is $$F\left(\left|\psi\right>, \rho\right) = \sqrt{\left<\psi\right|\rho\left|\psi\right>}.$$
Describing both the noise and error correction processes using Kraus operators, where $\mathcal N$ is the noise channel with Kraus operators $N_i$ and $\mathcal E$ is the error correction channel with Kraus operators $E_j$, the state after noise is $$\rho' = \mathcal N\left(\left|\psi\rangle\langle\psi\right|\right) = \sum_iN_i\left|\psi\rangle\langle\psi\right|N_i^\dagger$$ and the state after both noise and error correction is $$\rho = \mathcal E\circ\mathcal N\left(\left|\psi\rangle\langle\psi\right|\right) = \sum_{i, j}E_jN_i\left|\psi\rangle\langle\psi\right|N_i^\dagger E_j^\dagger.$$
The fidelity of this is given by \begin{align}F\left(\left|\psi\right>, \rho\right) &= \sqrt{\left<\psi\right|\rho\left|\psi\right>} \\ &= \sqrt{\sum_{i, j}\left<\psi\right|E_jN_i\left|\psi\rangle\langle\psi\right|N_i^\dagger E_j^\dagger\left|\psi\right>} \\&= \sqrt{\sum_{i, j}\left<\psi\right|E_jN_i\left|\psi\rangle\langle\psi\right|E_jN_i\left|\psi\right>^*} \\ &= \sqrt{\sum_{i, j}\lvert\left<\psi\right|E_jN_i\left|\psi\right\rangle\rvert^2}.\end{align}
For the error correction protocol to be of any use, we want the fidelity after error correction to be larger than the fidelity after noise, but before error correction, so that the error corrected state is less distinguishable from the non-corrected state. That is, we want $$F\left(\left|\psi\right>, \rho\right) > F\left(\left|\psi\right>, \rho'\right).$$ This gives $$\sqrt{\sum_{i, j}\lvert\left<\psi\right|E_jN_i\left|\psi\right\rangle\rvert^2} > \sqrt{\sum_i\lvert\left<\psi\right|N_i\left|\psi\right\rangle\rvert^2}.$$ As fidelity is positive, this can be rewritten as $$\sum_{i, j}\lvert\left<\psi\right|E_jN_i\left|\psi\right\rangle\rvert^2 > \sum_i\lvert\left<\psi\right|N_i\left|\psi\right\rangle\rvert^2.$$
Splitting $\mathcal N$ into the correctable part,$\mathcal N_c$ , for which $\mathcal E\circ\mathcal N_c\left(\left|\psi\rangle\langle\psi\right|\right) = \left|\psi\rangle\langle\psi\right|$ and the non-correctable part, $\mathcal N_{nc}$, for which $\mathcal E\circ\mathcal N_{nc}\left(\left|\psi\rangle\langle\psi\right|\right) = \sigma$. Denoting the probability of the error being correctable as $\mathbb P_c$ and non-correctable (i.e. too many errors have occurred to reconstruct the ideal state) as $\mathbb P_{nc}$ gives $$\sum_{i, j}\lvert\left<\psi\right|E_jN_i\left|\psi\right\rangle\rvert^2 = \mathbb P_c + \mathbb P_{nc}\left<\psi\vert\sigma\vert\psi\right> \geq \mathbb P_c,$$ where equality will be assumed by assuming $\left<\psi\vert\sigma\vert\psi\right> = 0$. That is a false 'correction' will project onto an orthogonal outcome to the correct one.
For $n$ qubits, with an (equal) probability of error on each qubit as $p$ (note: this is not the same as the noise parameter, which would have to be used to calculate the probability of an error), the probability of having a correctable error (assuming that the $n$ qubits have been used to encode $k$ qubits, allowing for errors on up to $t$ qubits, determined by the Singleton bound $n-k\geq 4t$) is \begin{align} \mathbb P_c &=\sum_j^t {n\choose j}p^j\left(1-p\right)^{n-j}\\ &= \left(1-p\right)^n + np\left(1-p\right)^{n-1} + \frac 12n\left(n-1\right)p^2\left(1-p\right)^{n-2} + \mathcal O\left(p^3\right) \\ &= 1 - {n\choose{t+1}}p^{t+1} + \mathcal O\left(p^{t + 2}\right)\end{align}.
Noise channels can also be written as $N_i = \sum_j\alpha_{i, j}P_j$ for a basis $P_j$, which can be used to define a process matrix $\chi_{j, k} = \sum_i\alpha_{i, j}\alpha^*_{i, k}$. This gives $$\sum_i\lvert\left<\psi\right|N_i\left|\psi\right\rangle\rvert^2 = \sum_{j, k}\chi_{j, k}\left<\psi\right|P_j\left|\psi\right\rangle\left<\psi\right|P_k\left|\psi\right\rangle\geq\chi_{0, ,0},$$ where $\chi_{0, 0} = \left(1-p\right)^n$ is the probability of no error occurring.
This gives that the error correction has been successfully in mitigating (at least some of) the noise when $$1 - {n\choose{t+1}}p^{t+1} \gtrapprox\left(1-p\right)^n.$$ While this is only valid for $\rho \ll 1$ and as a weaker bound has been used, potentially giving inaccurate results of when the error correction has been successful, this displays that error correction is good for small error probabilities as $p$ grows faster than $p^{t+1}$ when $p$ is small.
However, as $p$ gets slightly larger, $p^{t+1}$ grows faster than $p$ and, depending on prefactors, which depends on the size of the code and number of qubits to correct, will cause the error correction to incorrectly 'correct' the errors that have occurred and it starts failing as an error correction code. In the case of $n=5$, giving $t=1$, this happens at $p\approx 0.29$, although this is very much just an estimate.
Edit from comments:
As $\mathbb P_c + \mathbb P_{nc} = 1$, this gives $$\sum_{i, j}\lvert\left<\psi\right|E_jN_i\left|\psi\right\rangle\rvert^2 = \left<\psi\vert\sigma\vert\psi\right> + \mathbb P_c\left(1-\left<\psi\vert\sigma\vert\psi\right>\right).$$
Plugging this in as above further gives $$1-\left(1-\left<\psi\vert\sigma\vert\psi\right>\right){n\choose{t+1}}p^{t+1} \gtrapprox\left(1-p\right)^n,$$ which is the same behaviour as before, only with a different constant.
This also shows that, although error correction can increase the fidelity, it can't increase the fidelity to $1$, especially as there will be errors (e.g. gate errors from not being able to perfectly implement any gate in reality) arising from implementing the error correction. As any reasonably deep circuit requires, by definition, a reasonable number of gates, the fidelity after each gate is going to be less than the fidelity of the previous gate (on average) and the error correction protocol is going to be less effective. There will then be a cut-off number of gates at which point the error correction protocol will decrease the fidelity and the errors will continually compound.
This shows, to a rough approximation, that error correction, or merely reducing the error rates, is not enough for fault tolerant computation, unless errors are extremely low, depending on the circuit depth.
