It is unfortunately not true that for general quantum computations an error correcting code is exact. There is no QECC that can correct all possible errors.
Consider, for example, one of the simplest QECC's there is: the 3-bit repetition code.
In this code, the logical Hilbert space is spanned by the logical states $|\overline{0}\rangle = |000\rangle$ & $|\overline{1}\rangle = |111\rangle$.
By parity comparisons we can infer if certain errors happened. The errors we can detect are all the single qubit $X$-flips: $XII$, $IXI$ and $IIX$.
The important thing to realize is that any error model that only has these errors is not at all a realistic error model. If we take something more realistic, (but still overly simplified), we might have that all qubits individually are flipped with a probability $\epsilon$. That is to say, we have Kraus operators:
\begin{equation}
E_{1} = \sqrt{1-\epsilon}I, \,E_{2} = \sqrt{\epsilon}X,
\end{equation}
on all three qubits. Therefore, our entire system has $2^{3} = 8$ Kraus operators, namely the three-fold tensor product of $E_{1}$ and $E_{2}$:
\begin{equation}
\begin{split}
A_{1} = (1-\epsilon)^{3/2}III, \,& A_{8} = \epsilon^{3/2}XXX \\
A_{2} = (1-\epsilon)(\epsilon)^{1/2}XII, \,& A_{5} = (1-\epsilon)^{1/2}\epsilon IXX \\
A_{3} = (1-\epsilon)(\epsilon)^{1/2}IXI, \,& A_{6} = (1-\epsilon)^{1/2}\epsilon XIX \\
A_{4} = (1-\epsilon)(\epsilon)^{1/2}IIX, \,& A_{7} = (1-\epsilon)^{1/2}\epsilon XXI,\\
\end{split}
\end{equation}
where I have put Kraus operators giving raise to the same error syndrome in the same row.
If we have some logical state $\rho = \in \mathcal{H}_{l}$, and we put it through the above channel, and then through error correction, even after perfect error decoding we get the following mixtures (depending on the error syndrome we actually measured):
\begin{equation}
\begin{split}
\rho \rightarrow \frac{1}{(1-\epsilon)^{3} + \epsilon^{3}}\big((1-\epsilon)^{3}\rho + \epsilon^{3} XXX\rho XXX \big), \\
\rho \rightarrow \frac{1}{(1-\epsilon)^{2}\epsilon + (1-\epsilon)\epsilon^{2}}\big((1-\epsilon)^{2}\epsilon\rho + (1-\epsilon)\epsilon^{2} XXX\rho XXX \big), \\
\end{split}
\end{equation}
with the top row holding for a trivial error syndrome, and the bottom row holding for any non-trivial error syndrome.
So even in this very simple case our QECC is never perfect, because we can never correct all errors. We limit ourselves by correcting only a subset of all errors, and we normally choose the lower weight errors. There will necessarily be errors with the same error syndrome but with different correction operators, so we can never hope to correct all of these.
We just choose the more likely errors, and we design all of our interactions with the code (encoding, decoding, measurement, syndrome extraction, logical gates etc.) in such a way that we limit the number of higher weight errors. This last thing is broadly known as fault-tolerance.
All in all, a QECC only brings down the rate of erroneous states, but it never brings it to zero (note that in the last equation, the correct states are all more likely than the erroneous ones by a factor of at least $\sim \epsilon$. However, the erroneous states are not non-existant). By repeated QECC's (concatenated codes), we can bring down the error rate even more. Of course, all operations that we need to do to perform QECC introduce some error as well, and we also have to correct for these. The threshold theorem states that if $\epsilon$ is small enough, we can correct the new errors faster than they arise, thereby allowing us to get arbitrary small errors.
This also taps into your last question: essentially all error correction is of the approximate type. The error only 'vanishes' by the use of repeated and fault-tolerant implementation of codes.
