Maybe abstracting a bit the steps might help understanding the gist of it.
The quantum phase estimation algorithm can be seen in general as a gate decomposition implementing (approximately) an isometry $\mathcal U_U$ such that
$$\mathcal U_U|u_j\rangle=|u_j\rangle|\phi_j\rangle,$$
for some orthonormal basis $\{|u_j\rangle\}_j$, and with $\phi_j$ the (phases corresponding to the) eigenvalues of some fixed beforehand unitary $U$, $U|u_j\rangle=e^{2\pi i\phi_j}|u_j\rangle$.
It clearly follows that the "inverse quantum phase estimation", meaning $\mathcal U_U^\dagger$, maps $|u_j\rangle|\phi_j\rangle\to |u_j\rangle$.
So in the case of HHL,
- you start with a state of the form $\sum_j c_j |u_j\rangle$
- the QPE, $\mathcal U_U$, transforms the above into
$\sum_j c_j |u_j\rangle|\phi_j\rangle$,
- the "controlled rotation step" effectively amounts to a diagonal mapping of the form
$$\sum_j c_j |u_j\rangle|\phi_j\rangle
\to \sum_j c_j \alpha_j |u_j\rangle|\phi_j\rangle$$
for some set of coefficients $\alpha_j$ (which are related to the eigenvalues etc etc, but I don't care about this here),
- the inverse QPE step $\mathcal U_U$ now gives
$$\sum_j c_j \alpha_j |u_j\rangle|\phi_j\rangle
\to \sum_j c_j \alpha_j |u_j\rangle.$$
So in summary, you can think of the combination of QPE-controlledRotation-inverseQPE as a way to implement the mapping $|u_j\rangle\to \alpha_j |u_j\rangle$.
Of course, you might wonder why we couldn't just directly do this from the beginning, but note that we don't actually know the states $|u_j\rangle$, which in HHL are the eigenstates of $A$.
Using the QPE effectively allows you to apply a unitary diagonal in the basis $\{|u_j\rangle\}$ without needing to know the base itself.