One method to obtain the eigenvalues of a Hamiltonian $H$ is by applying quantum phase estimation to its time-evolution operator $U(t) = e^{-iHt}$. If I want to obtain an eigenvalue to $k$ bits in accuracy, I will need to apply the set of unitaries
$$ U^{2^j}(t) = U(2^jt), \quad 0 \leq j \leq k-1. $$
By the no-fast-forwarding theorem, implementing $U(t)$ takes $O(t)$ gates for a general Hamiltonian $H$. My question is: does this imply that quantum phase estimation, when applied to Hamiltonian simulation, has complexity that scales exponentially in the desired accuracy? I tried looking for the answer in Nielsen and Chuang and they seem to treat the controlled-$U^j$ operations as a black-box, and, to the best of my knowledge, don't discuss their complexity.