In the application of QFT for quantum phase estimation (QPE) of a unitary $\mathbf{U}$, one has to perform successive controlled operations using powers of $\mathbf{U}$. In order not to break the complexity, each of the controlled $\mathbf{U}^{2^i}$ gates must have the same complexity when applying them as $\mathbf{U}$.
Hence, if I want to apply a QPE on a gate whose circuit is known, how should I proceed? If I just concatente the circuit of $\mathbf{U}$ to itself $2^i$ times, applying $\mathbf{U}^{2^i}$ is $2^i$ times as long as applying $\mathbf{U}$, which breaks the complexity.
I've seen papers and posts about modular exponentiation, but I'm not sure this would work in my case, since I do not work modulo some whole number.
I understand that fundamentally, what I want to do is to implement a unitary which, given a state $|x\rangle\,|y\rangle$ returns the state $|x\rangle\,\mathbf{U}^x|y\rangle$, but I don't know how should I perform this.