A lot of papers and algorithms make use of phase shift unitaries such as $\exp(i \phi Z)$ (see, e.g., Quantum Fourier Transform). But is that really a thing that you can compute in polynomial time? I can understand that it is physically implementable by other means (e.g., letting an Hamiltonian evolve for a precise amount of time), but this solution doesn't really satisfy me from a complexity theory point of view.
Edit: let me give an example on the QFT. The QFT algorithm uses gates $$ \exp\left(i \frac{2\pi}{2^d} Z\right) $$
where $d = 1, \ldots, n$ where $n$ is the number of qubits (i.e. $N = 2^n$ is the period of the FT). This clearly depends on the size of the input $n$.