Usually we want a quantum computer that can perform all foreseeable unitary operations U($n$). A quantum processor that can naturally perform at least 2 rotation operators $R_k(\theta)=\exp(-i\theta\sigma_k/2)$, where $\sigma_k$ are the Pauli matrices; can generate any SU(2) rotation of the Bloch sphere. And the usual Pauli operations (up to a global phase) can be generated by just choosing the right angle $R_k(-\pi)=i\sigma_k$. By creating some controlled-$i\sigma_x$, I guess we could generate all SU($n$). Would a quantum processor restricted to SU($n$) operations limit the power to simulate quantum systems and other algorithms of interest?
Edit: note that controlled-SU(2) gates are still in SU(4).