This is an addendum to glS's answer and concern the issue of an "axis" in higher dimensions.
Let us set $D:=N^2-1$ such that we are interested in elements $R\in SO(D)$.
$R$ is in general only diagonalizable over the complex numbers and since it is a special orthogonal matrix, its eigenvalues generally come in conjugate pairs $\lambda_i = \omega_i$ and $\bar\lambda_i = \bar\omega_i$, where $\omega_i$ is a root of unity. More precisely, the spectrum depends on $D$ being even or odd.
$D$ even: Then, the spectrum consists entirely of complex conjugate pairs. In particular, the fixed point subspace of $R$ has to be even-dimensional and can be $\{0\}$. There is no axis of rotation in the sense of a one-dimensional fixed point space.
$D$ odd: In this case, $\lambda=1$ is always an eigenvalue. This is because $\det R = \det R^{-1} = 1$ and thus
$$
\det(R-I) = \det(R^{-1}R - R^{-1}) = \det(I - R)^\top = \det(I-R) = (-1)^D \det(R-I).
$$
Hence, if $D$ is odd, then $\lambda=1$ is an eigenvalue of each $R\in SO(D)$. The remaining spectrum comes in conjugate pairs. In particular, the fixed point space of $R$ is always odd-dimensional and an axis might or might not exist.
What does that imply for unitary rotations? The qudit dimension $N$ is even/odd if and only if $D=N^2-1$ is odd/even. Hence, for qubits, $N=2$, we have $D=3$ and the possible fixed point spaces are one-dimensional or full-dimensional (only for $R=I$ of course). In contrast, for odd-dimensional qudits, like $N=3$, we have $D=8$, so there are 5 possible dimensions for the fixed point spaces.
But as already pointed out, not all elements $R\in SO(N^2-1)$ come from unitaries $U\in U(N)$.
Here, we can say a bit more.
As any unitary is normal, we have an orthonormal eigenbasis $\psi_i$ for $i=1,\dots,N$. Note that $U|\psi_i\rangle\langle \psi_i|U^\dagger = |\psi_i\rangle\langle \psi_i| =: A_i$ since any eigenvalue has modulus 1.
We will now project these pure states onto the subspace of traceless, Hermitian matrices. To this end, it is not necessary to introduce an orthogonal basis like the Gell-Mann matrices so I will avoid this.
The projection is
$$
a_i := A_i - \frac{I}{N}.
$$
Since the operators $A_i$ are pure states, they have unit 2-norm. The decomposition into traceless subspace and identity subspace is orthogonal hence
$$
1 = \|A_i\|_2^2 = \| a_i \|_2^2 + \frac{1}{N} \quad \Rightarrow \quad \| a_i \|_2^2 = 1 - \frac{1}{N} = \frac{N-1}{N}.
$$
Furthermore, since the eigenstates are orthogonal,
$$
\langle a_i, a_j \rangle = \mathrm{tr}(a_i a_j) = \mathrm{tr}(A_i A_j) - \frac{2}{N} + \frac{1}{N} = - \frac{1}{N}.
$$
Defining a normalized version of the $a_i$'s as $\hat a_i := \sqrt{\frac{N}{N-1}} a_i$, we have
$$
\langle \hat a_i, \hat a_j \rangle = \begin{cases} - \frac{1}{N-1} & i \neq j, \\ 1 & \text{ else}. \end{cases}
$$
Note that for $N=2$, this is the fact that orthogonal states are mapped to antipodal points on the Bloch sphere!
In general, this is enough to conclude that the equiangular vectors $\hat a_i$ are isomorphic to the vertices of a $(N-1)$-simplex, in particular, the span of the $\hat a_i$ is $(N-1)$-dimensional.
Thus, we have shown that the fixed point space of the special orthogonal matrix $R=\mathrm{Ad}(U)$ is (at least) $(N-1)$-dimensional.
But again, for $U=I$, this subspace is $(N^2-1)$-dimensional, so I suspect that it could also be larger.
It might be possible to find a $(N-1)$-simplex of pure states in every $(N-1)$-dimensional subspace and thus every rotation fixing such a subspace could be induced by a unitary. But that's only speculation.