The definition you give for a graph state, and in particular the quantum Fourier transform $F$ and the controlled-$Z$ operator — where we take $ Z $ to be the unitary generalisation of the Pauli $Z$ operator, satisfying $Z = F XF^\dagger$ for $X$ a shift-by-one permutation operation — are all well-defined even in composite dimension. The Fourier transform is certainly an operation of interest for arbitrary definition; the controlled-$Z$ operation is still diagonal and unitary, and still has the relevant connections to $F$ as a tensor; there is nothing about the mathematical objects themselves which become troublesome in composite dimension.
The reason why you see so much emphasis on prime dimension is essentially that composite dimension qudits are inconvenient to analyse. The reasons for this arise from number theory: particularly in the fact that in composite dimension one must worry about zero divisors. Frankly, there aren't many in the field who think of themselves as number theorists, and very few researchers (either among the authors or the readers of articles) have much patience for number systems which are not fields such as the well-loved examples of $\mathbb C$, $\mathbb R$, $\mathbb Q$, and of course the integers modulo a prime $p$, $\mathbb Z_p$. For this reason, you will rarely see references to qudits of composite dimension anywhere in the field. Even when you do, the major concern of mathematical convenience will usually motivate some other restriction.
Quantum information theory may occasionally make use of number theory, and pure mathematics in general, but make no mistake: this field does not have much overlap with the priorities of pure mathematics. If a definition has been presented in a way that looks strangely restricted, it's reasonably likely that it's because it allows a result to be shown which would be much more challenging, or even just a little bit more awkward, to prove without that restriction — and it is considered more important to publicise examples of striking-sounding results than to present reasonably complete mathematical theories.