This question is from Nielson and Chuang's Quantum Computation and Quantum Information:
Here $|\alpha\rangle$ is given by: $$\frac{1}{\sqrt{N-M}}\sum_{x} "|x\rangle $$ where $\sum_{x}" |x\rangle$ is the sum of states which are not the solution.
and $|\beta\rangle $ is given by: $$ \frac{1}{\sqrt{M}}\sum_{x} ' |x\rangle $$ where it represents all $M$ solutions.
$\theta$ is defined by: $$\cos\Big(\frac{\theta}{2}\Big) = \sqrt{\frac {N-M}{N}}$$
I understand everything mentioned above as well as the further geometric visualization. However, I am having trouble trying to come up with a justified way to solve this exercise. The only thing that I understand is that $G$ must be a unitary Matrix made up of trigonometric functions. How can I approach this problem?