With $\mathcal{G}, A$ being unitary matrices and $|\psi\rangle$ the initial state.
First, the system is:
$\frac{1}{\sqrt{2}}(|0\rangle+|1\rangle)\;\otimes|\psi\rangle$
Next:
$\frac{1}{\sqrt{2}}(|0\rangle\otimes\mathcal{G}|\psi\rangle+|1\rangle\otimes A|\psi\rangle)$
This paper shows that:
A measurement of the ancilla selects one of the two branches and results in either the state $\mid\psi^{'}_0\rangle=\frac{1}{2 \sqrt{p_0}}(\mathcal{G}+A)\mid\psi\rangle$ with probability $$p_0=\frac{1}{4}\langle\psi\mid(\mathcal{G}+A)^\dagger(\mathcal{G}+A)\mid\psi\rangle$$
This sentence makes me confused about how to get the amplitude when it is a matrix? And how about the state $\mid\psi^{'}_0\rangle$, I think it should be $\frac{1}{2}(\mathcal{G}+A)$
Thanks for reading!