I'll cover a slightly more general case.
Let $P_k$, $k=1,...,N$ a complete set of orthogonal projectors: $\sum_k P_k=I$ and $P_j P_k=\delta_{jk}P_j$.
Consider the map $\mathcal E(\rho)=\sum_k P_k \rho P_k$.
We want to find a set of unitaries $\mathcal U_k$ and probabilities $p_k$ such that $\mathcal E(\rho)=\sum_\ell p_\ell\mathcal U_\ell\rho\,\mathcal U_\ell$.
Define the unitaries $\mathcal U_k=\mathcal U_k^\dagger=I-2P_k$, $k=1,...,N$. We then have
\begin{align}
\sum_k\mathcal U_k\rho\,\mathcal{U}_k &=N\rho-2\sum_k\{P_k,\rho\}+4\sum_k P_k\rho P_k \\
&=(N-4)\rho+4\sum_k P_k\rho P_k.
\end{align}
Thus,
$$\frac{1}{4}\sum_k \mathcal U_k\rho\,\mathcal U_k+\frac{4-N}{4}\rho=\sum_k P_k\rho P_k.\tag A$$
Lets work out explicitly a few cases:
$N=2$ case
This is the problem as originally stated (with $P_1=P$ and $P_2=Q$).
We have
$$\frac{1}{4}(\mathcal U_1\rho\,\mathcal U_1+\mathcal U_2\rho\,\mathcal U_2)
+\frac{1}{2}\rho=P_1\rho P_1+P_2\rho P_2.$$
$N=3$ case
$$\frac{1}{4}(\mathcal U_1\rho\,\mathcal U_1+\mathcal U_2\rho\,\mathcal U_2+\mathcal U_3\rho\,\mathcal U_3)
+\frac{1}{4}\rho
=P_1\rho P_1+P_2\rho P_2+P_3\rho P_3.$$
$N=4$ case
$$\frac{1}{4}\sum_{k=1}^4 \mathcal U_k\rho\,\mathcal U_k=\sum_{k=1}^4 P_k\rho P_k.$$
$N>4$ case
Interestingly, this method stops working for $N>4$ (which of course does not mean that we cannot decompose these channels using unitaries, only that this particular decomposition method does not work anymore). More precisely, (A) still holds, but it's no longer a valid Kraus decomposition because $4-N<0$.