Averaging over a single Haar-random unitary applied $t$ times

I'm trying to compute the following integral: $$\int U^{\otimes t}\left|x_1,\cdots,x_t\middle\rangle\middle\langle x'_1,\cdots,x_n'\right|\left(U^\dagger\right)^{\otimes t}\,\mathrm{d}\mu(U)$$ Where $$\mu$$ is the Haar-measure. There are two easy special cases:

1. If $$x_i=x'_i$$ for every $$i$$, then the integral is equivalent to the one in this answer, which means that it is equal to $$\begin{pmatrix}d+t-1\\t\end{pmatrix}^{-1}P_{\text{Sym}^t\left(\mathbb{C}^d\right)}$$, with $$P_{\text{Sym}^t\left(\mathbb{C}^d\right)}$$ being the projector on the symmetric subspace $$\left(\mathbb{C}^d\right)^{\otimes t}$$.
2. If $$t=1$$, then it is $$\frac1d\delta_{x_1,x_1'}I$$.

However, I'm not sure how to proceed in the general case. I wanted to use this answer, but I'm not sure about what are the irreps of $$U\mapsto U^{\otimes t}$$. For instance, in the case of $$t=2$$, if we take $$x_1=x_1'$$ and $$x_2\neq x_2'$$, I feel like the irreps are a sub-symmetric space and another one. More generally, I feel like I have to "group" the $$x_i$$ and $$x_i'$$ that are equal into sub-symmetric groups, but I'm unsure about how to do it since I'm not familiar with representation theory.

How to proceed in this case? Is there a simple argument that I'm missing? I've heard about Schur-Weyl duality, but do not understand it at the moment.

EDIT: possible duplicate of Random quantum states and Schur-Weyl duality ?

Consider the permutation matrices $$\Pi_\pi$$: $$\Pi_\pi|x_1,\cdots,x_t\rangle = |x_{\pi^{-1}(1)},\cdots,x_{\pi^{-1}(t)}\rangle,$$ where $$\pi \in S_t$$ is a permutation of $$\{1,\dots,t\}$$.

It's easy to see that the map $$\Phi(M) = \int U^{\otimes t}M\left(U^\dagger\right)^{\otimes t}\,\mathrm{d}\mu(U)$$ acts as identity for $$M=\Pi_\pi$$.

It's a consequence of Schur-Weyl duality that this map is actually the orthogonal projection of $$M$$ onto the subspace spanned by all matrices $$\Pi_\pi$$.

The problem is that matrices $$\Pi_\pi$$ are non-orthogonal, so in general it's not that easy to compute the projection onto the subspace they generate. The related answers cover the approaches.

It's easy for $$t=2$$ though. There are only two permutation matrices, the identity $$I$$ and the swap $$U_{sw}$$: $$U_{sw}|x,y\rangle = |y,x\rangle$$. You can orthogonalize them by taking $$U' = U_{sw} - {\rm Tr}(U_{sw})I/d^2=U_{sw}-I/d,$$ $$U''=U'/||U'||_{HS}=(U_{sw}-I/d)/\sqrt{d^2-1}.$$ Then $$\Phi(M) = {\rm Tr}(M)I/d^2 + {\rm Tr}(M(U'')^\dagger)U''.$$

Another way is to notice that $$P_{\rm sym} = \frac{1}{2}(I + U_{sw}),$$ $$P_{\rm asym} = \frac{1}{2}(I - U_{sw}),$$ where $$P_{\rm sym}$$, $$P_{\rm asym}$$ are projectors on the symmetric and asymmetric subspaces respectively. Clearly, $${\rm span}\langle P_{\rm sym}, P_{\rm asym} \rangle = {\rm span}\langle I, U_{sw} \rangle.$$
But $$P_{\rm sym}$$, $$P_{\rm asym}$$ are orthogonal to each other. So we can write $$\Phi(M) = {\rm Tr}(MP_{\rm sym})P_{\rm sym}\frac{2}{d(d+1)} + {\rm Tr}(MP_{\rm asym})P_{\rm asym}\frac{2}{d(d-1)},$$ where coefficients come from $$||P_{\rm sym}||_{HS}^2 = {\rm Tr}(P_{\rm sym})=\frac{d(d+1)}{2},$$ $$||P_{\rm asym}||_{HS}^2 = {\rm Tr}(P_{\rm asym})=\frac{d(d-1)}{2}.$$

In general, $${\rm span}\langle \Pi_\pi\rangle$$ equals to $${\rm span}\langle P_\lambda\rangle$$, where $$P_\lambda$$ are projections onto invariant subspaces (common eigenspaces) of the representation $$\pi \rightarrow \Pi_\pi$$ of $$S_t$$.