As mentioned e.g. in this answer, if we compute the average $$\Phi(X)\equiv \mathbb{E}_U[UXU^\dagger] \equiv \int_{\mathbf U(d)} d\mu(U) UXU^\dagger,$$ where $d\mu(U)$ is the Haar measure over the unitary group $\mathbf U(d)$, we get $\operatorname{Tr}(X) I/d$.
This is relatively standard and intuitive, and a common argument to support it is in terms of how $\Phi(X)$ is invariant under the action of any unitary, meaning here that $V\Phi(X)V^\dagger=\Phi(X)$ for any $V\in\mathbf U(d)$. This kind of operation is also often referred to as twirling, discussed e.g. in this other answer here. Reasoning in terms of symmetries, the statement follows from the fact that $[V,\Phi(X)]=0$ for all $V\in\mathbf U(d)$, and the fact that multiples of the identity are the only matrices that commute with all unitaries.
I have the vague notion that this can also be seen via Schur's lemma. In such terms, I think I'd want to consider an irreducible representation of $\mathbf U(d)$ and presumably find that $\Phi(X)$ commutes with this action, for any $X$, hence must be a multiple of the identity. The most obvious such representation to consider would be the representation of $\mathbf U(d)$ on the space of linear operators $\operatorname{Lin}(\mathbb{C}^d)$ defined via $V.X\equiv \rho(V)(X)\equiv VXV^\dagger$ for $V\in\mathbf U(d)$, $X\in\operatorname{Lin}(\mathbb{C}^d)$, and $\rho(V)\in\operatorname{Lin}(\operatorname{Lin}(\mathbb{C}^n))$ the representation (in other words, we're representing the unitaries as "quantum maps"). Then, we clearly have $\rho(V)(\Phi(X))= \Phi(\rho(V)(X))$, i.e. $[\rho(V),\Phi]=0$ for all $V$. However, following this line of thought I would end up concluding that $\Phi=c \operatorname{Id}$, which is different than the statement at hand, where $\Phi$ is not the identity (as in, the identity map) but rather the map sending all states to the identity operator. I'm not sure, but this wrong conclusion is probably due to this representation not being irreducible, and thus Schur's lemma not being actually applicable.
Assuming I'm not mistaken in all this, is there a way to make some similar kind of group-theoretic reasoning work? Maybe considering subrepresentations of the above that are actually irreducible? It feels like it should work but I'm not sure how to formalise the precise steps.