I will answer this question in a more general context. You might know that Haar's theorem tells you that on any locally compact group $G$, there is a unique left-invariant (Borel) measure $\mu$, up to a positive constant. Left-invariance means that $\mu(g A)=\mu(A)$ for any $g\in G$ and (measurable) set $A\subset G$. This is the (left-) Haar measure on $G$.
The same argument yields a unique right-invariant measure, up to a constant.
Note that we can always turn a left-invariant Haar measure into a right-invariant Haar measure by taking the inverse: $\tilde\mu(A) := \mu(A^{-1})$. That is exactly what you are interested in.
For certain groups, the left- and right-invariant Haar measures coincide (unimodular groups) and the unitary group $U(d)$ is such a group. For these groups, your statement is correct. The proof is straightforward:
Unimodularity implies that the "inverse" of the normalised Haar measure $\mu$ on $U(d)$ is itself a left- and right-invariant Haar measure on $U(d)$. By uniqueness, $\tilde\mu$ can only differ by a positive constant from $\mu$. However, it is clear that both measures are normalised, thus $\tilde\mu = \mu$.