Following up on @Martin's suggestions, there might be a $\mathsf{BQP}$ algorithm to test for what you want, as long as you allow for your input circuit $U$ to be only "close to" a classical permutation matrix, that is, "close to" having a $1$ in each row/column.
For example, let $\vert x\rangle$, $\vert y\rangle$ be computational basis states, and let $\vert w\rangle$ be some monkey state (perhaps also in superposition).
We have $U\vert x\rangle=\alpha\vert y\rangle+\beta\vert w\rangle$. As long as $\alpha^2\gg\beta^2$ for each computational basis, we can choose random computational basis $\vert x\rangle$, apply $U\vert x\rangle$, and measure a small number of times.
If we always measure the same $y$ for the same input $\vert x\rangle$, and never $w$, then we can conclude, at least for that basis state $\vert x\rangle$, that $\alpha^2\gg \beta^2$; that is, the $x$ row of $U$ has a single $1$ at the $y$ column.
We can repeat to amplify, by choosing another basis state. I don't think we need to explore a lot of the $2^n$ states to get some confidence that $U$ is effectively a permutation matrix.
I think if you want to do a lot better than this, you might run against the BBBV-theorem, because you'd be looking for a tagged input (unless there are classical tests about the number of Hadamard's like described in the comments.)