You have to review the definition of the matrix representation of an operator in a given basis: If the matrix $U$ is the representation of some operator $u$ in the basis $\{|00\rangle,|01\rangle,|10\rangle,|11\rangle\}$, it means that the first column of $U$ contains the coordinates of $u(|00\rangle)$ in that basis, the second column is $u(|01\rangle)$ and so on:
$\begin{array}{cc} \begin{array}{cccc}\phantom{2} \text{$ \tiny u(|00\rangle)$} & \text{$\tiny u(|01\rangle)$} & \text{$\tiny u(|10\rangle)$} & \text{$ \tiny u(|11\rangle)$} \\ \end{array} &\\ \frac{1}{2}\begin{bmatrix} \phantom{2}-1\phantom{2} & \phantom{}-1\phantom{}& \phantom{2}1\phantom{2}&\phantom{2}1\phantom{2}\\ 1 & -1 & 1& -1\\ 1 & -1& -1&1\\ 1 & -1 & 1& 1\\ \end{bmatrix}% % &\begin{array}{l} \left.\vphantom{\begin{bmatrix} 0\\ \end{bmatrix}}\right. \text{$\tiny |00\rangle$}\\ \left.\vphantom{\begin{bmatrix} 0\\ \end{bmatrix}}\right. \text{$\tiny |01\rangle$}\\ \left.\vphantom{\begin{bmatrix} 0\\ \end{bmatrix}}\right. \text{$\tiny |10\rangle$}\\ \left.\vphantom{\begin{bmatrix} 0\\ \end{bmatrix}}\right. \text{$\tiny |11\rangle$}\\ \end{array} \\ \end{array}$
So if you swap the 1st and 3rd columns, you swap $u(|00\rangle)$ and $u(|10\rangle)$, the others are left untouched.
(I used your operator $U$ in my answer but you should be aware it is not unitary)