@ahelwer has provided a smart construction, and I will provide a more brute-force method as a complementary for beginners like me. It actually starts out by listing all the possible 16 moves. Let x,y=0,1,2,3, then all the possible moves are
$
x=0, \quad y=(0123)\rightarrow(0123) \\
x=1, \quad y=(0123)\rightarrow(1230) \\
x=2, \quad y=(0123)\rightarrow(2301) \\
x=3, \quad y=(0123)\rightarrow(3012)
$
where by arrow we mean $y$ is treated as an output. So if we denote $|xy\rangle\equiv|x\rangle|y\rangle=|ab\rangle\ |cd\rangle$, for $a,b=0,1$, we have
$
|00cd\rangle \rightarrow |00cd\rangle
$
and
$
|0100\rangle \rightarrow |0101\rangle \\
|0101\rangle \rightarrow |0110\rangle \\
|0110\rangle \rightarrow |0111\rangle \\
|0111\rangle \rightarrow |0100\rangle
$
and
$
|1000\rangle \rightarrow |1010\rangle \\
|1001\rangle \rightarrow |1011\rangle \\
|1010\rangle \rightarrow |1000\rangle \\
|1011\rangle \rightarrow |1001\rangle
$
and
$
|1100\rangle \rightarrow |1111\rangle \\
|1101\rangle \rightarrow |1100\rangle \\
|1110\rangle \rightarrow |1101\rangle \\
|1111\rangle \rightarrow |1110\rangle
$
which are exactly the rules @ahelwer described. This is not surprised from the perspective of addition of binary numbers. In terms of matrices, the above actions read
$
I_4, \begin{bmatrix}
& I_3 \\
1
\end{bmatrix},
\begin{bmatrix}
& I_2 \\
I_2
\end{bmatrix},
\begin{bmatrix}
& 1\\
I_3
\end{bmatrix}
$
respectively for the above four sectors. Since the matrices acting on different sectors (the big $16\times16$ matrix is block-diagonal), we can construct them separately. In particular, we notice for the $|01\rangle$ sector,
$
\begin{bmatrix}
& I_3 \\
1
\end{bmatrix}=\begin{bmatrix}
I_2 & \\
& X
\end{bmatrix}
\begin{bmatrix}
1 & \\
& X\\
& & 1
\end{bmatrix}
\begin{bmatrix}
X & \\
& I_2
\end{bmatrix}
$
which is CNOT conditioned by the first qubit being zero, followed by SWAP, and followed by CNOT. Similarly for the $|11\rangle$ space, which is just the hermitian conjugate. For $|10\rangle$, it is
$
\begin{bmatrix}
& I_2 \\
I_2
\end{bmatrix} = X\otimes I_2
$
Thus the complete circuit is attached below. As we explained above, since the three units (boxed by dashed lines) acting on different subspace, they commute and we can move them in different orders.