An alternative method, as compared to the existing answer, is perhaps better suited to this specific case where you already know some of the structure of the output matrix.
Label each of the columns in binary in numerical order: 00, 01, 10, 11. Label the rows the same. Now, columns correspond to inputs, and rows correspond to outputs.
So, let's say you want to know the matrix element in the first row and second column. In effect, you are asking "what is the probability amplitude for a state that starts in $|01\rangle$ to end in $|00\rangle$?".
Thus, make the input $|01\rangle$. You can run through the action of the circuit
$$
|01\rangle\rightarrow|11\rangle\rightarrow |10\rangle\rightarrow |00\rangle.
$$
So, the probability amplitude for arriving in state $|00\rangle$ is 1. Hence, that matrix element is 1.
There's an extra trick you can use here. You evolution is unitary (there's no measurement). Unitary matrices have rows and columns that have sum-mod-square of 1. Hence, if you know one element is 1, every other element in that row and column must be 0.