Lets understand this first using Bra-Ket notation then we move on to answering : what does a matrix maps by just looking at it.
Lets get some basics out of the way,
the vectors $\vert 0 \rangle$ and $\vert 1 \rangle$ are represented by,
\begin{equation}
\vert 0 \rangle = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \ \ \ \ \ \& \ \ \
\vert 1 \rangle = \begin{bmatrix} 0 \\ 1 \end{bmatrix}.
\end{equation}
This implies,
\begin{equation}
\vert 0 \rangle \langle 1\vert = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}
\ \ \ \ \ \& \ \ \
\vert 1 \rangle \langle 0\vert = \begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix}.
\end{equation}
Now, I can write my $X$ gate as,
\begin{equation}
X = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}
+ \begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix} = \vert 0 \rangle \langle 1\vert + \vert 1 \rangle \langle 0\vert .
\end{equation}
The term $\vert 1 \rangle \langle 0\vert$ maps $\vert 0 \rangle \rightarrow \vert 1 \rangle$, and the term $\vert 0 \rangle \langle 1\vert$ maps $\vert 0 \rangle \rightarrow \vert 1 \rangle$. You can see this by:
\begin{equation}
\left(\vert 0 \rangle \langle 1 \vert\right) \vert 1\rangle = \vert 0 \rangle
\ \ \ \ \ \& \ \ \
\left(\vert 1 \rangle \langle 0 \vert\right) \vert 0\rangle = \vert 1 \rangle.
\end{equation}
Given a matix you can always write it in this bra-ket notation, and easily see how it maps.
As you gain experience working with these it is easy to see that a general matrix pictorially can be represented as,
\begin{equation}
\begin{bmatrix} a & b \\ c & d \end{bmatrix} = \begin{bmatrix} a\vert 0 \rangle\langle 0 \vert & b \vert 0 \rangle\langle 1 \vert \\ c\vert 1 \rangle\langle 0 \vert & d\vert 1 \rangle\langle 1 \vert \end{bmatrix}
\end{equation}
Note the word pictorially, cause the way I wrote it, in standard text it would imply a $4\times4$ matrix. Now given a matrix you can easily work out how each of the non-zero term acts on the basis states $\vert 0 \rangle, \vert 1 \rangle$ ans see where the matrix maps to.
For example,
\begin{equation}
Z = \begin{bmatrix}1 & 0 \\ 0 & -1 \end{bmatrix}
\end{equation}
has non-zero terms $\vert 0 \rangle\langle0 \vert$ which maps $\vert 0 \rangle \rightarrow \vert 0 \rangle$, and $-\vert 1 \rangle\langle1 \vert$ which maps $\vert 1 \rangle \rightarrow -\vert 1 \rangle$.
Now a homework for you, Understand the mapping of Hadamard gate and then move on to understand $CNOT$ gate.
Note: I did not worry about what $\vert 1 \rangle \langle 0\vert$ maps $\vert 1 \rangle$ to, cause of orthogonality.
EDIT : After writing this I realized maybe you are asking the following
Lets say someone gives you a mapping and ask you to find the matrix that applies the said mapping. Note that
$$
\vert x \rangle \langle b \vert
$$
maps $\vert b \rangle $ to $\vert x \rangle$.
So as an example if one asks you find the matrix that maps
$\vert 0 \rangle \rightarrow (\vert 0 \rangle + \vert 1 \rangle)/\sqrt{2}$
and $\vert 1 \rangle \rightarrow (\vert 0 \rangle - \vert 1 \rangle)/\sqrt{2}$ you can easily write the matrix being
$$
\frac{\vert 0 \rangle + \vert 1 \rangle}{\sqrt{2}}\langle 0 \vert + \frac{\vert 0 \rangle - \vert 1 \rangle}{\sqrt{2}}\langle 1 \vert
$$
$$
=\frac{1}{\sqrt{2}}(\vert 0 \rangle\langle 0 \vert + \vert 0 \rangle\langle 1 \vert + \vert 1 \rangle\langle 0 \vert - \vert 1 \rangle\langle 1 \vert) = \frac{1}{\sqrt{2}}\begin{bmatrix}1 & 1 \\ 1 & -1 \end{bmatrix}
$$