For a presentation from first principles, I like Ryan O'Donnell's answer. But for a slightly higher-level algebraic treatment, here's how I would do it.
The main feature of a controlled-$U$ operation, for any unitary $U$, is that it (coherently) performs an operation on some qubits depending on the value of some single qubit. The way that we can write this explicitly algebraically (with the control on the first qubit) is:
$$ \mathit{CU} \;=\; \def\ket#1{\lvert #1 \rangle}\def\bra#1{\langle #1 \rvert}\ket{0}\!\bra{0} \!\otimes\! \mathbf 1 \,+\, \ket{1}\!\bra{1} \!\otimes\! U$$
where $\mathbf 1$ is an identity matrix of the same dimension as $U$. Here, $\ket{0}\!\bra{0}$ and $\ket{1}\!\bra{1}$ are projectors onto the states $\ket{0}$ and $\ket{1}$ of the control qubit — but we are not using them here as elements of a measurement, but to describe the effect on the other qubits depending on one or the other subspace of the state-space of the first qubit.
We can use this to derive the matrix for the gate $\mathit{CX}_{1,3}$ which performs an $X$ operation on qubit 3, coherently conditioned on the state of qubit 1, by thinking of this as a controlled-$(\mathbf 1_2 \!\otimes\! X)$ operation on qubits 2 and 3:
$$
\begin{aligned}
\mathit{CX}_{1,3} \;&=\;
\ket{0}\!\bra{0} \otimes \mathbf 1_4 \,+\, \ket{1}\!\bra{1} \otimes (\mathbf 1_2 \otimes X)
\\[1ex]&=\;
\begin{bmatrix}
\mathbf 1_4 & \mathbf 0_4 \\
\mathbf 0_4 & (\mathbf 1_2 \!\otimes\! X)
\end{bmatrix}
\;=\;
\begin{bmatrix}
\mathbf 1_2 & \mathbf 0_2 & \mathbf 0_2 & \mathbf 0_2 \\
\mathbf 0_2 & \mathbf 1_2 & \mathbf 0_2 & \mathbf 0_2 \\
\mathbf 0_2 & \mathbf 0_2 & X & \mathbf 0_2 \\
\mathbf 0_2 & \mathbf 0_2 & \mathbf 0_2 & X
\end{bmatrix},
\end{aligned}
$$
where the latter two are block matrix representations to save on space (and sanity).
Better still: we can recognise that — on some mathematical level where we allow ourselves to realise that the order of the tensor factors doesn't have to be in some fixed order — the control and the target of the operation can be on any two tensor factors, and that we can fill in the description of the operator on all of the other qubits with $\mathbf 1_2$. This would allow us to jump straight to the representation
$$
\begin{alignat}{2}
\mathit{CX}_{1,3} \;&=&\;
\underbrace{\ket{0}\!\bra{0}}_{\text{control}} \otimes \underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{uninvolved}\!\!\!\!} \otimes \underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{target}\!\!\!\!}
&+\,
\underbrace{\ket{1}\!\bra{1}}_{\text{control}} \otimes \underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{uninvolved}\!\!\!\!} \otimes \underbrace{\; X\;}_{\!\!\!\!\text{target}\!\!\!\!}
\\[1ex]&=&\;
\begin{bmatrix}
\mathbf 1_2 & \mathbf 0_2 & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} \\
\mathbf 0_2 & \mathbf 1_2 & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} \\
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} \\
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2}
\end{bmatrix}
\,&+\,
\begin{bmatrix}
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} \\
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} \\
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & X & \mathbf 0_2 \\
\phantom{\mathbf 0_2} & \phantom{\mathbf 0_2} & {\mathbf 0_2} & X
\end{bmatrix}
\end{alignat}
$$
and also allows us to immediately see what to do if the roles of control and target are reversed:
$$
\begin{alignat}{2}
\mathit{CX}_{3,1} \;&=&\;
\underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{target}\!\!\!\!} \otimes \underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{uninvolved}\!\!\!\!} \otimes \underbrace{\ket{0}\!\bra{0}}_{\text{control}}
\,&+\,
\underbrace{\;X\;}_{\!\!\!\!\text{target}\!\!\!\!} \otimes \underbrace{\;\mathbf 1_2\;}_{\!\!\!\!\text{uninvolved}\!\!\!\!} \otimes \underbrace{\ket{1}\!\bra{1}}_{\text{control}}
\\[1ex]&=&\;
{\scriptstyle\begin{bmatrix}
\!\ket{0}\!\bra{0}\!\! & & & \\
& \!\!\ket{0}\!\bra{0}\!\! & & \\
& & \!\!\ket{0}\!\bra{0}\!\! & \\
& & & \!\!\ket{0}\!\bra{0}
\end{bmatrix}}
\,&+\,
{\scriptstyle\begin{bmatrix}
& & \!\!\ket{1}\!\bra{1}\!\! & \\
& & & \!\!\ket{1}\!\bra{1} \\
\!\ket{1}\!\bra{1}\!\! & & & \\
& \!\!\ket{1}\!\bra{1} & &
\end{bmatrix}}
\\[1ex]&=&\;
\left[{\scriptstyle\begin{matrix}
1 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 1
\end{matrix}}\right.\,\,&\,\,\left.{\scriptstyle\begin{matrix}
0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0
\end{matrix}}\right].
\end{alignat}
$$
But best of all: if you can write down these operators algebraically, you can take the first steps towards dispensing with the giant matrices entirely, instead reasoning about these operators algebraically using expressions such as $\mathit{CX}_{1,3} =
\ket{0}\!\bra{0} \! \otimes\!\mathbf 1_2\! \otimes\! \mathbf 1_2 +
\ket{1}\!\bra{1} \! \otimes\! \mathbf 1_2 \! \otimes\! X$
and
$\mathit{CX}_{3,1} =
\mathbf 1_2 \! \otimes\! \mathbf 1_2 \! \otimes \! \ket{0}\!\bra{0} +
X \! \otimes\! \mathbf 1_2 \! \otimes \! \ket{1}\!\bra{1}$.
There will be a limit to how much you can do with these, of course — a simple change in representation is unlikely to make a difficult quantum algorithm efficiently solvable, let alone tractable by manual calculation — but you can reason about simple circuits much more effectively using these expressions than with giant space-eating matrices.