It depends on what you mean by "elementary gates". Using only Pauli gates (that is, using only $X$, $Y$ and $Z$ gates), you cannot build a controlled $Z$ gate. One way to prove it is simply to show that the controlled $Z$ gate cannot be written as $A\otimes B$, with $A$ and $B$ being unitary matrices.
However, if we allow ourselves to use gates from the Clifford group, it is easy to build a controlled $Z$ gate using $I\otimes H$ gates and a CNOT gate:
$$\begin{align}(I\otimes H)\mathsf{C}X(I\otimes H) &= \frac12\begin{pmatrix}1&1&0&0\\1&-1&0&0\\0&0&1&1\\0&0&1&-1\end{pmatrix}\begin{pmatrix}1&0&0&0\\0&1&0&0\\0&0&0&1\\0&0&1&0\end{pmatrix}\begin{pmatrix}1&1&0&0\\1&-1&0&0\\0&0&1&1\\0&0&1&-1\end{pmatrix}\\&=\frac12\begin{pmatrix}1&1&0&0\\1&-1&0&0\\0&0&1&1\\0&0&-1&1\end{pmatrix}\begin{pmatrix}1&1&0&0\\1&-1&0&0\\0&0&1&1\\0&0&1&-1\end{pmatrix}\\&=\begin{pmatrix}1&0&0&0\\0&1&0&0\\0&0&1&0\\0&0&0&-1\end{pmatrix}\end{align}$$
Note that the circuit we are building is a CNOT applied on the second qubit, controlled by the first one, and surrounded by two $H$ gates on the second qubit.
It is however way easier to reason on the basis states. A common relation that may be useful to know is $HXH=Z$. Thus, if the first qubit is in state $|0\rangle$, we apply on the second qubit $HH=I$, but if the first qubit is in state $|1\rangle$, we apply $HXH=Z$, which is the behavior of a controlled-$Z$ gate.
