What does it mean to "measure a stabilizer"?

Say I have the stabilizer $$XXI$$, and a phase-flipped state $$|\psi\rangle$$. What does it mean to measure the stabilizer $$XXI$$?

What is the math behind "measurement"?

Measurements can be described by a Hermitian operator, $$A$$. In this case, $$A=X\otimes X\otimes I$$. That operator has a spectral decomposition $$A=\sum_i\lambda_iP_i,$$ where the $$\lambda_i$$ are the distinct (real) eigenvalues, and the $$P_i$$ are projectors onto the different eigenspaces, and satisfy the completeness relation $$\sum_iP_i=I$$.
So, basically, we're using $$A$$ to define the set $$\{P_i\}$$ which are the projectors in the formalism of projective measurement, i.e. we measure in this basis and get answer $$i$$ with probability $$p_i=\langle\psi|P_i|\psi\rangle$$. After the measurement, the system is in the state $$P_i|\psi\rangle/\sqrt{p_i}$$.
In your case, since $$A^2=I$$, you know the eigenvalues are $$\pm 1$$, and hence you can write $$P_{\pm}=\frac12(I\pm A).$$