(I'm unsure if you were asking for a derivation of the inner product, but hopefully this is insightful). Let's call the set of target states $T$. Recognize that, because $|\omega\rangle$ is the equal superposition of $ M$ states, each of the marked bitstrings will have a coefficient of $ \frac{1}{\sqrt{M}} $. For clarity, we can write:
$$ | \omega \rangle = \frac{1}{\sqrt{M}} \sum_{k \in T} | k \rangle $$
In contrast, $| s' \rangle $ is the orthogonal state, so it has $N - M $ states that are in a uniform superposition. Thus,
$$ |s' \rangle = \frac{1}{\sqrt{N - M}} \sum_{k \not \in T} | k \rangle $$
Finally, we note the initial state is a uniform superposition over all potential states. Thus, there are $N$ potential states, so we can write:
$$ |s \rangle = \frac{1}{\sqrt{N}} \sum_{k} | k \rangle = \frac{1}{\sqrt{N}} \Big( \sum_{k \in T} | k \rangle + \sum_{k \not \in T} |k \rangle \Big)$$
Thus, we take the inner product:
$$ \langle s' | s \rangle = \Big(\frac{1}{\sqrt{N - M}} \sum_{k \not \in T} \langle k | \Big) \Big( \frac{1}{\sqrt{N}} \sum_{j} |j \rangle \Big) $$
Note that unless $k = j $, $ \langle k | j \rangle $ vanishes as two distinct basis vectors are orthogonal. Thus, we can rearrange the summation:
$$ \sum_{k \not \in T} \frac{1}{\sqrt{N - M}} \frac{1}{\sqrt{N}} \langle k | k \rangle = \sum_{k \not \in T} \frac{1}{\sqrt{N- M }} \frac{1}{\sqrt{N}} $$
Recall that we have $M$ target states, so there are $N - M$ nontarget states (or $N - M$ times where $k \not \in T$), so:
$$ \frac{N - M}{\sqrt{N - M}} \frac{1}{\sqrt{N}} = \sqrt{\frac{N - M}{N}} $$.
As desired.
