Even though this gate wouldn't be unitary, it wouldn't even be linear. Let us write this down.
Let us call this gate $\mathbf{S}$. $\mathbf{S}$ has the following properties:
$$\mathbf{S}\,|x\rangle=\begin{cases}|0\rangle&\text{if } |x\rangle=|0\rangle\text{ or }|1\rangle\\|1\rangle&\text{otherwise}\end{cases}$$
Then we would have:
$$\mathbf{S}\,(\alpha\,|0\rangle+\beta\,|1\rangle)=|1\rangle$$
by assumption. But we also have:
$$\mathbf{S}\,(\alpha\,|0\rangle+\beta\,|1\rangle)=\alpha\,\mathbf{S}\,|0\rangle+\beta\,\mathbf{S}\,|1\rangle = \alpha\,|0\rangle+\beta\,|0\rangle$$
which is contradictory. Hence, such a gate doesn't exist. Actually, with the same argument, you cannot implement such a gate with additional qubits either: since the gate makes a distinction between superposed and non-superposed states, it cannot be linear and as such, cannot be a gate.
Maybe an algorithm that would do such a task, if you want a classical result rather than a quantum state, could be implemented as such:
- Create the state you're interested in.
- Measure it.
- Repeat.
If at least once you get different results (and if we don't consider noise that may affect the state), then you know that this quantum state is in a superposition. Of course, this is a very bad algorithm:
- Creating the state can be computationally hard.
- Noise can induce inaccuracies in the measurement.
- You're never sure that a state really is not in a superposed state.
Still, you would have noticed that a measurement is performed. Hence, the wavefunction is affected. Since you want to get a classical result, the only way you have for this is to perform a measurement.
Our only solution here is to add qubits to get this piece of information. Even there, it is not possible to do such a thing. We cannot get our answer as a quantum state using the same argument stated above. We cannot even use controlled gates to see whether $\alpha$ or $\beta$ is nil, since the measurement will then affect $|x\rangle$. You can refer/learn more about that by searching "Weak measurement"
Unless I'm mistaken, it is thus impossible to perform such an operation.