I'm studying Shor's algorithm. but I see from the beginning this empty circle. what this circle means??
It is a "negative" control.
In other terms, the gate in your circuit will apply a NOT on the last qubit if and only if the following conditions are both met:
- $\vert x_3\rangle = \vert 0 \rangle$
- the first ancilla qubit is in the state $\vert 1 \rangle$.
A "normal" control (i.e. not an empty circle) would change the first condition to $\vert x_3 \rangle = \vert 1 \rangle$.