A $Z$ gate's effect is "If the target is ON then negate the phase". A controlled gate's effect is "If the control is ON then apply the gate to the target". Therefore a controlled Z's effect is "If the control is ON then if the target is ON then negate the phase".
Said another way, the controlled Z's effect is "If the control and target are both ON, negate the phase". This is a symmetric effect because it conditions on both the control and the target in the same way. The fact that we distinguish between the control and the target is just an accident of history.
Thinking of the control and target as fundamentally different is particularly tricky when it comes to the CNOT gate, as you have noticed. Assuming that the control cannot be affected is a holdover from classical-style thinking that no longer applies in the quantum case.
Classically, you think of a NOT gate as toggling the target qubit. Quantumly, the NOT gate's effect can instead be thought of as negating the phase of the $|-\rangle$ state. The $X$ gate leaves $|+\rangle = \frac{1}{\sqrt{2}} (|0\rangle + |1\rangle)$ alone and negates $|-\rangle = \frac{1}{\sqrt{2}} (|0\rangle - |1\rangle)$, in the same way that the $Z$ gate leaves $|0\rangle$ alone and negates $|1\rangle$.
So, quantumly, a CNOT's effect is "If the control is ON then if the target is $|+\rangle$ then negate the phase". Equivalently, "If the control is ON and the target is $|+\rangle$ then negate the phase". I hope that makes it clear that "control" and "target" are not really the most appropriate names. It's more like "the Z-axis control and the X-axis control".