This is simply not true. It is the global phase that is not physically relevant -- which can be immediately seen from the linearity of the Schrodinger equation: two states $| \psi \rangle, e^{i \phi} | \psi \rangle$ are both solutions to the same equation.
However, consider a qubit state, $| \psi \rangle \in \mathbb{C}^{2}$, with two different choices of local (a.k.a. relative) phases: $| + \rangle \equiv \frac{1}{\sqrt{2}}(| 0 \rangle + | 1 \rangle) \text{ and } | - \rangle \equiv \frac{1}{\sqrt{2}}(| 0 \rangle - | 1 \rangle)$, which are orthogonal to each other and hence completely distinguishable (for example, see this answer). For an explanation of the difference(s) between global and local phases, see this answer.
One way to deal with the ambiguity of global phases is to work in a complex projective space (or with density matrices) -- which is what one does, for example in quantum computing, where we represent qubit states on the Bloch sphere.
Also, phases are relevant for both amplitudes and probabilities of outcomes (again, it is the global phase that is not physically relevant). For example, see this answer, where two states that have same probability of outcomes when measured in the $\sigma_z$ basis yield different probabilities when measured in a different basis (say $\sigma_x$). Apart from this, phases are clearly very important for several quantum computing protocols like quantum phase estimation, in cyclic evolutions i.e., Berry phase, etc.