This sort of "why?" is a deeply personal matter that depends on what you're willing to accept as a starting point. Ultimately, you might try to push it back to "why is the world the way it is? " for which I don't think there's a satisfactory answer.
Not pushing it quite so far, I choose to start from the theory of Quantum Mechanics itself (so this says nothing about the real world, it just happens that the theory is an incredibly good fit for what we see in the world around us).
The theory of Quantum Mechanics is based upon a set of assumptions/postulates. There is no real "why?" behind these, we just accept them as fact and derive the consequences. I would state the first of these as:
- Quantum states are described by unit vectors in a Hilbert space.
Right there, built in, is a Hilbert space, the key property being that it has an inner product.
The next two are, loosely,
- The probability amplitude for two independent events both happening is equal to the product of the probability amplitudes for those two events.
- The probability amplitude either of two (apparently) mutually exclusive events happening is the sum of their probability amplitudes.
Put together, this essentially describes that transformation from one state to another using matrix multiplication (and combined with the first postulate imposes that it's a unitary matrix). So, to me, the root cause is matrix multiplication (which is computed using the inner product). One particularly natural way for expressing the matrices, as you say, is using the outer product. To my mind, it is useful in several ways:
- spectral decomposition (which also leads you towards projectors, as required for measurements)
- unitaries describe basis transformations. The outer product represents that nicely as "this orthonormal basis transforms into that orthonormal basis". This is particularly important in the quantum information/computation context. At the theory level, we don't care about how a particular quantum gate works. All we care about is how inputs are transformed into outputs.
- Once you start taking the trace, there's a very tight connection between inner products and outer products.
- there is a mathematical distinction between an operator and its representation as a matrix. When talking about the operator, you pretty much have to use the outer product.