I'm not sure that anybody really knows the answer to this. Ultimately, this comes down to error correction. Universal quantum computers will have built-in error correction (and, indeed, fault-tolerance) that allows them to perform arbitrarily large computations in an essentially noise-free way.
To my knowledge (although I'm not an expert), while there has been some work towards error correction in the quantum annealing scenario (e.g. here), there's no complete fault-tolerant theory. Without it, quantum annealing is likely to be limited to finite sized computations.
So, does a fault-tolerant theory of quantum annealing exist, and can it be implemented? That's crystal ball gazing. My personal suspicion is that fault-tolerant theories exist, but they may well require connectivity of the qubits that is impractical.