Why is $|A\rangle$ (a column vector) often used as an input in many references? Is there a quantum encoding that can make the input a matrix of $A$ (an $n$ by $m$ matrix)? Then a new type of quantum gate is constructed to perform accelerated computation on the matrix.
You can find a good overview about the different methods that can be used to load a matrix into a quantum computer in Quantum algorithms for data analysis by Luongo.
Also, there is a chapter dedicated to one of these methods "Block encoding" in Lecture Notes on Quantum Algorithms for Scientific Computation by Lin.