There are two different issues at play here. First is the difficulty of the calculation. If you have an $n$-qubit unitary $U$, then to evaluate $Ux$, you have to multiply a $2^n\times 2^n$ matrix with a $2^n$-element vector. This takes a long time (by naive methods, $O(4^n)$), even if you understand that the method seems simple and straightforward. There's just so much data. In comparison, phase estimation is much quicker (depending on what assumptions you make, based on how easily you can produce controlled-$U^{2^k}$). But sometimes, as in the case of order finding, you get down to something that's polynomial in $n$.
The second issue is the application. Yes, it can seem pretty dumb that if you know the eigenvector, you have to make the eigenvector to find the eigenvalue. That, as a general rule, is not how the algorithms that use phase estimation actually work. For example, order finding does not input an eigenstate, it's just that you need the understanding of how it works on eigenstates in order to be able to process the overall function. Much the same is true of other algorithms such as HHL. Indeed, these algorithms use phase estimation to decompose an input state in terms of the eigenbasis even though you don't know what that basis is.