You cannot efficiently recover the absolute values of the amplitudes, but if you allow for arbitrary many samples, then you can estimate them to whatever degree of accuracy you like.
More specifically, if the input state is a single photon in each of the first $n$ modes, and one is willing to draw an arbitrary number of samples from the output, then it is in principle possible to estimate the permanent of $A$ to whatever degree of accuracy one likes, by counting the fraction of the times the $n$ input photons come out in the first $n$ different output ports.
It is to be noted though that this does not really have much to do with BosonSampling, as the hardness result holds in the regime of the number of modes much larger than the number of photons, and it's about the efficiency of the sampling.
BosonSampling
I'll try a very brief introduction to what boson sampling is, but it should be noted that I cannot possibly do a better job at this than Aaronson himself, so it's probably a good idea to have a look at the related blog posts of his (e.g. blog/?p=473 and blog/?p=1177), and links therein.
BosonSampling is a sampling problem. This can be a little bit confusing in that people are generally more used to think of problems having definite answers.
A sampling problem is different in that the solution to the problem is a set of samples drawn from some probability distribution.
Indeed, the problem a boson sampler solves is that of sampling from a specific probability distribution. More specifically, sampling from the probability distribution of the possible outcome (many-boson) states.
Consider as a simple example a case with 2 photons in 4 modes, and let's say we fix the input state to be $(1,1,0,0)\equiv|1,1,0,0\rangle$ (that is, a single photon in each of the two first two input modes).
Ignoring the output states with more than one photon in each mode, there are $\binom{4}{2}=6$ possible output two-photon states:
$(1,1,0,0), (1,0,1,0), (1,0,0,1), (0,1,1,0), (0,1,0,1)$ and $(0,0,1,1)$.
Let us denote for convenience with $o_i, i=1,.,6$ the $i$-th one (so, for example, $o_2=(1,0,1,0)$).
Then, a possible solution to BosonSampling could be the series of outcomes:
$$o_1, o_4, o_2, o_2, o_5.$$
To make an analogy to a maybe more familiar case, it's like saying that we want to sample from a Gaussian probability distribution.
This means that we want to find a sequence of numbers which, if we draw enough of them and put them into a histogram, will produce something close to a Gaussian.
Computing permanents
It turns out that the probability amplitude of a given input state $|\boldsymbol r\rangle$ to a given output state $|\boldsymbol s\rangle$ is (proportional to) the permanent of a suitable matrix built out of the unitary matrix characterizing the (single-boson) evolution.
More specifically, if $\boldsymbol R$ denotes the mode assignment list${}^{(1)}$ associated to $|\boldsymbol r\rangle$, $\boldsymbol S$ that of $|\boldsymbol s\rangle$, and $U$ is the unitary matrix describing the evolution, then the probability amplitude $\mathcal A(\boldsymbol r\to\boldsymbol s)$ of going from $|\boldsymbol r\rangle$ to $|\boldsymbol s\rangle$ is given by
$$\mathcal A(\boldsymbol r\to\boldsymbol s) =
\frac{1}{\sqrt{\boldsymbol r!\boldsymbol s!}} \operatorname{perm} U[\boldsymbol R|\boldsymbol S],
$$
with $U[\boldsymbol R|\boldsymbol S]$ denoting the matrix built by taking from $U$ the rows specified by $\boldsymbol R$ and the columns specified by $\boldsymbol S$.
Thus, considering the fixed input state $|\boldsymbol r_0\rangle$, the probability distribution of the possible outcomes is given by the probabilities
$$p_{\boldsymbol s} = \frac{1}{\boldsymbol r_0! \boldsymbol s!} \lvert
\operatorname{perm}U[\boldsymbol R|\boldsymbol S]
\rvert^2.$$
BosonSampling is the problem of drawing "points" according to this distribution.
This is not the same as computing the probabilities $p_s$, or even computing the permanents themselves.
Indeed, computing the permanents of complex matrices is hard, and it is not expected even for quantum computers to be able to do it efficiently.
The gist of the matter is that sampling from a probability distribution is in general easier than computing the distribution itself.
While a naive way to sample from a distribution is to compute the probabilities (if not already known) and use those to draw the points, there might be smarter ways to do it.
A boson sampler is something that is able to draw points according to a specific probability distribution, even though the probabilities making up the distribution itself are not known (or better said, not efficiently computable).
Furthermore, while it may look like the ability to efficiently sample from a distribution should translate into the ability of efficiently estimating the underlying probabilities, this is not the case as soon as there are exponentially many possible outcomes.
This is indeed the case of boson sampling with uniformly random unitaries (that is, the original setting of BosonSampling), in which there are $\binom{m}{n}$ possible $n$-boson in $m$-modes output states (again, neglecting states with more than one boson in some mode). For $m\gg n$, this number increases exponentially with $n$.
This means that, in practice, you would need to draw an exponential number of samples to even have a decent chance of seeing a single outcome more than once, let alone estimate with any decent accuracy the probabilities themselves (it is important to note that this is not the core reason for the hardness though, as the exponential number of possible outcomes could be overcome with smarter methods).
In some particular cases, it is possible to efficiently estimate the permanent of matrices using a boson sampling set-up. This will only be feasible if one of the submatrices has a large (i.e. not exponentially small) permanent associated with it, so that the input-output pair associated with it will happen frequently enough for an estimate to be feasible in polynomial time. This is a very atypical situation, and will not arise if you draw unitaries at random. For a trivial example, consider matrices that are very close to identity - the event in which all photons come out in the same modes they came in will correspond to a permanent which can be estimated experimentally. Besides only being feasible for some particular matrices, a careful analysis of the statistical error incurred in evaluating permanents in this way shows that this is not more efficient than known classical algorithms for approximating permanents (technically, within a small additive error) ${}^{(2)}$.
Columns involved
Let $U$ be the unitary describing the one-boson evolution.
Then, basically by definition, the output amplitudes describing the evolution of a single photon entering in the $k$-th mode are in the $k$-th column of $U$.
The unitary describing the evolution of the many-boson states, however, is not actually $U$, but a bigger unitary, often denoted by $\varphi_n(U)$, whose elements are computed from permanents of matrices built out of $U$.
Informally speaking though, if the input state has photons in, say, the first $n$ modes, then naturally only the first $n$ columns of $U$ must be necessary (and sufficient) to describe the evolution, as the other columns will describe the evolution of photons entering in modes that we are not actually using.
(1) This is just another way to describe a many-boson state. Instead of characterizing the state as the list of occupation numbers for each mode (that is, number of bosons in first mode, number in second, etc.), we characterize the states by naming the mode occupied by each boson.
So, for example, the state $(1, 0, 1, 0)$ can be equivalently written as $(1, 3)$, and these are two equivalent ways to say that there is one boson in the first and one boson in the third mode.
(2): S. Aaronson and T. Hance. "Generalizing and Derandomizing Gurvits's Approximation Algorithm for the Permanent". https://eccc.weizmann.ac.il/report/2012/170/