This problem can be approached without regards to Kraus representations (even if the motivation is to prove the convexity of entropy) or whether A is a normal matrix or not. Rather, this is a feature of the choice of $\{ U_{j} \}$. In particular, there exists a choice such that their action is to ``coarse-grain'' all the information in a state.
Here's a single qubit example to illustrate my point: consider the set $p_{j} = \frac{1}{4}, U_{j} = \sigma_{j}$ for $j \in \{ 1,2,3,4 \}$, where, $\sigma_{j}$ are the Pauli matrices (with $\sigma_{0} = \mathbb{I}$). Then, its action on a single qubit is,
$$ \sum\limits_{j} p_{j} U_{j} \rho U^{\dagger}_{j} = \frac{1}{4} \left( \mathbb{I} \rho \mathbb{I} + \sigma_{x} \rho \sigma_{x} + \sigma_{y} \rho \sigma_{y} + \sigma_{z} \rho \sigma_{z} \right) = \cdots = \operatorname{Tr}\left( \rho \right) \frac{\mathbb{I}}{2},$$
where the $\cdots$ can be evaluated using the anticommutativity of the Pauli matrices (Hint: use the relation $\sigma_{j} \sigma_{k} \sigma_{j} = - \sigma_{k}$ for $j \neq k$).
Now, since any matrix $A$ can be written as $A = H + iK$ for hermitian matrices $H,K$; and any hermitian matrix $H$ can be written as $H = H_{1} - H_{2}$ for positive semidefinite matrices, you can write $A = H_{1} - H_{2} + i(K_{1} - K_{2})$. Rewriting each of the matrices as $H_{1} = \operatorname{Tr}\left( H_{1} \right) (\frac{1}{\operatorname{Tr}\left( H_{1} \right)} H_{1})$, we have that $\frac{1}{\operatorname{Tr}\left( H_{1} \right)} H_{1}$ is a density matrix and so the above result applies. Continuing this, you'll find, using the linearity of trace, that for the $2 \times 2$ case, the above unitaries give you $\mathrm{Tr}(A) \frac{\mathbb{I}}{d}$.
The generalization to $n \times n$ matrices is left as an exercise to the OP (where you need to find a set of unitaries analogous to the Pauli matrices).
Edit: One way to obtain the result in $d$ dimensions is to use the $d^2$ Heisenberg-Weyl operators (or the finite dimensional representation of the Heisenberg-Weyl algebra). If $X(i)Z(j)$ is the $(i,j)$th operator then, we have, $\frac{1}{d^{2}} \sum_{i, j=0}^{d-1} X(i) Z(j) \rho Z^{\dagger}(j) X^{\dagger}(i)=\frac{\mathbb{I}}{d}$. See, for example, Page 176 of this book.