I apologise in advance. This is a rough and hand-waivy answer.
You can give "information-theoretic" lower bounds by noting that the measurements can be seen as a linear map $M$ from quantum states to outcome probabilities $y$. For instance, if you have a POVM $(E_i)_{i=1,\dots,N}$, then the probability vector is $y = \sum_i \mathrm{tr}(E_i \rho) e_i$, where $e_i$ is the standard basis of $\mathbb{R}^N$.
Tomography is about finding an approximate solution to the equation
$$y = Mx$$
where $x$ is a description of your state. Without further assumptions, this only works if $M$ has full rank, in particular your POVM has to have size $N=\Omega(d^2)$ when $d$ is your Hilbert space dimension.
However, if your state has low rank $r$, the problem can be solved more efficiently.
For instance, this occurs when $\rho$ is a pure state (rank 1).
In this case, a lower bound is $N=\Omega(rd)$ which can be achieved e.g. with ideas from compressed sensing and low-rank matrix recovery.
However, the number of measurement settings is not the only important quantity in tomography. Sample complexity (i.e. the number of copies of your state), robustness to measurement errors, post-processing time etc. are important aspects as well.
Low rank assumptions can also be used to reduce the sample complexity.
There's a lot of literature on this ... with different flavors. You can have a look at the introduction and the references in