As shown e.g. in Watrous' book (Proposition 6.6, page 314), a separable state $\rho$ can always be written as a convex combination of at most $\mathrm{rank}(\rho)^2$ pure, separable states.
More precisely, using the notation in the book, any separable state $\xi\in\mathcal X\otimes\mathcal Y$ can be decomposed as $$\xi = \sum_{a\in\Sigma} p(a) \, x_a x_a^*\otimes y_a y_a^*,\tag1$$ for some probability distribution $p$, sets of pure states $\{x_a: a\in\Sigma\}\subset\mathcal X$ and $\{y_a: a\in\Sigma\}\subset\mathcal Y$, and alphabet $\Sigma$ with $\lvert\Sigma\rvert\le \mathrm{rank}(\xi)^2$. This is shown by observing that $\xi$ is an element of the real affine space of hermitian operators $H\in\mathrm{Herm}(\mathcal X\otimes\mathcal Y)$ such that $\mathrm{im}(H)\subseteq\mathrm{im}(\xi)$ and $\mathrm{Tr}(H)=1$. This space has dimension $\mathrm{rank}(\xi)^2-1$, and thus from Carathéodory we get the conclusion.
Consider the case of the totally mixed state in a space $\mathcal X\otimes\mathcal Y$ with $\mathrm{dim}(\mathcal X)=d, \mathrm{dim}(\mathcal Y)=d'$. For this state, $\xi\equiv \frac{1}{dd'}I = \frac{I}{d}\otimes\frac{I}{d'}$, we have $\mathrm{rank}(\xi)=\lvert\Sigma\rvert=dd'$ for the standard choice of decomposition. Generating random convex combinations of product states I also always find $\lvert\Sigma\rvert=\mathrm{rank}(\xi)$ (albeit, clearly, the numerics doesn't check for the existence of an alternative decomposition with less than ${\rm rank}(\xi)$ components). In the case $\lvert\Sigma\rvert=1$, it is trivial to see that we must also always have $\lvert\Sigma\rvert=\mathrm{rank}(\rho)$.
What are examples in which this is not the case? More precisely, what are examples of states for which there is no alphabet $\Sigma$ with $\lvert\Sigma\rvert\le\mathrm{rank}(\xi)$, such that $\xi=\sum_{a\in\Sigma}p(a)x_a x_a^*\otimes y_a y_a^*$?
The following is the Mathematica snippet I used to generate random convex combinations of product states:
RandomUnitary[m_] := Orthogonalize[
Map[#[[1]] + I #[[2]]&, #, {2}]& @ RandomReal[
NormalDistribution[0, 1], {m, m, 2}
]
];
randomPureDM[dim_] := First@RandomUnitary@dim // KroneckerProduct[#, Conjugate@#] &;
With[{numComponents = 4, bigDim = 10},
With[{
mats = Table[KroneckerProduct[randomPureDM@bigDim, randomPureDM@bigDim], numComponents],
probs = RandomReal[{0, 1}, numComponents] // #/Total@# &
},
Total[probs*mats] // Eigenvalues // Chop
]
]
A related question on physics.SE is What is the minimum number of separable pure states needed to decompose arbitrary separable states?.