In this proof, we assume that the trace distance between $\rho$ and $\sigma$ is upper-bounded by $\frac{1}{\mathrm{e}}<\frac12$.
As mentioned by glS in the comments, the trace distance is lower-bounded by half the sum of the $\left|r_i-s_i\right|$. Suppose that for some $i$, we have $\left|r_i-s_i\right|>\frac12$. At this point, half this sum is already $\frac14$. Intuitively, since the sum of the $r_i$ and of the $s_i$ are both equal to $1$, there got to be a term where the difference between the twos will be too large for the trace distance to be less than $\frac{1}{\mathrm{e}}$. There is surely an elegant way to prove it, but I didn't find it, so here's a reasoning by induction.
To be more formal, the statement we want to prove is the following one:
Let $\left(r_i\right)_i$ and $\left(s_i\right)_i$ be two sequences of length $d$ such that:
- $\sum_ir_i=\sum_js_i=1$
- There is an index $j$ such that $\left|r_j-s_j\right|>\frac12$
Then $\frac12\sum_i\left|r_i-s_i\right|>\frac12$.
First, let us consider the qubit case, that is $d=2$. Let us suppose without loss of generality that $r_0=\frac12+s_0+\varepsilon$, with $0<\varepsilon\leqslant\frac12$. Then we necessarily have $r_1=\frac12-\varepsilon-s_0$, since $r_0+r_1=1$. We now replace $s_0$ by $1-s_1$ to find that $r_1=-\frac12-\varepsilon+s_1$, which finally gives us a lower-bound for the trace distance of $\frac12+\varepsilon$, which is strictly larger than $\frac12$.
Let us now assume that the proposition is true for some $d$ and let us show it for $d+1$. Since $d+1\geqslant3$, we know that there are at least two couples $\left(r_x,s_x\right)$ and $\left(r_y,s_y\right)$ such that $r_x\geqslant s_x$ and $r_y\geqslant s_y$ or the other way around. In this case, we can regroup these two couples into a single one $\left(r_x+r_y,s_x+s_y\right)$ without changing the computation of the lower-bound. These new sequences are of length $d$ and sum to $1$, and we still have the existence of an index on which the absolute value of the difference of the terms is larger than $\frac12$, we can thus use our assumption and conclude that the trace distance is necessarily lower-bounded by $\frac12$.
There is probably a (way) more elegant way to show this property, but I think that's rigorous. All in all, your counterexample doesn't apply since the trace distance between the density matrices that you're considering is equal to $\frac34$, which is larger than $\frac{1}{\mathrm{e}}$.