There is a geometric interpretation that you certainly can take seriously, but the geometry that you get is not as clean as you might have hoped.
Trace distance between operator states is an example of a Banach norm on a vector space $V$. The rules for such a norm are that $||v|| > 0$ when $0 \ne v \in V$, $||\lambda v|| = |\lambda|\cdot||v||$ for $\lambda$ as scalar, and $||w+v|| \le ||v||+||w||$. (There is also the condition that Cauchy sequences converge. This is automatic for any norm in finite dimensions and also happens to be true for density operators on an infinite-dimensional Hilbert space.) Any Banach norm can be thought of as a geometry of some kind. The norm $||v||$ can be thought as the length of $v$; $||v-w||$ can be thought of as the distance from $v$ to $w$, i.e., it makes $V$ a metric space; and the unit ball of the norm is some convex body that is analogous to the unit ball in Euclidean geometry. However, general Banach norm doesn't usually come from an inner product. When it doesn't, the unit ball isn't round and the geometry that you get isn't Euclidean.
The norm and metric in this case are defined on density operators with trace 1. This is not technically a vector space; it is a translate of the vector space of density operators with trace 0. You have to be careful about that in calculations, but this is close enough for our purposes. For a $d$-state qudit, this vector space has dimension $d^2-1$. When $d=2$, as you say, the unit ball of the norm is a round ball and the geometry is Euclidean. When $d > 2$, first of all the Bloch ball itself is replaced by a more complicated state region of positive semi-definite operators with trace 1. Both the state region itself and the unit ball of the trace norm are somewhat nondescript convex bodies. Since they are invariant under conjugation by $U(d)$, they have enough symmetry to have an unambiguous center point; but they don't have enough symmetry to be round. They are not smooth convex bodies and they are not polytopes with corners either. Rather, they have ridges of several different dimensions and otherwise they are smooth.
There is in fact an accompanying Euclidean geometry on density operators given by the Hilbert-Schmidt norm, but the trace norm is not that. The fact that the two norms are equivalent in the Bloch ball for qubits is a coincidence induced by symmetry.
You can get a simplified picture if you restrict to diagonal density operators. Diagonal density operators are equivalent to probability distributions, and in this case the trace distance is also called total variation distance. In this restricted geometry, the state region and the unit ball of trace distance are both polytopes. The state region is a simplex with $d+1$ corners, while the unit ball of trace distance is a more complicated polytope with $d^2+d$ corners. For instance if you have a classical trit, the state region is an equilateral triangle and the unit ball of trace distance is a regular hexagon with three sides parallel to those of the triangle. Since every density operator is diagonal in some basis, geometric slices representing the states of a classical digit are a helpful glimpse of the larger geometry of states of a qudit.