In the following I'm assuming you're talking about a two-qubit system. You don't explicitly say, but I infer from the calculation you're showing. If not, this can all be generalised easily enough...
I would start by writing
$$
\langle j_2|\langle j_1| U(|0\rangle\!\langle0|\otimes\rho)U^\dagger|j_2\rangle|j_1\rangle=\text{Tr}\left(U(|0\rangle\!\langle0|\otimes\rho)U^\dagger|j_2\rangle\langle j_2|\otimes|j_1\rangle\langle j_1|\right).
$$
If it's not clear to you why that's correct, work backwards - when you take the trace you can sum over any orthonormal basis. Just pick any basis such that $|j_2\rangle|j_1\rangle$ is a member of that basis, so all other elements are orthogonal.
Next, I can use the cyclic properties of the trace to move the $U$.
$$
\text{Tr}\left((|0\rangle\!\langle0|\otimes\rho)U^\dagger|j_2\rangle\langle j_2|\otimes|j_1\rangle\langle j_1|U\right)
$$
Next, define $|\Psi\rangle=U^\dagger|j_2\rangle|j_1\rangle$, so this is the same as
$$
\text{Tr}\left((|0\rangle\!\langle0|\otimes\rho)|\Psi\rangle\langle \Psi|\right)
$$
Now, I can perform the trace by first applying the partial trace on the first qubit, and then the trace on the second qubit:
$$
\text{Tr}\text{Tr}_1\left((|0\rangle\!\langle0|\otimes\rho)|\Psi\rangle\langle \Psi|\right)=\text{Tr}\left(\rho\cdot\text{Tr}_1\left((|0\rangle\!\langle0|\otimes I)|\Psi\rangle\langle \Psi|\right)\right).
$$
So, if we define
$$
M=\text{Tr}_1\left((|0\rangle\!\langle0|\otimes I)|\Psi\rangle\langle \Psi|\right),
$$
then the answer is expressed as $\text{Tr}(M\rho)$.
Finally, we can certainly write (without knowing anything more about it) that
$$
|\Psi\rangle=\alpha|0\rangle|\phi_0\rangle+\beta|1\rangle|\phi_1\rangle,
$$
for an arbitrary pair of single-qubit states $|\phi_0\rangle,|\phi_1\rangle$ and $|\alpha|^2+|\beta|^2=1$, which would mean that
$$
M=|\alpha|^2|\phi_0\rangle\langle\phi_0|.
$$
If you want it to be the more specific form given in the question ($|\alpha|^2=\frac12$), then you need more information about $U$.