For the review, you can have a look at this tutorial by Kliesch and Roth.
For the technical points on RB, I think salix's answer captures the basic idea. However, I have some additional remarks, since I am not agreeing with everything they said.
The inversion is costly. In standard Clifford RB, $n$-qubit Clifford unitaries are used which have to be compiled into the native gate set. This means that the circuits produced by a standard RB protocol with sequence length $m$ have gate count $O(m n^2/\log(n))$. In practise, this is the reason why standard Clifford RB is basically not used for more than $n=2$ qubits: The circuits are too long, thus the effective noise is way to strong and your signal decays too fast to be estimated. What people mostly do, is that they apply random generators directly (instead of $n$-qubit Cliffords. This is ok, because you converge to a 2-design). In this way, you have a better control over the circuit depth. However: If you're still doing the inversion, then this gate, as a product of sufficiently many generators, will be a global $n$-qubit unitary and you agaub need a long circuit to implement it.
Leaving out the inversion. Yes, you can leave out the inversion, good observation. Instead, you can effectively invert "by post-processing". As salix pointed out, this is what XEB does, however, here it is not done for Cliffords, but for rather general unitaries, which is why the post-processing is very costly. With Cliffords, this would be much more efficient. In the RB context, this is described for instance in Sec. VIII of Helsen et al.: "A general framework for randomized benchmarking", and more prominently in Helsen et al. "Estimating gate-set properties from random sequences"
RB as a benchmark.
In an idealized scenario (namely gate-independent noise), the decay rate produced by a standard Clifford randomized benchmarking protocol is directly linked to the average gate fidelity of the Clifford unitaries, averaged over the group. There are also interleaved RB schemes to estimate the average gate fidelity of a fixed target unitary. However, in a more realistic scenario, there are certain problems in interpreting the decay rate as average gate fidelity. First, there is technically no direct relation. Second, and more severly, there is a "gauge freedom" in RB, meaning that the "noise" in the random sequences in actually not well-defined, but somewhat "gauge-dependent". The rates, however, are not. Hence, it can be problematic to relate those two concepts. This is discussed in Proctor et al., Wallman, Merkel et al., and Helsen et al.. IMO, this problem is not really resolved. I think it would be more responsible to see the decay rate as an independent benchmarking quantity which reflects the quality of your gates -- and not connect it to AGF (cf. benchmarks like 3DMark or whatever for classical computers).