Given a pair of entangled particles: $\alpha \vert HL \rangle + \beta \vert VR \rangle$
where:
Is it possible to measure one of the particles in $L$ and $R$ basis using a polarizer (without any normalization etc.)? Or does quantum mechanics forbids it? And if yes, does the probability of it being $L$ or $R$ given by $\alpha^2$ and $\beta^2$ respectively?
The overall state must have unit norm, i.e. for $|\psi\rangle=\alpha |HL\rangle+\beta|VR\rangle$ we should have $\langle\psi|\psi\rangle=1$. If I understood the question, by your assumption $\langle H |V\rangle=0$, and $\langle L |R\rangle=0$, hence $$\langle\psi|\psi\rangle=|\alpha|^2\langle H|H\rangle \langle L|L\rangle +|\beta|^2\langle V|V\rangle \langle R|R\rangle=1$$
Assuming further that $|H\rangle$ and $|V\rangle$ have unit norm and $|\beta|^2=1-|\alpha|^2$ this reduces to $$|\alpha|^2\langle L|L\rangle +(1-|\alpha|^2)\langle R|R\rangle=1$$ Although for states $|L\rangle$ and $|R\rangle$ of unit norm this constraint is satisfied, they do not need to have unit norm, but only subject to satisfy the constraint.
With all that said, defining normalized versions of $|R\rangle$, $|L\rangle$ and adjusting the coefficients $\alpha,\beta$ correspondingly would probably make the analysis much simpler.
----------(addition)
If the vectors $|L\rangle, |R\rangle$ are not normilized, then coefficients $|\alpha|^2, |\beta|^2$ in decomposition $|\psi\rangle=\alpha |L\rangle+\beta |L\rangle$ do not correspond to measurement probabilities. Consider a simple example $|L\rangle = \frac1{\sqrt{2}}|0\rangle, |R\rangle = \sqrt{\frac{3}{2}}|1\rangle$ and $$|\psi\rangle=\frac{1}{\sqrt{2}}|L\rangle+\frac{1}{\sqrt{2}}|R\rangle=\frac12|0\rangle+\frac{\sqrt{3}}{2}|1\rangle$$ The actual measurement probabilities are $\frac14,\frac34$. It is not clear to me though why go into the trouble of working with non-normilized states at all in this context.
