Note that
$$[(a^\dagger)^n,a] = -n(a^{\dagger})^{n-1},
\qquad [(a^\dagger)^n a^m,a] = -n (a^\dagger)^{n-1}a^m,
\qquad [a^n,a]=0.$$
Consider an arbitrary function of the mode operators, that we assume be written in normal formal:
$$f(a,a^\dagger) = \sum_{n,m=0}^\infty c_{n,m} (a^\dagger)^n a^m.$$
We know that
$$e^{f(a,a^\dagger)}a e^{-f(a,a^\dagger)} = \sum_{k=0}^\infty \frac{1}{k!}\operatorname{ad}(f(a,a^\dagger))^k\cdot a,$$
where $\operatorname{ad}(A)^k\cdot B \equiv [\underbrace{A,...,[A}_k,B]\cdots]$
denotes the operation of taking the repeated commutator of $A$ with $B$.
For example, for $k=2$, $\operatorname{ad}(A)^2\cdot B\equiv [A,[A,B]]$.
We know that
$$[f(a,a^\dagger),a] = -\sum_{n,m\ge 0}c_{n,m} n (a^\dagger)^{n-1} a^m,$$
$$\operatorname{ad}(f(a,a^\dagger))^k\cdot a
= (-1)^k \sum_{n,m\ge0} c_{n,m} \frac{n!}{(n-k)!} (a^{\dagger})^{n-k}a^m.$$
Thus
$$e^{f(a,a^\dagger)}a e^{-f(a,a^\dagger)}
= \sum_{n,m\ge0 } c_{n,m} (a^\dagger-1)^n a^m
= f(a, a^\dagger-1). $$
This shows you explicitly that properties of $f$ such as its being quadratic permain in $e^f a e^{-f}$. Thus, in particular, you get your result by choosing $f=-iH t$.