The Bernstein-Vazirani problem:
Let $f$ be a function from bit strings of length $n$ to a single bit,
$$f: \{ 0, 1\}^n \to \{0, 1\} $$
thus all input bit strings $x \in \{0,1\}^n$. There exists a secret string $s \in \{0,1\}^n$ such that
$$ f(x) = x\cdot s$$
where $\cdot$ denotes the inner product mod 2. Find $s$ by querying $f$ as a few times as possible.
This problem can be solved using 1 query using QFT. The algorithm construction only uses an $X$ gate, Hadamard ($H$) gates, and $CNOT$ gates.
Now, according to the Gottesman-Knill theorem, quantum algorithms which utilise only the operations belonging to a certain restricted set (Clifford group $C_n$, which is nothing but the normalizer of the Pauli group $P_n$) are efficiently simulable classically.
This implies that the quantum circuit we construct including the oracle can be implemented efficiently classically. So why do we say this problem can be solved exponentially faster with a quantum computer?
I understand that if you want to develop a classical algorithm then you do have to query the oracle $N$ times... but can't we just implement the entire circuit classically in polynomial time based on the Gottesman-Knill theorem.
What am I missing here? Thank you!