TL;DR: Different computers.
One of the main advantages behind quantum computers is that they are reversible computers, however there are irreversible functions, that classical computers can compute, that are difficult to port to quantum computers.
An example of a function that a classical algorithm can solve far better would be the modulo operator, e.g. $15\pmod7~=~1\pmod7$, in a classical computer we can just 'throw away' everything above the 3'rd bit, after calculating the modular operation.
Assuming we would want to implement said function in the quantum world, then we cannot just simply throw away qubits. (See $^1$ for a slight detail). If we do not restrict the input, and input, let's say $15$ and $22$, then the result should be $1$, but if two inputs have the same output, then it is clearly not reversible. And so, we need extra qubits (ancillas) or extra input assumptions to implement this 'irreversible' function. At this point it depends on your definition of 'fast', but it is likely safe to assume that both in temporal time and the amount of operations, this will be out scaled by the classical algorithm.
One can also think about it like this, assuming one were to implement said modular function, you could not use it in superposition, since you would need to measure or reset some qubits to make all the higher bits be $0$ again, hence collapsing a superposition and the main way to get any form of 'quantum advantage'.
Just for the sake of completeness, the input assumption done to implement modular exponentiation in Shor's Algorithm is that the input state is in $[1, p-1]$, this makes the creation of the operator 'easier'. Throughout the algorithm the modular operations are kept in $[1, p-1]$.
So where is the advantage in quantum computation?
The clear advantage is that we are no longer computing in the discrete space, with access to the Hilbert space, we can compute with the reals and hence complex numbers, though only to a certain degree, since we can only do so in superposition and can only extract 1 state from said superposition. (This is a very broad oversimplification)
This allows, e.g. Shor's Algorithm to solve certain problems exponentially faster than a classical computer.
What about quantum speedups for classical algorithms?
There are in-fact a wide-range of quantum speedups for classical algorithms, often classed as 'Grover's algorithms', but not only limited to them. The classic example is searching in an unstructured list. Assuming you have your list encoded in a quantum computer, then you can search for an element in it, in $O(\sqrt{N})$ time. And you can apply this to a wide range of search and oracle problems to gain square root speedups. This means that there are problems that if we scale them into very large numbers, then the quantum computer does beat a classical computer in the amount of operations, this also highly depends on the type of problem: decision problem, functional problem and if said problem can be converted into a search problem one can apply Grover's on.
The caveat on (some) quantum speedups
They assume you have your input encoded in a quantum computer. If we wanted to search through a classically stored list, then we would first need to encode that list in a quantum computer, which would take $O(N)$ time, you might as well search through the list at that point. The real question is then, can you encode the data you want to search through in less than $O(N)$ time, if you are e.g. searching over all the numbers from $0 \dots 2^n-1$, then you can, other cases may not be so easy.
$^1$:
Theoretically, this is not a hard constraint, if we do not take advantage of superposition (then we can simply reset the qubits), however then we have no advantage at all over classical computation (it would be identical and likely slower on a quantum computer)
This is my first attempt at answering questions in the QCSE, some answers might not be completely accurate, feel free to correct me.