In quantum and classical algorithms, we often need to do "function calls." Quantum algorithms such as Grover's algorithm or the Deutsch–Jozsa algorithm can take a fewer number of function calls than their classical counterparts, and this is often argued as a reason why these algorithms are more efficient. However, I see these arguments as having a gap.
My main question is, how do we know a "quantum function call" is worth the same amount of time as a "classical function call?"
For example, in an unstructured search problem, we have a function $f$ such that $f(x) = 0$ if $x$ is not a solution to our search problem and $f(x) = 1$ if $x$ is a solution to our search problem. In Grover's algorithm, we utilize an oracle $\hat{O}_{f}:|x\rangle\mapsto (-1)^{f(x)}|x\rangle$. Now we could imagine a scenario where an implementation of $\hat{O}_{f}$ that would require $N$ seconds. Meanwhile a classical call to $f$ for a single item might take $1$ second. In this scenario, Grover's algorithm would take $O(\sqrt{N})$ oracle calls, but the entire search would take $O(N\sqrt{N})$ seconds to complete, which is not faster than $O(N)$ seconds classically.