Let’s say I have a problem that can only be solved with a trapdoor, but regardless of whether the trapdoor is right or wrong, you can’t check if you have found the solution to the problem. To what class of complexity theory does such a problem belong? And more importantly, can a quantum computer solve such a problem?
Ok, now I understand what you mean. You want something like the one-time pad, where for any message of the correct length there is an encryption key that gives you that message. It is not possible to check which of the messages is the correct decryption, you need to have the correct encryption key.
It is not possible to solve this problem at all, with a quantum computer or anything. This is proven rigorously in information theory: the encrypted text has zero mutual information with the source text.
As for the complexity class, it doesn't have one. Complexity theory deals with problems where all the necessary information is available.