IBMBackendValueError: number of circuits exceeds the maximum for this backend, 300

Question

I am trying to run the QAOA on the IBMQ device. The QAOA circuit consists of 6 qubits, two layers. The number of total trainable parameters for the QAOA circuit is 14. The toy code is as follows:

dev = qml.device('qiskit.ibmq', wires=args.dataset.n_node, backend='ibm_kyoto', ibmqx_token=token)
circuit = QAOA(dev)
opt_b = qml.AdamOptimizer(args.lr_qaoa)
param_b, loss = opt_b.step_and_cost(lambda param_b: circuit(param_b, circuit.param_c), circuit.param_b)

After running the code, it successfully ran on the IBMQ backend in the beginning by checking the IBMQ dashboard of my account. However, it soon raised the error qiskit_ibm_provider.exceptions.IBMBackendValueError: Number of circuits, 418 exceeds the maximum for this backend, 300). The detailed pennylane version information is listed in the following:

Name: PennyLane
Version: 0.35.1
Summary: PennyLane is a cross-platform Python library for quantum computing, quantum machine learning, and quantum chemistry. Train a quantum computer the same way as a neural network.
Home-page: https://github.com/PennyLaneAI/pennylane
Author:
Author-email:
License: Apache License 2.0
Location: /home/yqia7342/qml/lib/python3.10/site-packages
Requires: appdirs, autograd, autoray, cachetools, networkx, numpy, pennylane-lightning, requests, rustworkx, scipy, semantic-version, toml, typing-extensions
Required-by: PennyLane-qiskit, PennyLane_Lightning

Platform info:           Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Python version:          3.10.12
Numpy version:           1.26.4
Scipy version:           1.12.0
Installed devices:
- default.clifford (PennyLane-0.35.1)
- default.gaussian (PennyLane-0.35.1)
- default.mixed (PennyLane-0.35.1)
- default.qubit (PennyLane-0.35.1)
- default.qubit.autograd (PennyLane-0.35.1)
- default.qubit.jax (PennyLane-0.35.1)
- default.qubit.legacy (PennyLane-0.35.1)
- default.qubit.tf (PennyLane-0.35.1)
- default.qubit.torch (PennyLane-0.35.1)
- default.qutrit (PennyLane-0.35.1)
- null.qubit (PennyLane-0.35.1)
- lightning.qubit (PennyLane_Lightning-0.35.1)
- qiskit.aer (PennyLane-qiskit-0.35.1)
- qiskit.basicaer (PennyLane-qiskit-0.35.1)
- qiskit.ibmq (PennyLane-qiskit-0.35.1)
- qiskit.ibmq.circuit_runner (PennyLane-qiskit-0.35.1)
- qiskit.ibmq.sampler (PennyLane-qiskit-0.35.1)
- qiskit.remote (PennyLane-qiskit-0.35.1)

Could you please provide some advice on how to solve this issue?

Answer 1

I think you need to restrict the maximal number of function call by qml.AdamOptimizer(args.lr_qaoa) to be below 300, make it 290. Executing too many circuits on real HW would block it for a long time, IBM HW does only ~1k shots per second (give or take), so 300 circuits is already 5 minutes on the HW.

Note1: A trick you may use is to pre-train the ML model on ideal simulator w/Pennylane and for IBM load initial weights.

Note2: What is your QAOA() circuit CX-count after transpiling to IBM HW (which does not have all-to-all connectivity)? If it is more than 20-40 CX a chance for success are low due to the finite gate fidelity: $0.98^{40}=0.5$.

Answer 2

Right, I also see no max explicit parameter capping the number of iterations in qml.AdamOptimizer(). You can always solve it 'manually' by adding a counter of function call in your cost function and aborting the training loop if this counter exceeds 290

It would take ~10 lines of extra code and ... it is a lost cause for IBM HW - it is still too noisy (but IBM gets better every year, especially Heron family) . But you can gain a skill how to do it using a simulator.

And there is one more trouble you need to address: qml.AdamOptimizer() will not work with shot-based backend (HW is always shot-based, no state vector :). The only one I found would work is qml.SPSAOptimizer() and this one requires you provide max number of steps (so it knows how to reduce the learning rate per step). Since you know what is your batch size (aka step size) you can pre-compute maxsteps to result with max number of circuits to be 290