Why did "job_manager" submit the same job twice and how to fix it?

Question

In order to avoid the problem posted in Why "job_manager" did not submit some of the jobs?, (the current issue is related but quite different, so I think it is more appropriate to ask separately), I divide the job to batches, with each batch having 75*5=375 circuits. Then I use ("item" below contains a list of 375 circuits)

job_set = job_manager.run(item, backend=backend, name='L_5_vqe_qc' , shots = 8192 )

with the hope that "job_mamager" will submit the 5 jobs, each with 75 circuits. However, it turns out that the same job was submitted twice. See the screenshot below, where "L_5_vqe_qc_2" was submitted twice. Is this normal?

Answer 1

Job Manager submits the same job twice because there was a partial failure in the first submit. The latest Qiskit (0.23.2) has a fix that will cancel these phantom jobs earlier so they do not consume the job limit.

Details:

Submitting a job is actually a 2-step process:

1. inform the server a job is to be submitted
2. actually upload the data

If Job Manager receives an error at step 1 (i.e. due to networking issues), it'd retry the step. However, sometimes the original request would actually make it to the server even though it gave an error. In such case you will see the same job twice in the dashboard, but the first instance will be stuck in CREATING and never processed (since step 2 was not performed). The server will eventually cancel the phantom job.

Answer 2

This is not an answer to the question above, but it is a workaround for the issue, and the one posted in Why "job_manager" did not submit some of the jobs?. Essentially, I notice that after we submit the job with

job_set = job_manager.run(item, backend=backend, name='L_5_vqe_qc' , shots = 8192 )

we can continue to do other stuff. Previously I thought that the notebook will be occupied and we have to wait until the result comes back before we can run the subsequent code. Fortunately this is not the case. Thus, we can do the following, where "qclist" is a list, with each item contains 375 circuits. In this way, I think we can fully make use of the available resources in a controlled way.

result_qc = []
for item_ind in range(len(qclist)):
    item = qclist[item_ind]
    
    job_set_list = []

    # This for loop will submit five jobs, each at a time
    # After that, the notebook will proceed to the next for loop
    for i in range(5): 
        job_set_list.append( job_manager.run(item[i*75 : (i+1)*75], backend=backend, shots = 8192 ) )
    
    # This for loop will collect the results
    # We have to wait until the result comes back here
    for i in range(5):
        result_qc.append( job_set_list[i].results().get_counts(0) )
        
print( result_qc )