QueueEvents does not pick up all completed jobs.

[pcace]

hi there i have a question about QueueEvents.
i am trying to set up concurrent jobs for a calculation. the problem i am having is, that my logic fully works if i have concurrency = 1. When i try to make it work in parallel a strange behaviour occures: only some jobs get noticed by the event "on completed".

the whole thing consists of two jobs within each other:
first there is

this is what i am doing:

in the main app:

// this is the queue for job nr. 1
const queueName = 'wssQueue'
const jobque = new Queue(queueName, connection)

// this is the queue for all jobs inside job nr: 1
const extruderQueueName = 'extrudeQueue'
const extruderJobque = new Queue(extruderQueueName, connection)

const jobReturnValue = await new Promise<String>((resolve, rejects) => {
        wssWorker.on('completed', async job => {
            if (job.name === jobid) {
                console.log('JOB COMPLETED')
                resolve(job.returnvalue)
            }
        })
        wssWorker.on('failed', async job => {
            console.log('JOB FAILED')
            rejects('JOB failed')
        })
    })

the worker(wssWorker) looks like this:

export const wssWorker = new Worker(
    'wssQueue',
    async job => {
        const { mesh, center, polygons, volume } = await generateMesh(geojson, t_srs, workingfolder, skipMesh)

        // ... do some magic

        return {
            uuid: uuid,
            vertices: skipMesh ? [] : vertices,
            indices: skipMesh ? [] : indices,
            projectCenter: center,
            volume: volume,
            contourlines: contourlines
        }
    },
    {
        autorun: true,
        // here i set the concurrency, 1 works reliably, everything above does not (or only sometimes...)
        concurrency: 1,
        connection: {
            host: process.env.REDIS_SERVER?.toString(),
            port: parseInt(process.env.REDIS_PORT ? process.env.REDIS_PORT.toString() : '6379'),
            password: process.env.REDIS_PASSWORD?.toString(),
        },
    }
).setMaxListeners(30)

inside the generateMesh function i do need to run some more jobs. but here is the problem what i dont understand: when i run the first job without concurrency or with concurrency = 1 then all jobs within generateMesh run completely. if i run this with concurrency of 2 or more the jobs inside generateMesh sometimes do not finish or better do not show up. the higher the concurrency value is, the more unlikely it is, that all jobs inside gererateMesh return their returnvalue.

here is the job part from the generateMesh function:

const extruderQueueName = 'extrudeQueue'
const extruderJobque = new Queue(extruderQueueName, connection)
const queueEvents = new QueueEvents('extrudeQueue');

// ... some magic...

// highZ (max Z value) is set, so we can now go through extrusion
// and add all extrusion jobs to the job queue
const queues: Job[] = []
while (distance <= highZ + 1) {
    queues.push(await extruderJobque.add(
        `${distance}`,
        {
            folder, polygonFile, distance, skipMesh, weightsFile
        },
        {

            removeOnComplete: true,
            removeOnFail: true,
        }
    ))

    distance++
}

// lets now listen to the completed jobs and resolve the Promise when 
// no jobs are active anymore
await new Promise((resolve, reject) => {
    queueEvents.on('completed', async job => {
        if (queues.map((id) => id.id).includes(job.jobId)) {
            const returnValue = job.returnvalue as any
            volume = returnValue.volume
            polygons.push(...returnValue.polygons)
        }
        let jobCounts = await extruderJobque.getJobCounts();
        console.log(jobCounts)
        if (jobCounts.active === 0 && jobCounts.waiting === 0) {
            resolve("")
        }
    })
    extrudeWorker.on('failed', async job => {
        console.log('JOB FAILED')
        reject("")
    })
})

if anyone could spot if this is an expected behavior or possibly a bug (possibly not.. i am very new to bullmq ;) ) or what is going on here please please let me know!!!

thank you so much!!

[manast]

It is difficult to see what could be wrong as the code is quite large. I recommend that you instead write a minimal case that works and then continue adding functionality until you spot the issue.