Running multiple jobs in parallel?

[arpadgabor]

Hi, as the title suggests, is there any way to configure code build to run jobs in parallel? Seems like every job is run sequentially on CodeBuild side. Do we need to setup multiple providers and pass those in manually or are there any more automated ways?

Great project by the way, thanks for the work put into this!

[kichik]

It should be running in parallel. Are you maybe hitting some quota on your account?

Here's an example of it running in parallel on my account. I don't recall changing any quotas. I am using the default CodeBuild compute size.

[arpadgabor]

Hmmm I'll take a look later today, but I tried adding 2 jobs in parallel to my workflow and only 1 ran at a time. I'll check again later today and let you know if I find anything.

[arpadgabor]

So I've tested now, and it looks like the jobs do spawn, but for some reason they don't all start at the same time even though it's the same workflow. What I notice is that it takes about a minute for an instance to cold start, and only after that will another instance startup. I'll have to test with longer jobs and see if it's the same.

[kichik]

In the job detail page there is a tab called "Phase details" that might help.

[arpadgabor]

Yeah... so all started at Oct 28, 2022 10:40 AM but:

• Job 1 was queued for 53 secs
• Job 2 was queued for 123 secs
• Job 3 was queued for 201 secs

All other timings are the same (except for a margin of error of about ~1s in some metrics). Found this on reddit, someone asked for a lift on the quota even though the limit is 60 and helped...

[arpadgabor]

This is in more detail... I guess all of the jobs are running on the same runner so it cannot run them in different processes.

[kichik]

I'm not sure why the queue holds you up here. For me all three go immediately from queued to provisioning. Are you on us-east-1? Do you have other projects running at the same time? Maybe you're running out of some secondary resource like ENIs or IPs? Does CloudTrail have any hints?

AWS Support might be better suited to help here.

If you want to avoid the issue, you can try the Fargate provider instead.

[arpadgabor]

Thanks for the response! So looks like this should indeed work. I'm on eu-west-1, but I'll check with us-east-1 and/or contact support if nothing else works. Will update the discussion here if I have any news for anyone that might experience the same problem. Thanks!