Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CircleCI restrained their build time again. Now a job is limited to 1 hour. Make kung fu needed #1058

Closed
tlaurion opened this issue Nov 16, 2021 · 8 comments

Comments

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 16, 2021

Not sure what to do from here.

  • GitlabCI was too restrictive in total build time which was 2 hours if I recall well.
  • CircleCI now kills job after consuming 1 hour of build time with: "context deadline exceeded"

Last failed attempt on my instance https://app.circleci.com/pipelines/github/tlaurion/heads/847/workflows/518245a6-5802-4854-888a-6e44bbc7d110

@MrChromebox raised the issue.

What now? Should we raise money to get paid plan on CircleCI?
@MrChromebox @jans23?

@tlaurion
Copy link
Collaborator Author

So confused by this https://circleci.com/blog/building-open-source-projects-on-circleci/

Does that mean that osresearch/heads and pull requests only could receive free open source tier and unlimited build time? I have found nothing public and CircleCI doesn't answer.

@tlaurion
Copy link
Collaborator Author

Project is suspended on osresearch/heads, will open ticket

@tlaurion
Copy link
Collaborator Author

tlaurion commented Nov 25, 2021

This information states to use large ressource in free tier.
https://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only-

Implemented under tlaurion@2f803bb and building happening under https://app.circleci.com/pipelines/github/tlaurion/heads/851/workflows/b30064d1-1b6f-4cc8-a879-cd7fe6d02b84/jobs/2343/parallel-runs/0/steps/0-103

The documentation doesn't define what a build is under new limitation "builds on the Free plan are limited to a maximum duration of 1 hour."

If the "build" time is for the total pipeline, then we are now out of the free tier for sure. If builds means steps, then we can still optimize this a bit, and change steps and workspaces and caching to match the new limitation.

Will come back at documenting this once we have result of https://app.circleci.com/pipelines/github/tlaurion/heads/851/workflows/b30064d1-1b6f-4cc8-a879-cd7fe6d02b84/jobs/2343/parallel-runs/0/steps/0-103

@tlaurion
Copy link
Collaborator Author

tlaurion commented Nov 25, 2021

The good news is that the new limit of 1h is for a "job". This just means we need to decouple things more (musl-cross-make, coreboot.buildstack and boards builds).

In current CircleCI Heads usage, this means that one hour limit needs to be able to reuse workspace, build something, and be able to create a workspace archive to be passed to another job which will reuse what was built.

The strategy here is to split what takes the most time into smaller tasks.

  • Arch's musl-cross-make
  • Arch's coreboot's version musl-cross (buildstack)
  • Arch's board having most modules compiled.

From past experience which permitted to cache that, we were able to build on top of cache under 15 minutes, without having a large resource associated with CircleCI configs.

In last attempt on my side, once musl-cross-make is built separately (see tlaurion@acd6061), building with CircleCI large resource identifier a board, after musl-cross-make is built seperately, fits just under the one hour limit (but fails when creating workspace cache), see here: https://app.circleci.com/pipelines/github/tlaurion/heads/862/workflows/58f32954-1aff-4118-9ff0-17c3621cfeab, specifically https://app.circleci.com/pipelines/github/tlaurion/heads/862/workflows/58f32954-1aff-4118-9ff0-17c3621cfeab/jobs/2360 which fails after having uploaded artifacts, that is, because musl-cross of the build was included into that job; which is the part that takes the most time for a board's build time if not already existing.

So if we were building coreboot's musl-cross, we could still use the free tier. But Heads doesn't currently permit to only build musl-cross. The only supported coreboot's module supported make statements are:

  • coreboot
  • coreboot-blobs.clean
  • coreboot.saveconfig
  • coreboot-blobs
  • coreboot.clean
  • coreboot.menuconfig

We would need an independent coreboot.buildstack make target.

@osresearch @Thrilleratplay @Tonux599 : if make BOARD=x230 coreboot.buildstack existed, then we could continue to use CircleCI to build all the boards on free tier. (while #984 is still needed and the faster we get there, the easiest it would be...)

I tried to create coreboot.buildstack under modules/coreboot but failed. Any insights/PR so that we could have the buildstack (musl-cross toolchain) built for a BOARD=xyz (so a specific version?)

@tlaurion tlaurion changed the title CircleCI restrained their build time again. Now a step is limited to 1 hour. CircleCI now unfit. CircleCI restrained their build time again. Now a job is limited to 1 hour. Nov 25, 2021
@tlaurion tlaurion changed the title CircleCI restrained their build time again. Now a job is limited to 1 hour. CircleCI restrained their build time again. Now a job is limited to 1 hour. Make kung fu needed Nov 27, 2021
@tlaurion
Copy link
Collaborator Author

tlaurion commented Nov 29, 2021

Once again, new public info is available.

  • Xlarge docker is supposed to be avail to free tier container, while testing proves large only is available.
  • parallelization was augmented, meaning that we can build 30 jobs in parallel. This doesn't help with the time limit per job of one hour imposed now

Let's see.... What can be done....
Trying to improve my make Kung fu.
Rereading the whole global Makefile.

@tlaurion
Copy link
Collaborator Author

Interestingly enough, it seems that #984 is mostly complete

  • MAKE_JOBS under global Makefile needs to be uncommented to have the proper number of jobs automatically detected, while limiting the load so that builds work smoothly under CI and host systems with proper amount of RAM available. CircleCI free tier now offers the "large" resource pool, and 30 jobs in parallel.
  • The only thing I had to hack (and not sure why it happens, seriously) is newt module, to force it into using one single make job to build its module, otherwise a race condition was happening.

@tlaurion
Copy link
Collaborator Author

@tlaurion
Copy link
Collaborator Author

@Tonux599 as you can see under this testing branch, kgpe-d16 builds (workstation's kernel) is taking more then 40 minutes to build, which make the build fails....

https://app.circleci.com/pipelines/github/tlaurion/heads?branch=test_parallel

I will have to remove kgpe-d16 board of CircleCI as current mitigation to have other builds succeed since CircleCI is stopping merge of other PR regression testing for other boards.

tlaurion added a commit to tlaurion/heads that referenced this issue Dec 1, 2021
CircleCI: We currently drop coreboot 4.11 builds.
- There is a file missing in the builds. Not sure why/how this is happening
src/soc/intel/fsp_broadwell_de/romstage/romstage.c:41:10: fatal error: build.h: No such file or directory
Example:https://app.circleci.com/pipelines/github/tlaurion/heads/877/workflows/7d0248d2-459c-42ad-b741-8fd56a75d527/jobs/2487
- kgpe-d16_workstation building for all GPUs is unfortunately taking too much time to build (40 minutes).
- Not sure why, but it seems that the kernel build paralellization is not working for 4.11 while it works for 4.13
Makefile: Uncomment MAKE_JOBS which passes the number of jobs to numbers cores by default and --max-load of 16
CircleCI: Remove CPUS statement to use Makefile default
modules/newt: force build with one make job, otherwise there is a race condition in module which fails randomly expecting build modules. (TODO: FIX)

Interestingly, building all coreboot 4.13 boards is happening on a clean commit just above 1h limit.

More details:
- CircleCI changed job build time to a maximum of 1h each.
- CircleCI now permits parallelization of 30 jobs
- 6000 build minutes a month.
- Still waiting for osresearch/heads CircleCI project to be unlocked (currently not recognized as open source project?!)
tlaurion added a commit to tlaurion/heads that referenced this issue Dec 1, 2021
CircleCI: We currently drop coreboot 4.11 builds.
- There is a file missing in the builds. Not sure why/how this is happening
src/soc/intel/fsp_broadwell_de/romstage/romstage.c:41:10: fatal error: build.h: No such file or directory
Example:https://app.circleci.com/pipelines/github/tlaurion/heads/877/workflows/7d0248d2-459c-42ad-b741-8fd56a75d527/jobs/2487
- kgpe-d16_workstation building for all GPUs is unfortunately taking too much time to build (40 minutes).
- Not sure why, but it seems that the kernel build paralellization is not working for 4.11 while it works for 4.13
Makefile: Uncomment MAKE_JOBS which passes the number of jobs to numbers cores by default and --max-load of 16
CircleCI: Remove CPUS statement to use Makefile default
modules/newt: force build with one make job, otherwise there is a race condition in module which fails randomly expecting build modules. (TODO: FIX)

Interestingly, building all coreboot 4.13 boards is happening on a clean commit just above 1h limit.

More details:
- CircleCI changed job build time to a maximum of 1h each.
- CircleCI now permits parallelization of 30 jobs
- 6000 build minutes a month.
- Still waiting for osresearch/heads CircleCI project to be unlocked (currently not recognized as open source project?!)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant