[Tooling] Add Claude Build Analysis to any failing build job #25104

iangmaia · 2026-01-05T18:01:18Z

Fixes AINFRA-1695

Description

Adds to the project Claude Build Analysis, similarly to what has been done on Android (with the follow-up on wordpress-mobile/WordPress-Android#22459).
Claude will then from now on run and analyze any build job that has failed, adding a report annotation about the build failure. We'll also add a PR comment mentioning the Claude build analysis.

This PR is built on top of #25120, which was something I implemented in order to make the logs more compact. If we decide to not use it, we can retarget this PR to trunk.

Testing

See failed test build with annotation:

If you want, you can also check the annotation by creating a Pull Request based on this branch and breaking any of the jobs in the build.

dangermattic · 2026-01-05T18:01:48Z

	1 Message
📖	This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

wpmobilebot · 2026-01-05T18:13:34Z

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	`30331`
	Version	`PR #25104`
	Bundle ID	`com.jetpack.alpha`
	Commit	`0b0f017`
	Installation URL	0vp4fmcb8t7q0

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-01-05T18:13:42Z

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	`30331`
	Version	`PR #25104`
	Bundle ID	`org.wordpress.alpha`
	Commit	`0b0f017`
	Installation URL	7ue2f52sooot8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-01-07T16:14:57Z

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

iangmaia · 2026-01-08T11:57:04Z

.buildkite/shared-pipeline-vars


 export IMAGE_ID="xcode-$XCODE_VERSION"
 export CI_TOOLKIT_PLUGIN="automattic/a8c-ci-toolkit#$CI_TOOLKIT_PLUGIN_VERSION"
+export CLAUDE_PLUGIN="iangmaia/claude-summarize#iangmaia/add-build-log-mode"


I've created a Pull Request to the claude-summarize-buildkite-plugin project introducing the build_log_mode attribute, allowing us to send to for analysis Claude only failed jobs, saving a lot of the tokens usage.
Hopefully it won't take long to merge it and we can revert to the main repo again.

Otherwise, we can always keep using or fork, whic I think you mentioned this elsewhere, too.

This reverts commit d3aa906.

sonarqubecloud · 2026-01-08T12:01:49Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mokagio · 2026-01-09T04:58:22Z

.buildkite/claude-analysis.yml

+          build_log_mode: "failed"
+          trigger: "on-failure"
+          max_log_lines: 1500
+          custom_prompt: "Do not mention successful points, focus on failures and recovery to keep the analysis succint. Only add the `Best Practices` section if the changes in this PR are directly relevant to it. Only add a `Root Cause` section when you're sure about the issues' causes."


Suggested change

custom_prompt: "Do not mention successful points, focus on failures and recovery to keep the analysis succint. Only add the `Best Practices` section if the changes in this PR are directly relevant to it. Only add a `Root Cause` section when you're sure about the issues' causes."

custom_prompt: "Do not mention successful points, focus on failures and recovery to keep the analysis succinct. Only add the `Best Practices` section if the changes in this PR are directly relevant to it. Only add a `Root Cause` section when you're sure about the issues' causes."

mokagio · 2026-01-09T04:59:49Z

.buildkite/pipeline.yml

+      queue: upload
+


Nitipick, no trailing newline.

Suggested change

queue: upload

queue: upload

mokagio · 2026-01-09T05:03:02Z

.buildkite/shared-pipeline-vars


 export IMAGE_ID="xcode-$XCODE_VERSION"
 export CI_TOOLKIT_PLUGIN="automattic/a8c-ci-toolkit#$CI_TOOLKIT_PLUGIN_VERSION"
+export CLAUDE_PLUGIN="iangmaia/claude-summarize#iangmaia/add-build-log-mode"


Otherwise, we can always keep using or fork, whic I think you mentioned this elsewhere, too.

mokagio · 2026-01-09T05:05:35Z

.buildkite/pipeline.yml

-      - label: "🔬 Keystone Unit Tests"
-        command: .buildkite/commands/run-unit-tests-for-scheme.sh Keystone
-        # Disabled till https://github.com/wordpress-mobile/WordPress-iOS/pull/24537 is completed
-        #
-        # The boolean value has to be a string explicitly.
-        # See validation error against schema.
-        if: "false"
-        plugins: [$CI_TOOLKIT_PLUGIN]
-        artifact_paths:
-          - "build/results/*"
-        notify:
-          - github_commit_status:
-              context: "Unit Tests Keystone"
+      # Disabled till https://github.com/wordpress-mobile/WordPress-iOS/pull/24537 is completed
+      # - label: "🔬 Keystone Unit Tests"
+      #   command: .buildkite/commands/run-unit-tests-for-scheme.sh Keystone
+      #   plugins: [$CI_TOOLKIT_PLUGIN]
+      #   artifact_paths:
+      #     - "build/results/*"
+      #   notify:
+      #     - github_commit_status:
+      #         context: "Unit Tests Keystone"


Why commenting the whole step?

I mean, it makes sense so the parser doesn't have to bother with something that is hardcoded if: "false" and the result is the same, but was it to help with this PR or something that stood out?

Perhaps given the commit it the change is in, e349939, it was to reduce the logs?

iangmaia added this to the 26.6 milestone Jan 5, 2026

iangmaia self-assigned this Jan 5, 2026

iangmaia added the Tooling Build, Release, and Validation Tools label Jan 5, 2026

iangmaia force-pushed the iangmaia/claude-build-analysis branch 12 times, most recently from dfddb66 to 7cf07fb Compare January 7, 2026 15:47

iangmaia force-pushed the iangmaia/claude-build-analysis branch 3 times, most recently from 263a056 to daf0d87 Compare January 7, 2026 18:08

iangmaia added 8 commits January 8, 2026 11:32

Add Claude Build Analysis to any failing build job

c9f6e28

Add allow_dependency_failure to final wait step

5e61c83

Test wait step dependency to not depend on optional job

7a21351

Remove wait step to favor an explicit dependency in the Claude step

8fe005a

Add Claude debug as build artifact

cc6e119

Update Claude analysis to make it clear the trigger is only on failures

2f8c9a5

Use forked Claude Summarize Plugin version

9b8c843

Add buildkite logs as artifact

5c6f962

Comment out Keystone Unit Tests and increase max log lines

e349939

iangmaia force-pushed the iangmaia/claude-build-analysis branch from 7601812 to e349939 Compare January 8, 2026 10:33

iangmaia changed the base branch from trunk to iangmaia/disable-wiremock-logs-on-ci January 8, 2026 10:34

[TO REVERT] Test compilation error

d3aa906

iangmaia commented Jan 8, 2026

View reviewed changes

Revert "[TO REVERT] Test compilation error"

0b0f017

This reverts commit d3aa906.

iangmaia requested review from mokagio and twstokes January 8, 2026 16:44

iangmaia marked this pull request as ready for review January 8, 2026 16:45

mokagio approved these changes Jan 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Tooling] Add Claude Build Analysis to any failing build job #25104

[Tooling] Add Claude Build Analysis to any failing build job #25104

iangmaia commented Jan 5, 2026 •

edited

Loading

Uh oh!

dangermattic commented Jan 5, 2026

Uh oh!

wpmobilebot commented Jan 5, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Jan 5, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Jan 7, 2026 •

edited

Loading

Uh oh!

iangmaia Jan 8, 2026

Uh oh!

mokagio Jan 9, 2026

Uh oh!

sonarqubecloud bot commented Jan 8, 2026

Uh oh!

mokagio Jan 9, 2026

Uh oh!

mokagio Jan 9, 2026

Uh oh!

mokagio Jan 9, 2026

Uh oh!

mokagio Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	custom_prompt: "Do not mention successful points, focus on failures and recovery to keep the analysis succint. Only add the `Best Practices` section if the changes in this PR are directly relevant to it. Only add a `Root Cause` section when you're sure about the issues' causes."
	custom_prompt: "Do not mention successful points, focus on failures and recovery to keep the analysis succinct. Only add the `Best Practices` section if the changes in this PR are directly relevant to it. Only add a `Root Cause` section when you're sure about the issues' causes."

[Tooling] Add Claude Build Analysis to any failing build job #25104

Are you sure you want to change the base?

[Tooling] Add Claude Build Analysis to any failing build job #25104

Conversation

iangmaia commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

dangermattic commented Jan 5, 2026

Uh oh!

wpmobilebot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Build Failure Analysis

Uh oh!

iangmaia Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jan 8, 2026

Quality Gate passed

Uh oh!

mokagio Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iangmaia commented Jan 5, 2026 •

edited

Loading

wpmobilebot commented Jan 5, 2026 •

edited

Loading

wpmobilebot commented Jan 5, 2026 •

edited

Loading

wpmobilebot commented Jan 7, 2026 •

edited

Loading