Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: rerun workflow from failed #28143

Merged
merged 33 commits into from
Nov 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
936457d
rerun from failed
seaona Oct 29, 2024
789ea52
failure on purpose to test
seaona Oct 29, 2024
8ac2c9a
remove extra lines
seaona Oct 29, 2024
78de128
rerun script always
seaona Oct 29, 2024
7068048
required completed or canceled
seaona Oct 29, 2024
f911c7f
run always
seaona Nov 6, 2024
8098eea
rerun using trigger
seaona Nov 10, 2024
26da987
typo
seaona Nov 10, 2024
02cde93
add description
seaona Nov 10, 2024
29a895e
description
seaona Nov 10, 2024
f801957
rerun from failed
seaona Nov 11, 2024
2cc00ae
remov extre deps
seaona Nov 11, 2024
4c0ef28
config fix
seaona Nov 11, 2024
08d53b1
edit name
seaona Nov 11, 2024
6e99343
Merge branch 'develop' into rerun-workflow-failed
seaona Nov 11, 2024
894483d
testing
seaona Nov 12, 2024
6aad247
add logs
seaona Nov 12, 2024
1fac883
parse response
seaona Nov 12, 2024
65a2bc2
add more logs
seaona Nov 12, 2024
81baf57
api-v2
seaona Nov 12, 2024
bc5d452
add logging on each step
seaona Nov 12, 2024
6f1a1d0
final logs
seaona Nov 12, 2024
8206ed0
finished testing, adding filter back
seaona Nov 12, 2024
6a45bf8
another test for rerun
seaona Nov 13, 2024
a0d82ee
id
seaona Nov 13, 2024
63fd294
add filters back
seaona Nov 13, 2024
48529ba
remove token for get requests, and remove filter for tesst
seaona Nov 15, 2024
39017df
add filter back
seaona Nov 15, 2024
88c0e66
Merge branch 'develop' into rerun-workflow-failed
seaona Nov 19, 2024
e95eec3
addressed comment: uncaught promise error
seaona Nov 20, 2024
a4dd676
address comment for tsx use
seaona Nov 20, 2024
eb7ef3a
remove filters to test last changes
seaona Nov 20, 2024
e01f7ee
tested successfully - add filters back
seaona Nov 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,19 @@ workflows:
requires:
- prep-build-ts-migration-dashboard

rerun-from-failed:
when:
condition:
equal: ["<< pipeline.schedule.name >>", "rerun-from-failed"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this workflow will only run in develop, and if it's triggered with this exact name

Screenshot from 2024-11-11 16-37-41

jobs:
- prep-deps
- rerun-workflows-from-failed:
filters:
branches:
only: develop
requires:
- prep-deps

locales_only:
when:
matches:
Expand Down Expand Up @@ -928,6 +941,17 @@ jobs:
paths:
- development/ts-migration-dashboard/build/final

rerun-workflows-from-failed:
executor: node-browsers-small
steps:
- run: *shallow-git-clone-and-enable-vnc
- run: sudo corepack enable
- attach_workspace:
at: .
- run:
name: Rerun workflows from failed
command: yarn ci-rerun-from-failed

test-yarn-dedupe:
executor: node-browsers-small
steps:
Expand Down
211 changes: 211 additions & 0 deletions .circleci/scripts/rerun-ci-workflow-from-failed.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
const CIRCLE_TOKEN = process.env.API_V2_TOKEN;

interface Actor {
login: string;
avatar_url: string | null;
}

interface Trigger {
received_at: string;
type: string;
actor: Actor;
}

interface VCS {
origin_repository_url: string;
target_repository_url: string;
revision: string;
provider_name: string;
branch: string;
}

interface WorkflowItem {
id: string;
errors: string[];
project_slug: string;
updated_at: string;
number: number;
state: string;
created_at: string;
trigger: Trigger;
vcs: VCS;
}

interface CircleCIResponse {
next_page_token: string | null;
items: WorkflowItem[];
}

interface WorkflowStatusItem {
pipeline_id: string;
id: string;
name: string;
project_slug: string;
tag?: string;
status: string;
started_by: string;
pipeline_number: number;
created_at: string;
stopped_at: string;
}

interface WorkflowStatusResponse {
next_page_token: string | null;
items: WorkflowStatusItem[];
}

/**
* Fetches the last 20 CircleCI workflows for the given branch.
* Note: the API returns the first 20 workflows by default.
* If we wanted to get older workflows, we would need to use the 'page-token' we would get in the first response
* and perform a subsequent request with the 'page-token' parameter.
* This seems unnecessary as of today, as the amount of daily PRs merged to develop is not that high.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good decision. Easier to run this multiple times throughout the day, rather than support paging through more runs.

*
* @returns {Promise<WorkflowItem[]>} A promise that resolves to an array of workflow items.
* @throws Will throw an error if the CircleCI token is not defined or if the HTTP request fails.
*/
async function getCircleCiWorkflowsByBranch(branch: string): Promise<WorkflowItem[]> {
if (!CIRCLE_TOKEN) {
throw new Error('CircleCI token is not defined');
}

const url = `https://circleci.com/api/v2/project/github/${process.env.CIRCLE_PROJECT_USERNAME}/${process.env.CIRCLE_PROJECT_REPONAME}/pipeline?branch=${branch}`;
const options = {
method: 'GET',
headers: {
'Content-Type': 'application/json',
}
};

try {
const response = await fetch(url, options);
if (!response.ok) {
const errorBody = await response.text();
console.error('HTTP error response:', errorBody);
throw new Error(`HTTP error! status: ${response.status}`);
}
const body = await response.json();
console.log('Circle Ci workflows fetched successfully!');
return body.items;
} catch (error) {
console.error('Error:', error);
throw error;
}
}

/**
* Fetches the status of a specific CircleCI workflow.
*
* @param {string} workflowId - The ID of the workflow to fetch the status for.
* @returns {Promise<WorkflowStatusResponse>} A promise that resolves to the workflow status response.
* @throws Will throw an error if the CircleCI token is not defined or if the HTTP request fails.
*/
async function getWorkflowStatusById(workflowId: string): Promise<WorkflowStatusResponse> {
if (!CIRCLE_TOKEN) {
throw new Error('CircleCI token is not defined');
}

const url = `https://circleci.com/api/v2/pipeline/${workflowId}/workflow`;
const options = {
method: 'GET',
headers: {
'Content-Type': 'application/json',
}
};

try {
console.log(`Fetching workflow ${workflowId}...`);

const response = await fetch(url, options);
if (!response.ok) {
const errorBody = await response.text();
console.error('HTTP error response:', errorBody);
throw new Error(`HTTP error! status: ${response.status}`);
}
const workflowStatus = await response.json();

console.log(`Number of runs: ${workflowStatus.items.length}`);
console.log(`Workflow status from last run: ${workflowStatus.items[0].status}`);

return workflowStatus;

} catch (error) {
console.error('Error:', error);
throw error;
}
}

/**
* Reruns a CircleCI workflow by its ID.
*
* @param {string} workflowId - The ID of the workflow to rerun.
* @throws Will throw an error if the CircleCI token is not defined or if the HTTP request fails.
*/
async function rerunWorkflowById(workflowId: string) {
if (!CIRCLE_TOKEN) {
throw new Error('CircleCI token is not defined');
}

const url = `https://circleci.com/api/v2/workflow/${workflowId}/rerun`;
const options = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Circle-Token': CIRCLE_TOKEN,
},
body: JSON.stringify({
enable_ssh: false,
from_failed: true,
sparse_tree: false, // mutually exclusive with the from_failed parameter
})
};

try {
console.log(`Rerunning workflow ${workflowId}...`);
const response = await fetch(url, options);
if (!response.ok) {
const errorBody = await response.text();
console.error('HTTP error response:', errorBody);
throw new Error(`HTTP error! status: ${response.status}`);
}
const body = await response.json();
console.log('Workflow rerun successfully!');
console.log(body);
} catch (error) {
console.error('Error:', error);
}
}

/**
* Re-runs failed CircleCI workflows from develop branch.
* The workflow will only be re-runed if:
* 1. It has the status of 'failed'
* 2. It has only been run once
* 3. It is among the most recent 20 workflows
* 4. It was triggered by the 'github-merge-queue[bot]' user
*
* @throws Will throw an error if fetching the workflows or re-running a workflow fails.
*/
async function rerunFailedWorkflowsFromDevelop() {
console.log('Getting Circle Ci workflows from develop branch...');
const workflows = await getCircleCiWorkflowsByBranch('develop');

console.log('Assessing if any of the workflows needs to be rerun...');
for (const item of workflows) {
if (item.trigger.actor.login === 'github-merge-queue[bot]') {
const workflowStatus = await getWorkflowStatusById(item.id);

if (workflowStatus.items.length === 1 && workflowStatus.items[0].status === 'failed') {
await rerunWorkflowById(workflowStatus.items[0].id);
console.log(`Rerun workflow with ID: ${workflowStatus.items[0].id}`);
}
}
}
console.log('Task completed successfully!');
}

rerunFailedWorkflowsFromDevelop()
.catch((error) => {
console.error(error);
process.exitCode = 1;
});
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@
"master-sync": "node development/master-sync.js",
"update-mock-cdn": "node test/e2e/mock-cdn/update-mock-cdn-files.js",
"attributions:check": "./development/attributions-check.sh",
"attributions:generate": "./development/generate-attributions.sh"
"attributions:generate": "./development/generate-attributions.sh",
"ci-rerun-from-failed": "tsx .circleci/scripts/rerun-ci-workflow-from-failed.ts"
},
"resolutions": {
"chokidar": "^3.6.0",
Expand Down
Loading