-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🔄 Fetch with Retry for HTML build #1793
Merged
+76
−19
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
--- | ||
"myst-cli": patch | ||
--- | ||
|
||
Retry html pages build and limit initial outgoing connections. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
import type { RequestInfo, RequestInit, Response } from 'node-fetch'; | ||
import type { ISession } from '../session/types.js'; | ||
|
||
/** | ||
* Recursively fetch a URL with retry and exponential backoff. | ||
*/ | ||
export async function fetchWithRetry( | ||
session: Pick<ISession, 'log' | 'fetch'>, | ||
/** The URL to fetch. */ | ||
url: URL | RequestInfo, | ||
/** Options to pass to fetch (e.g., headers, method). */ | ||
options?: RequestInit, | ||
/** How many times total to attempt the fetch. */ | ||
maxRetries = 3, | ||
/** The current attempt number. */ | ||
attempt = 1, | ||
/** The current backoff duration in milliseconds. */ | ||
backoff = 250, | ||
): Promise<Response> { | ||
try { | ||
const resp = await session.fetch(url, options); | ||
if (resp.ok) { | ||
// If it's a 2xx response, we consider it a success and return it | ||
return resp; | ||
} else { | ||
// For non-2xx, we treat it as a failure that triggers a retry | ||
session.log.warn( | ||
`Fetch of ${url} failed with HTTP status ${resp.status} for URL: ${url} (Attempt #${attempt})`, | ||
); | ||
} | ||
} catch (error) { | ||
// This covers network failures and other errors that cause fetch to reject | ||
session.log.warn(`Fetch of ${url} threw an error (Attempt #${attempt})`, error); | ||
} | ||
|
||
// If we haven't reached the max retries, wait and recurse | ||
if (attempt < maxRetries) { | ||
session.log.debug(`Waiting ${backoff}ms before retry #${attempt + 1}...`); | ||
await new Promise((resolve) => setTimeout(resolve, backoff)); | ||
return fetchWithRetry(session, url, options, maxRetries, attempt + 1, backoff * 2); | ||
} | ||
|
||
// If we made it here, all retries have been exhausted | ||
throw new Error(`Failed to fetch ${url} after ${maxRetries} attempts.`); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Unchanged files with check annotations Beta
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import { updateLinkTextIfEmpty, withoutHttp } from './utils.js'; | ||
const RESOLVER = 'https://github.com/'; | ||
const TRANSFORM_SOURCE = 'LinkTransform:Github'; | ||
type GithubFileLink = { | ||
kind: 'file'; | ||
return withoutHttp(uri).startsWith(withoutHttp(RESOLVER)); | ||
} | ||
transform(link: Link, file: VFile): boolean { | ||
const urlSource = link.urlSource || link.url; | ||
const parsed = parseGithubIssue(urlSource) || parseGithubFile(urlSource); | ||
if (!parsed) { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
constructor(opts?: { logger?: Logger }) { | ||
this.log = opts?.logger ?? chalkLogger(LogLevel.debug, process.cwd()); | ||
} | ||
fetch(url: URL | RequestInfo, init?: RequestInit | undefined): Promise<Response> { | ||
Check warning on line 10 in packages/myst-cli-utils/src/session.ts
|
||
throw new Error('fetch not implemented on session'); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import type { DirectiveData, DirectiveSpec, GenericNode } from 'myst-common'; | ||
import { RuleId, fileWarn, normalizeLabel } from 'myst-common'; | ||
import { CODE_DIRECTIVE_OPTIONS, getCodeBlockOptions } from './code.js'; | ||
import type { Include } from 'myst-spec-ext'; | ||
import type { VFile } from 'vfile'; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import type { VFile } from 'vfile'; | ||
import type { GenericNode } from 'myst-common'; | ||
import { fileError, fileWarn, toText, getMetadataTags } from 'myst-common'; | ||
import { captionHandler, containerHandler, getDefaultCaptionSupplement } from './container.js'; | ||
import type { | ||
Handler, | ||
ITypstSerializer, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import type { JSONSchema7Definition } from 'json-schema'; | ||
const LicenseSchema: JSONSchema7Definition = { | ||
$id: '', | ||
description: | ||
'The license information, which can be either a string or an object with `code` and `content`.', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import { cleanOutput } from './utils/cleanOutput.js'; | ||
import { getFileContent } from './utils/getFileContent.js'; | ||
import { resolveFrontmatterParts } from '../utils/resolveFrontmatterParts.js'; | ||
import { parseMyst } from '../process/myst.js'; | ||
function exportOptionsToCFF(exportOptions: ExportWithOutput): CFF { | ||
// Handle overlap of key "format" between CFF and export |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdast: GenericParent, | ||
references: References, | ||
frontmatter: PageFrontmatter, | ||
templateYml: TemplateYml | null, | ||
printGlossaries: boolean, | ||
) { | ||
const pipe = unified().use(mystToTypst, { | ||
math: frontmatter?.math, |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Main changes here, limiting connections to 5 and having a retry with exponential backoff.