Confluence loader API request retry capability

[RPKosec]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Currently, if loading from Confluence and connection to API is not very stable, error can occur while loading. This effectively stops the document ingestion and user has to restart the process.
It would help to add retry capability with max retries attribute. This would eliminate the failing API/API connection issue.

Motivation

Not an issue in small Confluence spaces, but can be painful in larger, where error is more likely and loads take more time. Adding the capability would eliminate the issue for most cases allowing to load larger Confluence spaces from API with stability issues. In cases tested, this happened using Confluence on prem and more often in combination with VPN.

Error experienced when connection fails (with actual values omitted):

Error: Error: Failed to fetch https://[on prem confluence url]/rest/api/content?spaceKey=[space key]&limit=&start=0&expand=body.storage,version from Confluence: TypeError: fetch failed
    at ConfluencePagesLoader.fetchConfluenceData (/home/node/Flowise/node_modules/.pnpm/@langchain+community@0.2.17_@aws-crypto+sha256-js@5.2.0_@aws-sdk+client-bedrock-runtime@3.422_jcdygvtk44q5oagndqvka4fvyq/node_modules/@langchain/community/dist/document_loaders/web/confluence.cjs:134:19)
    at async ConfluencePagesLoader.fetchAllPagesInSpace (/home/node/Flowise/node_modules/.pnpm/@langchain+community@0.2.17_@aws-crypto+sha256-js@5.2.0_@aws-sdk+client-bedrock-runtime@3.422_jcdygvtk44q5oagndqvka4fvyq/node_modules/@langchain/community/dist/document_loaders/web/confluence.cjs:144:22)
    at async ConfluencePagesLoader.load (/home/node/Flowise/node_modules/.pnpm/@langchain+community@0.2.17_@aws-crypto+sha256-js@5.2.0_@aws-sdk+client-bedrock-runtime@3.422_jcdygvtk44q5oagndqvka4fvyq/node_modules/@langchain/community/dist/document_loaders/web/confluence.cjs:102:27)
    at async Confluence_DocumentLoaders.init (/home/node/Flowise/packages/components/dist/nodes/documentloaders/Confluence/Confluence.js:105:20)

Proposal (If applicable)

Small change needs to be done in langchain/src/document_loaders/web/confluence.ts fetchConfluenceData function.
Proposed solution:

protected async fetchConfluenceData(
    url: string
  ): Promise<ConfluenceAPIResponse> {
    let retryCounter = 0;
    // eslint-disable-next-line no-constant-condition
    while (true) {
      retryCounter += 1;
      try {
        const initialHeaders: HeadersInit = {
          "Content-Type": "application/json",
          Accept: "application/json",
        };

        const authHeader = this.authorizationHeader;
        if (authHeader) {
          initialHeaders.Authorization = authHeader;
        }

        const response = await fetch(url, {
          headers: initialHeaders,
        });

        if (!response.ok) {
          throw new Error(
            `Failed to fetch ${url} from Confluence: ${response.status}. Retrying...`
          );
        }

        return await response.json();
      } catch (error) {
        if (retryCounter >= this.maxRetries)
          throw new Error(
            `Failed to fetch ${url} from Confluence (retry: ${retryCounter}): ${error}`
          );
      }
    }
  }

[RPKosec]

I implemented and manually tested the changes, proposed above and submitted draft PR.
However, proper test probably should be implemented, but I have trouble mocking the fetch function in protected fetchConfluenceData function to test response error. Help would be appreciated.