Skip to content

Conversation

@o-nikolas
Copy link
Contributor

Similar to the KB creation retry logic here:

def execute(self, context: Context) -> str:
def _create_kb():
# This API call will return the following if the index has not completed, but there is no apparent
# way to check the state of the index beforehand, so retry on index failure if set to do so.
# botocore.errorfactory.ValidationException: An error occurred (ValidationException)
# when calling the CreateKnowledgeBase operation: The knowledge base storage configuration
# provided is invalid... no such index [bedrock-sample-rag-index-abc108]
try:
return self.hook.conn.create_knowledge_base(
name=self.name,
roleArn=self.role_arn,
knowledgeBaseConfiguration=self.knowledge_base_config,
storageConfiguration=self.storage_config,
**self.create_knowledge_base_kwargs,
)["knowledgeBase"]["knowledgeBaseId"]
except ClientError as error:
error_message = error.response["Error"]["Message"].lower()
is_known_retryable_message = (
"no such index" in error_message
# It may also be that permissions haven't even propagated yet to check for the index
or "server returned 401" in error_message
or "user does not have permissions" in error_message
)
if all(
[
error.response["Error"]["Code"] == "ValidationException",
is_known_retryable_message,
self.wait_for_indexing,
self.indexing_error_max_attempts > 0,
]
):
self.indexing_error_max_attempts -= 1
self.log.warning(
"Vector index not ready, retrying in %s seconds.", self.indexing_error_retry_delay
)
self.log.info("%s retries remaining.", self.indexing_error_max_attempts)
sleep(self.indexing_error_retry_delay)
return _create_kb()
raise
self.log.info("Creating Amazon Bedrock Knowledge Base %s", self.name)
knowledge_base_id = _create_kb()

We need retries around starting the ingestion job because the index may not be ready or accessible.

I also added tests to cover the retry behaviour.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@o-nikolas o-nikolas requested a review from eladkal as a code owner September 9, 2025 23:49
@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Sep 9, 2025
@o-nikolas o-nikolas merged commit 0f4cc7c into apache:main Sep 10, 2025
79 checks passed
@o-nikolas o-nikolas deleted the onikolas/bedrock_index_not_ready branch September 10, 2025 17:28
suman-himanshu pushed a commit to suman-himanshu/airflow that referenced this pull request Sep 17, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
Brunda10 pushed a commit to Brunda10/airflow that referenced this pull request Sep 17, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Sep 30, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 1, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 2, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 3, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 4, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
We need retries around starting the ingestion job because the index may not be ready or accessible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants