Fix: Health lambas still time out occasionally (#6097) #6118
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #6118 +/- ##
========================================
Coverage 84.89% 84.90%
========================================
Files 156 156
Lines 20690 20692 +2
========================================
+ Hits 17565 17568 +3
+ Misses 3125 3124 -1 ☔ View full report in Codecov by Sentry. |
hannes-ucsc
reviewed
Apr 4, 2024
There was a problem hiding this comment.
I think this is becoming unwieldy and a little brittle.
Can you try something like this, which is based on how we handle the thresholds:
Subject: [PATCH] [A 2/2] Derive manifest config for AnVIL from field mapping (#6110)
Name special fields in camel case on AnVIL. This repairs the IT.
---
Index: src/azul/chalice.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/chalice.py b/src/azul/chalice.py
--- a/src/azul/chalice.py (revision 3b2831b32aa4c463218ddff66830a3df41567f14)
+++ b/src/azul/chalice.py (date 1712209742788)
@@ -108,6 +108,13 @@
value: int
+@attr.s(auto_attribs=True, frozen=True, kw_only=True)
+class Retry:
+ lambda_name: str
+ handler_name: Optional[str] = attr.ib(default=None)
+ value: int
+
+
C = TypeVar('C', bound='AppController')
@@ -526,6 +533,24 @@
value=threshold))
return thresholds
+ def retry(self, retries: int):
+ def wrapper(f):
+ assert isinstance(f, chalice.app.EventSourceHandler), f
+ f.retries = 2
+ return f
+
+ return wrapper
+
+ def retries(self) -> list[Retry]:
+ lambda_name, _ = config.unqualified_resource_name(self.app_name)
+ retries = []
+ for handler_name, handler in self.handler_map.items():
+ if isinstance(handler, chalice.app.EventSourceHandler):
+ value = getattr(handler, 'retries', 0)
+ retries.append(MetricThreshold(lambda_name=lambda_name,
+ handler_name=handler_name,
+ value=value))
+
@attr.s(auto_attribs=True, frozen=True, kw_only=True)
class AppController:
Index: lambdas/indexer/app.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lambdas/indexer/app.py b/lambdas/indexer/app.py
--- a/lambdas/indexer/app.py (revision 3b2831b32aa4c463218ddff66830a3df41567f14)
+++ b/lambdas/indexer/app.py (date 1712209286051)
@@ -337,6 +337,7 @@
app.index_controller.contribute(event, retry=True)
+@app.retry(2)
@app.log_forwarder(
config.alb_access_log_path_prefix(deployment=None)
)
Index: terraform/api_gateway.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/api_gateway.tf.json.template.py b/terraform/api_gateway.tf.json.template.py
--- a/terraform/api_gateway.tf.json.template.py (revision 3b2831b32aa4c463218ddff66830a3df41567f14)
+++ b/terraform/api_gateway.tf.json.template.py (date 1712208689183)
@@ -5,11 +5,14 @@
import json
from azul import (
+ cached_property,
config,
)
+from azul.chalice import AzulChaliceApp
from azul.deployment import (
aws,
)
+from azul.modules import load_app_module
from azul.objects import (
InternMeta,
)
@@ -40,6 +43,10 @@
*config.api_lambda_domain_aliases(name)
],
policy=json.dumps(getattr(policy_module, 'policy')))
+
+ @cached_property
+ def chalice(self) -> AzulChaliceApp:
+ return load_app_module(self.name).app
apps = [
@@ -372,16 +379,10 @@
'aws_lambda_function_event_invoke_config': {
function_name: {
'function_name': '${aws_lambda_function.%s.function_name}' % function_name,
- 'maximum_retry_attempts': retry_attempts
+ 'maximum_retry_attempts': retry.value
}
- for function_name, retry_attempts in
- [
- (f'indexer_{lm}', 0)
- for lm in ['forward_alb_logs', 'forward_s3_logs']
- if config.enable_log_forwarding
- ] + [
- (f'{lm}_{lm}cachehealth', 1) for lm in ['indexer', 'service']
- ]
+ for app in apps
+ for retry in app.chalice.retries()
}
},
*(
This has not been tested and may not handle config.enable_log_forwarding being False.
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
3 times, most recently
from
342a8a6 to
d97fda1
Compare
achave11-ucsc
commented
Apr 4, 2024
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
3 times, most recently
from
5d71952 to
6dc6024
Compare
|
@hannes-ucsc, I've dropped the patch you suggested during PL (adding it to the end of this comment), since the default retry limit is two and in the conversation of when this retry constrictions to the forwarder lambdas were employed (#4920 (comment)) it's also confirmed. There is no longer a need for these lambdas to be part of the Subject: [PATCH] Increase retry for the log forwarder Lambdas
---
Index: lambdas/indexer/app.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lambdas/indexer/app.py b/lambdas/indexer/app.py
--- a/lambdas/indexer/app.py
+++ b/lambdas/indexer/app.py
@@ -338,6 +338,7 @@
app.index_controller.contribute(event, retry=True)
+@app.retry(2 if config.enable_log_forwarding else None)
@app.log_forwarder(
config.alb_access_log_path_prefix(deployment=None)
)
@@ -345,6 +346,7 @@
app.log_controller.forward_alb_logs(event)
+@app.retry(2 if config.enable_log_forwarding else None)
@app.log_forwarder(
config.s3_access_log_path_prefix(deployment=None)
)
Index: src/azul/chalice.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/chalice.py b/src/azul/chalice.py
--- a/src/azul/chalice.py
+++ b/src/azul/chalice.py
@@ -533,7 +533,7 @@
value=threshold))
return thresholds
- def retry(self, retries: int):
+ def retry(self, retries: int | None):
"""
Use this decorator to specify a custom number of retries that should be
different from the default (which is two) for any of the Azul async
@@ -541,8 +541,9 @@
https://docs.aws.amazon.com/lambda/latest/dg/invocation-retries.html
"""
def wrapper(f):
- assert isinstance(f, chalice.app.EventSourceHandler), f
- f.retries = retries
+ if retries is not None:
+ assert isinstance(f, chalice.app.EventSourceHandler), f
+ f.retries = retries
return f
return wrapper
|
hannes-ucsc
requested changes
Apr 5, 2024
hannes-ucsc
added
1 review
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
2 times, most recently
from
ad335e0 to
e25a9a1
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
branch
3 times, most recently
from
a915bb0 to
b24d6f0
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
branch
5 times, most recently
from
e2fb053 to
e3c8430
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
branch
3 times, most recently
from
c61cf7e to
0b7c3c4
Compare
hannes-ucsc
force-pushed
the
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
branch
from
c6124a7 to
6a4d709
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
branch
2 times, most recently
from
484f245 to
1854a8e
Compare
achave11-ucsc
changed the base branch from
issues/achave11-ucsc/5622-bump-fwdr-alarm-threshold
to
develop
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
2 times, most recently
from
ea8e82e to
a75eb6e
Compare
hannes-ucsc
approved these changes
Jun 11, 2024
Security design review
|
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
from
a75eb6e to
63b8a78
Compare
achave11-ucsc
added
the
sandbox
achave11-ucsc
deleted the
issues/achave11-ucsc/6097-retry-cachehealth-failures
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Connected issues: #6097
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all connected issuespartiallabelAuthor (chains)
baseor this PR is not chained to another PRchainedor is not chained to another PRAuthor (reindex, API changes)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAPIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make image_manifests.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifyimage_manifests.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues connected to this PRAuthor (before every review)
develop, squashed old fixupsmake requirements_updateor this PR does not modifyrequirements*.txt,common.mk,MakefileandDockerfileRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator (before pushing merge the commit)
reindex:…labels andrcommit title tagno demodevelop_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator
dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (before pushing merge the commit)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnersandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices indevanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevpif the PR is also labeledpartialOperator (chain shortening)
developor this PR is not labeledbasechainedlabel from the blocked PR or this PR is not labeledbasebasebaselabel from this PR or this PR is not labeledbaseOperator (after pushing the merge commit)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem