[P2] Investigate Transient Error when provisioning RTSP stream container

[CESARDELATORRE]

Not sure if this is just a transient error, but..:

ERROR:
0/1 nodes are available: 1 Insufficient akri.sh/akri-camera-2814d0. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
Back-off restarting failed container daprd in pod ubuntu-20-04-vm-akri-camera-2814d0-pod_mec-accelerator(7e6e6b03-bb5c-4403-a6f3-9677f2c0a7e6)

This was the creation:

Traces:

DEBUG:botocore.hooks:Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.hooks:Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7f1f7079b700>
DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f1f707a08b0>
DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f1f6fb20e50>>
DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'images', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.mec-accelerator.svc.cluster.local:9000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
DEBUG:botocore.regions:Endpoint provider result: http://minio.mec-accelerator.svc.cluster.local:9000/images
DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4"
DEBUG:botocore.regions:Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
DEBUG:botocore.hooks:Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f1f7079bc10>
DEBUG:botocore.hooks:Event before-call.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.hooks:Event before-call.s3.HeadObject: calling handler <function add_recursion_detection_header at 0x7f1f7079b3a0>
DEBUG:botocore.hooks:Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7f1f7079cf70>
DEBUG:botocore.endpoint:Making request for OperationModel(name=HeadObject) with params: {'url_path': '/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.34.33 md/Botocore#1.34.33 ua/2.0 os/linux#5.15.0-1054-azure md/arch#x86_64 lang/python#3.8.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.33'}, 'body': b'', 'auth_path': '/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'url': 'http://minio.mec-accelerator.svc.cluster.local:9000/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f1f6fe20790>, 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'images', 'params': {'Bucket': 'images', 'Key': 'ed82a37e-8af6-418b-9707-afdf8070439b.jpg'}}, 'S3Express': {'bucket_name': 'images'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
DEBUG:botocore.hooks:Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1f6fe206d0>>
DEBUG:botocore.hooks:Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7f1f7079b5e0>
DEBUG:botocore.hooks:Event before-sign.s3.HeadObject: calling handler <function remove_arn_from_signing_path at 0x7f1f707a0820>
DEBUG:botocore.hooks:Event before-sign.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.auth:Calculating signature using v4 auth.
DEBUG:botocore.auth:CanonicalRequest:
HEAD
/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg
host:minio.mec-accelerator.svc.cluster.local:9000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240211T182321Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG:botocore.auth:StringToSign:
AWS4-HMAC-SHA256
20240211T182321Z
20240211/us-east-1/s3/aws4_request
9d1a952d96731e6c13807fd38c6a2a3c1eace167e7f56b60a297df687bef7a41
DEBUG:botocore.auth:Signature:
9775f7380f072b3d389bc1e9c78ac761a4d35fdedd0dacd6052a3faa42f9c6ce
DEBUG:botocore.hooks:Event request-created.s3.HeadObject: calling handler <function add_retry_headers at 0x7f1f707a0670>
DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=http://minio.mec-accelerator.svc.cluster.local:9000/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg, headers={'User-Agent': b'Boto3/1.34.33 md/Botocore#1.34.33 ua/2.0 os/linux#5.15.0-1054-azure md/arch#x86_64 lang/python#3.8.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.33', 'X-Amz-Date': b'20240211T182321Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20240211/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9775f7380f072b3d389bc1e9c78ac761a4d35fdedd0dacd6052a3faa42f9c6ce', 'amz-sdk-invocation-id': b'99e8e50f-b13c-4035-92e8-e5672cb981c6', 'amz-sdk-request': b'attempt=1'}>
DEBUG:urllib3.connectionpool:http://minio.mec-accelerator.svc.cluster.local:9000 "HEAD /images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg HTTP/1.1" 404 0
DEBUG:botocore.parsers:Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8', 'X-Amz-Request-Id': '17B2E217FE0E8DD4', 'X-Content-Type-Options': 'nosniff', 'X-Minio-Error-Code': 'NoSuchKey', 'X-Minio-Error-Desc': '"The specified key does not exist."', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 18:23:21 GMT'}
DEBUG:botocore.parsers:Response body:
b''
DEBUG:botocore.hooks:Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f1f6fb20df0>
DEBUG:botocore.retryhandler:No retry needed.
DEBUG:botocore.hooks:Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f1f6fb20e50>>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7f1f7079c550>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7f1f7079b940>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7f1f7079ce50>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7f1f7079b8b0>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f1f707a0700>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f1f6fb20e50>>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7f1f7079b700>
DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f1f707a08b0>
DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f1f6fb20e50>>
DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'images', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.mec-accelerator.svc.cluster.local:9000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
DEBUG:botocore.regions:Endpoint provider result: http://minio.mec-accelerator.svc.cluster.local:9000/images
DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4"
DEBUG:botocore.regions:Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function conditionally_calculate_checksum at 0x7f1f70999a60>
DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7f1f7079bc10>
DEBUG:botocore.handlers:Adding expect 100 continue header to request.
DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x7f1f7079b3a0>
DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7f1f7079cf70>
DEBUG:botocore.endpoint:Making request for OperationModel(name=PutObject) with params: {'url_path': '/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.33 md/Botocore#1.34.33 ua/2.0 os/linux#5.15.0-1054-azure md/arch#x86_64 lang/python#3.8.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.33', 'Content-MD5': 'bp4p3cBZ1QKT3uQLGB+TDw==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f1f6fd1f860>, 'auth_path': '/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'url': 'http://minio.mec-accelerator.svc.cluster.local:9000/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f1f6fe20790>, 'has_streaming_input': True, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'images', 'params': {'Body': <_io.BytesIO object at 0x7f1f6fd1f860>, 'Bucket': 'images', 'Key': 'ed82a37e-8af6-418b-9707-afdf8070439b.jpg'}}, 'S3Express': {'bucket_name': 'images'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1f6fe206d0>>
DEBUG:botocore.hooks:Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f1f7079b5e0>
DEBUG:botocore.hooks:Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f1f707a0820>
DEBUG:botocore.hooks:Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f1f6fb20f10>>
DEBUG:botocore.auth:Calculating signature using v4 auth.
DEBUG:botocore.auth:CanonicalRequest:
PUT
/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg
content-md5:bp4p3cBZ1QKT3uQLGB+TDw==
host:minio.mec-accelerator.svc.cluster.local:9000
x-amz-content-sha256:54dace8e0876e7e11a48055d271ec6e95672fcb153866e1047160d61afbc62fe
x-amz-date:20240211T182321Z
content-md5;host;x-amz-content-sha256;x-amz-date
54dace8e0876e7e11a48055d271ec6e95672fcb153866e1047160d61afbc62fe
DEBUG:botocore.auth:StringToSign:
AWS4-HMAC-SHA256
20240211T182321Z
20240211/us-east-1/s3/aws4_request
2fd6f2cf58ed7e2f24c9fe67f0971109d56a5e9a09c410154e7a53ebab535cc8
DEBUG:botocore.auth:Signature:
6471b002cdc1c6a7d469ff11a07c966f020c35dee8479179abff525b3c57d639
DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f1f707a0670>
DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://minio.mec-accelerator.svc.cluster.local:9000/images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg, headers={'User-Agent': b'Boto3/1.34.33 md/Botocore#1.34.33 ua/2.0 os/linux#5.15.0-1054-azure md/arch#x86_64 lang/python#3.8.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.33', 'Content-MD5': b'bp4p3cBZ1QKT3uQLGB+TDw==', 'Expect': b'100-continue', 'X-Amz-Date': b'20240211T182321Z', 'X-Amz-Content-SHA256': b'54dace8e0876e7e11a48055d271ec6e95672fcb153866e1047160d61afbc62fe', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20240211/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=6471b002cdc1c6a7d469ff11a07c966f020c35dee8479179abff525b3c57d639', 'amz-sdk-invocation-id': b'3052b5f2-b03c-45bb-9ce9-1da50f1ca50d', 'amz-sdk-request': b'attempt=1', 'Content-Length': '424981'}>
DEBUG:botocore.awsrequest:Waiting for 100 Continue response.
DEBUG:botocore.awsrequest:100 Continue response seen, now sending request body.
DEBUG:urllib3.connectionpool:http://minio.mec-accelerator.svc.cluster.local:9000 "PUT /images/ed82a37e-8af6-418b-9707-afdf8070439b.jpg HTTP/1.1" 200 0
DEBUG:botocore.parsers:Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'ETag': '"6e9e29ddc059d50293dee40b181f930f"', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8', 'X-Amz-Request-Id': '17B2E217FEC6426B', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 11 Feb 2024 18:23:21 GMT'}
DEBUG:botocore.parsers:Response body:
b''
DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f1f6fb20df0>
DEBUG:botocore.retryhandler:No retry needed.
DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f1f6fb20e50>>
DEBUG:asyncio:Using selector: EpollSelector
Logs from Feb 11, 2024 to Feb 11, 2024 UTC

After a while, looks liek it's working.
But it' be good to investigate why this error was happening...