Dataset API failed

[allwin-winfred]

We are currently testing all APIs from the sample Postman collection you provided.
However, we've encountered an issue with the dataset API service.

dataset API application pod goes down and we inspect the application we found this error,

kafka connection arranged succesfully...
/opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8
    const response = new exceptionCtor({
                     ^
PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
    at throwDefaultError (/opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_ListObjectsV2CommandError (/opt/api-service/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/api-service/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /opt/api-service/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
    at async /opt/api-service/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /opt/api-service/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
    at async /opt/api-service/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
    at async /opt/api-service/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 301,
    requestId: '67ZY8F3ZE998QXD5',
    extendedRequestId: 'rvE+KCfQ+qcRw2qE4mD+OFDsrFAErJ52MRy8NK28FEfKRRei48lEsRwyqgWoVyIZLIQhm67ZeFw=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'PermanentRedirect',
  Endpoint: 'obsrv-dev-905418313089.s3.ap-south-1.amazonaws.com',
  Bucket: 'obsrv-dev-905418313089',
  RequestId: '67ZY8F3ZE998QXD5',
  HostId: 'rvE+KCfQ+qcRw2qE4mD+OFDsrFAErJ52MRy8NK28FEfKRRei48lEsRwyqgWoVyIZLIQhm67ZeFw='
}

It's not getting any response, can you help solve this issue

[anandp504]

@allwin-winfred We are looking into the issue. We will check and get back to you. In the meanwhile, can you please provide more details about the issue? Is the pod going down when you are trying to access any specific API?

[allwin-winfred]

@anandp504 yes sure, we are trying to get a Data Exhaust API for our DataSet API pod it went down and the pod logs are,

kafka connection arranged succesfully...
/opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8
    const response = new exceptionCtor({
                     ^
PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
    at throwDefaultError (/opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /opt/api-service/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_ListObjectsV2CommandError (/opt/api-service/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/api-service/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /opt/api-service/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
    at async /opt/api-service/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /opt/api-service/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
    at async /opt/api-service/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
    at async /opt/api-service/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 301,
    requestId: '67ZY8F3ZE998QXD5',
    extendedRequestId: 'rvE+KCfQ+qcRw2qE4mD+OFDsrFAErJ52MRy8NK28FEfKRRei48lEsRwyqgWoVyIZLIQhm67ZeFw=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'PermanentRedirect',
  Endpoint: 'obsrv-dev-905418313089.s3.ap-south-1.amazonaws.com',
  Bucket: 'obsrv-dev-905418313089',
  RequestId: '67ZY8F3ZE998QXD5',
  HostId: 'rvE+KCfQ+qcRw2qE4mD+OFDsrFAErJ52MRy8NK28FEfKRRei48lEsRwyqgWoVyIZLIQhm67ZeFw='
}

[allwin-winfred]

@sowmya-dixit - we provided the correct region and configured it like this on dataset_api_values.yaml.tfpl,

exhaust_service:
  CLOUD_STORAGE_PROVIDER: "aws"
  CLOUD_STORAGE_REGION: "ap-south-1"
  CONTAINER: "obsrv-dev-905418313089" # Bucket name.
  CONTAINER_PREFIX: "telemetry-data"
  STORAGE_URL_EXPIRY: 3600
  EXHAUST_QUERY_RANGE: 31 # In Days

And the issue is still the same.

[sowmya-dixit]

@allwin-winfred We are looking into the issue. Meanwhile can you check and let us know if the ServiceAccount is created with proper annotations under the dataset-api namespace and has access to the bucket you are using? Also let us know if bucket is public/private.

[allwin-winfred]

@sowmya-dixit - The service account didn't have the IAM role with S3fullaccess attached. I tried attaching the role to SA manually but still getting the same error. The bucket is private.

[sowmya-dixit]

@allwin-winfred We are looking into the issue and trying to reproduce in our environment as well. We will give the patch in upcoming release for this issue.

Meanwhile Can you please make below changes under dataset_api helm chart and let us know if these changes works -

Under dataset_api.yaml.tfpl
---------------------------
exhaust_service:
  CONTAINER: ${s3_bucket}
  CLOUD_STORAGE_CONFIG: '{"identity": "AWS_ACCESS_KEY", "credential": "AWS_SECRET_KEY", "region": "AWS_REGION"}'

Under configmap.yaml
--------------------
cloud_storage_config: "{{ .Values.exhaust_service.CLOUD_STORAGE_CONFIG }}"

[allwin-winfred]

@sowmya-dixit - after we made above changes in YML files it's working fine. thanks