Unify Logger Config for Tasks

[noah-paige]

Feature or Bugfix

• Refactoring

Detail

• Unify Logger Config in Backend (focused on /tasks)
• Fix Log Level setting

Relates

• Unable to view logs in ECS share manager logs with code from main  #1680
• log API handler response only for LOG_LEVEL DEBUG. Set log level INFO for prod deployments #1662

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

• Does this PR introduce or modify any input fields or queries - this includes
  fetching data from storage outside the application (e.g. a database, an S3 bucket)?
  • Is the input sanitized?
  • What precautions are you taking before deserializing the data you consume?
  • Is injection prevented by parametrizing queries?
  • Have you ensured no eval or similar functions are used?
• Does this PR introduce any functionality or component that requires authorization?
  • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  • Are you logging failed auth attempts?
• Are you using or adding any cryptographic features?
  • Do you use a standard proven implementations?
  • Are the used keys controlled by the customer? Where are they stored?
• Are you introducing any new policies/roles/users?
  • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[noah-paige]

Testing:

• Pipeline Succeeds

Run ECS Tasks and Verify Proper Logs + Formatting w/ Log Level - DEBUG

• Env stack updater
• Cdkproxy
• Omics Workflow Fetcher
• Table Syncer
• Dataset Subscription Task
• Share Manager

Run ECS Tasks and Verify Proper Logs + Formatting w/ Log Level - Info

• Cdkproxy
• Catalog Indexer
• Table Syncer
• Share Manager

[petrkalos]

If I understand correctly this statement here will affect all data.all backend services (lambda, ecs tasks etc).
Should we be removing ALL the per file logging configs? With a quip grep I see 206 files
grep -rail "logging.getLogger(" backend/ | wc -l

[dlpzx]

Do we? I thought that the getlogger is just getting the logger, not configuring it

[noah-paige]

You are correct it affects every file under /backend/dataall/ - I though this would simplify how we were formerly managing logs in each individual location

It also solves the issue we have where a task (like share manager task - at dataall/backend/dataall/modules/shares_base/tasks/share_manager_task.py ) was not recording the logs from SharingService (at dataall/backend/dataall/modules/shares_base/services/sharing_service.py) or the Processors and we were missing logs in CloudWatch

For Reference in dataall-sbx-backend-graphql the log group formatting before the change:

[INFO]	2024-11-19T15:28:43.058Z	a725615b-f975-4a66-8ccb-7e69830f18b9	Current maintenance window status - INACTIVE
[INFO]	2024-11-19T15:28:43.060Z	a725615b-f975-4a66-8ccb-7e69830f18b9	SSM Parameter session in central account

And after the code change (same formatting):

[INFO]	2024-11-20T22:34:02.450Z	2133b99e-3316-44de-9b9c-68f02f96611c	Current maintenance window status - INACTIVE
[INFO]	2024-11-20T22:34:02.452Z	2133b99e-3316-44de-9b9c-68f02f96611c	SSM Parameter session in central account

[noah-paige]

To note the above differs from the format='[%(levelname)s] %(message)s', structure. I believe Lambda has its own default logging formatter that is taking precedence... this log record format of [%(levelname)s] %(message)s', is following in all of the ECS tasks for example:

[petrkalos]

@dlpzx we do configure at a lot of places
@noah-paige I am fine if you want to refactor and remove the configs from all the files keeping only the top one

[noah-paige]

For a full breakdown of where we call logger.setLevel(...) which I think is the main config we do in a number of places (was ~26 places before this PR now at 15 files):

• Backend Files not under dataall/backend/dataall/ (5):
  • dataall/backend/api_handler.py
  • dataall/backend/aws_handler.py
  • dataall/backend/local_graphql_server.py
  • dataall/backend/search_handler.py
• For rest of Backend Code:
  • dataall/backend/dataall/init.py
• For CDK specific activity:
  • dataall/backend/dataall/base/cdkproxy/app.py (creates new logger named cdkapp process always with level INFO) -- chose to leave as is
• Lambdas from data.all resource Custom Resource CDK
  • dataall/backend/dataall/modules/s3_datasets/cdk/assets/gluedatabasecustomresource/index.py
  • dataall/backend/dataall/modules/s3_datasets/cdk/assets/lakeformationdefaultsettings/index.py
• Trigger Function Lambdas data.all pipeline
  • dataall/backend/deployment_triggers/dbmigrations_handler.py
  • dataall/backend/deployment_triggers/dbsnapshots_handler.py
  • dataall/backend/deployment_triggers/saveperms_handler.py
• Custom Resource Lambdas data.all Deployment
  • dataall/deploy/custom_resources/cognito_config/cognito_urls.py
  • dataall/deploy/custom_resources/cognito_config/cognito_users.py
  • dataall/deploy/custom_resources/custom_authorizer/custom_authorizer_lambda.py
  • dataall/deploy/custom_resources/custom_authorizer/jwt_services.py

[noah-paige]

@dlpzx @petrkalos - the places where we configure logs is more intentional now and for the majority of backend it is from dataall/backend/dataall/__init__.py which I think is best

Some that are in different compute functions or parts of deployment will remain separate