Structured Logging Handler should be project aware

[averikitsch]

I want to use StructuredLogHandler directly to ensure printing to stdout regardless of location, especially locally. I want my project Id to be automatically inferred so the trace ID includes my project ID ("logging.googleapis.com/trace": "projects/PROJECT_ID/traces/abcdef",).

Neither locally or remotely, will the project id be set automatically:

    logging_client = Client()
    from google.cloud.logging_v2.handlers import StructuredLogHandler, setup_logging
    handler = StructuredLogHandler()
    setup_logging(handler)

    import logging

    msg = "Hello, world!"
    metadata = {"component": "arbitrary-property", "severity":"error"}
    logging.info(msg, extra={"json_fields": metadata})

The solution is to set the project ID:

handler = StructuredLogHandler(project_id="PROJECT_ID")
setup_logging(handler)

Or use CloudLoggingHandler (note this doesn't print to stdout unless env vars are set):

logging_client = Client()
logging_client.setup_logging()

[daniel-sanche]

Hey Averi,

When running the library on GCP, the logging agent should attach the project_id automatically (I just double checked on GCF to configm). If we let the user pass in their project id manually, that would just confuse things, because ultimately, the log will end up associated with the project the agent is running in, not what they passed in manually

I want to use StructuredLogHandler directly to ensure printing to stdout regardless of location, especially locally.

Can you describe this use-case a bit? Why do you want to run the StructuredLogHandler locally, when there is no agent to actually send the logs to GCP? This is typically where the CloudLoggingHandler would be used

[averikitsch]

I would like to set this specially, so I can ensure that the structured logging format is used. I want to be able to test that the formatting of the logs is correct (example: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/functions/v2/http_logging/main_test.py#L30) and visually see it in my stdout. As a googler, I know that I can force the serverless environment using environment variables, but we shouldn't force our customers to know that.

In the same "environment" specifying the structured logging handler, will change the outcome of the trace field.
(Using sample https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/functions/v2/http_logging/main.py)

# cloud_logging_client.setup_logging()
handler = StructuredLogHandler()
setup_logging(handler)

AssertionError: assert 'projects/pro...traces/abcdef' == 'abcdef'
  - abcdef
  + projects/project-id/traces/abcdef

I have confirmed the trace is not formatted with the project id when using the structured logging handler on GCF gen1/gen2 and Cloud Run

[daniel-sanche]

Ok thanks, I think I understand your use case. As you mentioned, the way to make the handler project-aware is to pass in a project ID on init. If you already have a client, you can grab it from there:

handler = StructuredLogHandler(project_id=client.project)

We could add some code into the StructuredLogHandler to detect the project when it's not passed in. But that could get complicated, would break abstractions, and would be hard to justify.

Maybe we should change the docs to include the project when creating a StructuredLogHandler, so more users see this behaviour by default?