Allow log message formatting to be controlled programmatically

[martincostello]

Describe the Feature

A configuration setting of some kind that allows the formatting type for logs emitted by a lambda to be set in code, rather than relying on the AWS_LAMBDA_HANDLER_LOG_FORMAT environment variable.

Is your Feature Request related to a problem?

As described in #973 (comment), we have an internal Lambda library that uses the Amazon.Lambda.RuntimeSupport library on which we build internal applications to reduce the amount of boilerplate required to build AWS Lambda functions using .NET via a custom runtime so we can use .NET 6 separately to the built-in .NET runtimes. This functionality predates the availability of Lambda's container image support.

As part of our migration to .NET 6, our log ingestion that uses a JSON format was broken due to the log messages being prepended with the timestamp, log level and AWS request ID.

Making this formatting option available in code would allow us to set this up in our library and roll it out transparently via NuGet package update automation, rather than requiring the additional Lambda configuration to also set the environment variable.

Proposed Solution

A public API surface area (maybe an enum and a property on a class somewhere) that allows the log format to be controlled from code without infrastructure changes (i.e. setting the environment variable).

Describe alternatives you've considered

Setting the environment variable using Environment.SetEnvironmentVariable(string, string), but this seemed a bit hacky to me compared to a first-class API surface, and also I wasn't sure if it would take effect in the AWS runtime environment or not once the process had initialized.

Additional Context

The original change broke our structured logging due to it not being in JSON format any more. The workaround via the environment variable in #973 works to resolve this, but requires it to be set on every individual function we have.

Environment

N/A

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[ashishdhingra]

@normj Related to #973 to expose the setting programatically.

[normj]

@martincostello Is this a nice to have or a significant blocker? I'm trying to have this code locked for the upcoming managed runtime.

Right now the environment variable is evaluated once during startup so changing the environment variable in your Lambda code would have no effect. That being said it would be easier to implement the environment variable being checked every time. The perf difference would be insignificant since environment variables are just a dictionary lookup. I could also slip this in later if I can't get this done in time for the upcoming managed runtime.

Making a dedicated API is not as simple because the API would have to go on Amazon.Lambda.Core since in a managed runtime you don't have access to Amazon.Lambda.RuntimeSupport. The Amazon.Lambda.Core sort of represents the common Lambda programming model so if we add APIs there then we are affectively extending the Lambda programming model and we have to consider what this means for other runtimes. Also since we would have to change Amazon.Lambda.Core we would need to get this done before the managed runtime. It is very hard to add new methods to Amazon.Lambda.Core that affect the runtime client behavior after the managed runtime has launched.

[martincostello]

@normj This isn't a blocker - we've gone down the environment variable route for now in our serverless configuration, it's just more to update (and remember to set for new functions) than if we could just set it centrally via the code in our base library.

It might not be 100% of them, but the vast majority of our .NET Lambda functions will have relied on the previous logging behaviour to send the log message as-is so we could send the message in JSON format to be ingested into our logging cluster.

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.