Telemetry for SAM CLI

[sriram-mv]

Description

SAM CLI is adding telemetry. In addition to existing feedback, telemetry allows us to better understand our customer’s needs, common CLI scenarios and existing issues. The information collected will include generic usage metrics, system and environment information, and errors (see below for types of information collected). These generic metrics will inform our product decisions and help us continue to deliver features and enhancements that improve the customer experience. This feature does not collect personal information, such as usernames or email addresses, or extract sensitive project-level information and customers control whether telemetry is enabled and can change their settings at any point of time. To learn more about AWS policies:

• https://aws.amazon.com/service-terms/
• https://aws.amazon.com/compliance/data-privacy-faq/

Experience

Customers new to SAM CLI, upon first use after installation, will be provided with a message prompt informing them that telemetry collection is enabled and will provide the customer with instructions on how to opt out (see instructions for opt out below).
Customers already using SAM CLI, including as a part of automated CI/CD systems, will be provided with the message prompt upon first use after update or next install. If telemetry remains enabled, we will send telemetry data in the background without requiring any additional customer interaction.

NOTE: The message prompt will only appear upon the first update or install after the launch of the telemetry feature, and not upon every update thereafter.

How to opt out

At any point, customers have the option to disable telemetry by setting an environment variable SAM_CLI_TELEMETRY=0 in their shell. Once the customer has opted-out, we will cease collecting telemetry data. Data collected through telemetry adheres to the AWS data privacy policies.

Types of information collected

1. Usage information: generic commands and sub-commands executed.
2. Errors and diagnostic information: Status and duration of command run, including exit code and internal exception name. Failures when connecting to Docker.
3. System and environment information: Python version, Operating system (Windows, Linux or Mac), environment in which SAM CLI is executed (e.g. CodeBuild, AWS IDEToolkit or Terminal).

[henrikbjorn]

Instead of the various environment variables it might be time to have a configuration file. This could also have the benefit of settings defaults for local invoke such as the network to use.

[jfuss]

@henrikbjorn Great call out! We have been toying with a SAMRC file and this could be a central location for everything. The team talked about going down this path but decided to start with an env var for now while we still think more about what SAMRC is and how it looks.

[henrikbjorn]

@jfuss Some companies might have a policy not to allow collection of stats. By having an environment variable to disable this makes it hard to enforce. As it is up to each employee to set the variable instead of a project file with the setting.

Better to have a very simple .samrc and then evolve it over time.

[jfuss]

@henrikbjorn Not sure we considered that honestly, but is a very valid point and use-case. Given where we are, I don't think it is reasonable for the team to get .samrc out for the release. I will bring it up with the team and see if we can't support this in the near future.

[jfuss]

Closing as this was released in v0.19.0.

[adrian-skybaker]

FYI, despite having no issue with the telemetry collection, I've just disabled this in our Azure Devops pipeline purely to suppress the message prompt that spammed the logs every pipeline execution. The persistence mechanism seems unlikely to work well in CI/CD envs which ensure clean agents.