Merge pull request #1074 from Dexterp37/source_tags

Bug 1634064 - Implement the 'X-Source-Tags' header

docs/user/debugging/android.md

@@ -21,7 +21,8 @@ In the above:
 |---|----|-----------|
 | `logPings` | boolean (`--ez`)  | If set to `true`, pings are dumped to logcat; defaults to `false` |
 | `sendPing` | string (`--es`)  | Sends the ping with the given name immediately |
-| `tagPings` | string (`--es`)  | Tags all outgoing pings as debug pings to make them available for real-time validation, on the [Glean Debug View](./debug-ping-view.md). The value must match the pattern `[a-zA-Z0-9-]{1,20}` |
+| `debugViewTag` | string (`--es`)  | Tags all outgoing pings as debug pings to make them available for real-time validation, on the [Glean Debug View](./debug-ping-view.md). The value must match the pattern `[a-zA-Z0-9-]{1,20}`. **Important**: in older versions of the Glean SDK, this was named `tagPings` |
+| `sourceTags` | string array (`--esa`)  | Tags outgoing pings with a maximum of 5 comma-separated tags. The tags must match the pattern `[a-zA-Z0-9-]{1,20}`. The `automation` tag is meant for tagging pings generated on automation: such pings will be specially handled on the pipeline (i.e. discarded from [non-live views](https://docs.telemetry.mozilla.org/cookbooks/bigquery/querying.html#table-layout-and-naming)). Tags starting with `glean` are reserved for future use. Subsequent calls of this overwrite any previously stored tag |
 
 All [the options](https://developer.android.com/studio/command-line/adb#am) provided to start the activity are passed over to the main activity for the application to process.
 This is useful if SDK users wants to debug telemetry while providing additional options to the product to enable specific behaviors.  
@@ -34,7 +35,7 @@ For example, to direct a release build of the Glean sample application to (1) du
 adb shell am start -n org.mozilla.samples.gleancore/mozilla.telemetry.glean.debug.GleanDebugActivity \
   --ez logPings true \
   --es sendPing metrics \
-  --es tagPings test-metrics-ping
+  --es debugViewTag test-metrics-ping
 ```
 
 The `logPings` command doesn't trigger ping submission and you won't see any output until a ping has been sent. You can use the `sendPing` command to force a ping to be sent, but it could be more desirable to trigger the pings submission on their normal schedule. For instance, the `baseline` and `events` pings can be triggered by moving the app out of the foreground and the `metrics` ping can be triggered normally if it is overdue for the current calendar day.
@@ -54,7 +55,7 @@ persist until the application is closed or manually reset.
 > adb shell am start -n org.mozilla.samples.gleancore/mozilla.components.service.glean.debug.GleanDebugActivity \
 >   --ez logPings true \
 >   --es sendPing metrics \
->   --es tagPings test-metrics-ping
+>   --es debugViewTag test-metrics-ping
 > ```
 
 ### Glean Log messages

docs/user/debugging/index.md

@@ -10,11 +10,12 @@
 
 ### Available commands and query format
 
-There are 3 available commands that you can use with the Glean SDK debug tools
+There are 4 available commands that you can use with the Glean SDK debug tools
 
 - `logPings`: This is either true or false and will cause pings that are submitted to also be echoed to the device's log.
-- `tagPings`: This command will tag outgoing pings with the provided value, in order to identify them in the Glean Debug View. Tags need to be string with upper and lower case letters, numbers and dashes, with a max length of 20 characters.
+- `debugViewTag`: This command will tag outgoing pings with the provided value, in order to identify them in the Glean Debug View. Tags need to be string with upper and lower case letters, numbers and dashes, with a max length of 20 characters.
 - `sendPing`: This command expects a string name of a ping to force immediate collection and submission of.
+- `sourceTags`: Tags outgoing pings with a maximum of 5, comma-separated, tags.
 
 Different platforms have different ways to send these commands.
 
@@ -24,7 +25,9 @@ Some of the debugging features described above may also be enabled using environ
 
 - `logPings`: May be set by the `GLEAN_LOG_PINGS` environment variable. The accepted values are
 `true` or `false`. Any other value will be ignored.
-- `tagPings`: May be set by the `GLEAN_DEBUG_VIEW_TAG` environment variable. Any valid HTTP header value maybe set here
+- `debugViewTag`: May be set by the `GLEAN_DEBUG_VIEW_TAG` environment variable. Any valid HTTP header value maybe set here
+(e.g. any value that matches the regex `[a-zA-Z0-9-]{1,20}`). Invalid values will be ignored.
+- `sourceTags`: May be set by the `GLEAN_SOURCE_TAGS` environment variable. A comma-separated list of valid HTTP header values may be set here
 (e.g. any value that matches the regex `[a-zA-Z0-9-]{1,20}`). Invalid values will be ignored.
 
 These variables must be set at runtime, not at compile time. They will be checked upon Glean initialization.

docs/user/debugging/ios.md

@@ -17,7 +17,7 @@ For debugging and validation purposes on iOS, Glean makes use of a custom URL sc
 There are 3 available commands that you can use with the Glean SDK debug tools
 
 - `logPings`: This is either true or false and will cause pings that are submitted to also be echoed to the device's log
-- `tagPings`: This command will tag outgoing pings with the provided value, in order to identify them in the Glean Debug View. Tags need to be string with upper and lower case letters, numbers and dashes, with a max length of 20 characters.
+- `debugViewTag`: This command will tag outgoing pings with the provided value, in order to identify them in the Glean Debug View. Tags need to be string with upper and lower case letters, numbers and dashes, with a max length of 20 characters. **Important**: in older versions of the Glean SDK, this was named `tagPings`.
 - `sendPing`: This command expects a string name of a ping to force immediate collection and submission of.
 
 The structure of the custom URL uses the following format:
@@ -90,7 +90,7 @@ Perhaps the simplest way to invoke the Glean SDK debug functionality is to open
 Using the glean-sample-app as an example: to activate ping logging, tag the pings to go to the Glean Debug View, and force the `events` ping to be sent, enter the following URL in a web browser on the iOS device:
 
 ```shell
-glean-sample-app://glean?logPings=true&tagPings=My-ping-tag&sendPing=events
+glean-sample-app://glean?logPings=true&debugViewTag=My-ping-tag&sendPing=events
 ```
 
 This should cause iOS to prompt you with a dialog asking if you want to open the URL in the Glean Sample App, and if you select "Okay" then it will launch (or resume if it's already running) the application with the indicated commands and parameters and immediately force the collection and submission of the events ping.
@@ -104,7 +104,7 @@ It is also possible to encode the URL into a 2D barcode or QR code and launch th
 This method is useful for testing via the Simulator, which typically requires a Mac with Xcode installed, including the Xcode command line tools.  In order to perform the same command as above with using the browser to input the URL, you can use the following command in the command line terminal of the Mac:
 
 ```shell
-xcrun simctl openurl booted "glean-sample-app://glean?logPings=true&tagPings=My-ping-tag&sendPing=events"
+xcrun simctl openurl booted "glean-sample-app://glean?logPings=true&debugViewTag=My-ping-tag&sendPing=events"
 ```
 
 This will launch the simulator and again prompt the user with a dialog box asking if you want to open the URL in the Glean Sample App (or whichever app you are instrumenting and testing).

docs/user/pings/index.md

@@ -104,6 +104,8 @@ A pre-defined set of headers is additionally sent along with the submitted ping:
 | `Date` | e.g. `Mon, 23 Jan 2019 10:10:10 GMT+00:00` | Submission date/time in GMT/UTC+0 offset |
 | `X-Client-Type` | `Glean` | Custom header to support handling of Glean pings in the legacy pipeline |
 | `X-Client-Version` | e.g. `0.40.0` | The Glean SDK version, sent as a custom header to support handling of Glean pings in the legacy pipeline |
+| `X-Debug-ID` | *Optional*, e.g. `test-tag` | Debug header attached to Glean pings when using the [debug tools](../../user/debugging/index.md) |
+| `X-Source-Tags` | *Optional*, e.g. `automation, perf` | A list of tags to associate with the ping, useful for clustering pings at analysis time, for example to tell data generated from CI from other data. |
 
 
 ## Defining foreground and background state

glean-core/android/src/main/java/mozilla/telemetry/glean/Glean.kt

@@ -81,6 +81,9 @@ open class GleanInternalAPI internal constructor () {
     // Keep track of this value before Glean is initialized
     private var logPings: Boolean = false
 
+    // Keep track of source tags if set before Glean is initialized.
+    private var sourceTags: Set<String>? = null
+
     // This object holds data related to any persistent information about the metrics ping,
     // such as the last time it was sent and the store name
     internal lateinit var metricsPingScheduler: MetricsPingScheduler
@@ -186,6 +189,10 @@ open class GleanInternalAPI internal constructor () {
                 setLogPings(logPings)
             }
 
+            // The source tags might have been set before initialize,
+            // get the cached value and set them.
+            sourceTags?.let { setSourceTags(it) }
+
             // Get the current value of the dirty flag so we know whether to
             // send a dirty startup baseline ping below.  Immediately set it to
             // `false` so that dirty startup pings won't be sent if Glean
@@ -634,6 +641,28 @@ open class GleanInternalAPI internal constructor () {
         }
     }
 
+    /**
+     * Set the source tags to be applied as headers when uploading pings.
+     *
+     * If any of the tags is invalid nothing will be set and this function will
+     * return `false`, although if we are not initialized yet, there won't be any validation.
+     *
+     * This is only meant to be used internally by the `GleanDebugActivity`.
+     *
+     * @param tags A list of tags, which must be valid HTTP header values.
+     */
+    internal fun setSourceTags(tags: Set<String>): Boolean {
+        return if (isInitialized()) {
+            val tagList = StringArray(tags.toList().toTypedArray(), "utf-8")
+            LibGleanFFI.INSTANCE.glean_set_source_tags(tagList, tags.size).toBoolean()
+        } else {
+            sourceTags = tags
+            // When setting the source tags before initialization,
+            // we don't validate the tags, thus this function always returns true.
+            true
+        }
+    }
+
     /**
      * Set the logPing debug option, when this is `true`
      * the payload of assembled ping requests get logged.

glean-core/android/src/main/java/mozilla/telemetry/glean/debug/GleanDebugActivity.kt

@@ -37,7 +37,14 @@ class GleanDebugActivity : Activity() {
          * Tags all outgoing pings as debug pings to make them available for real-time validation.
          * The value must match the pattern `[a-zA-Z0-9-]{1,20}`.
          */
-        const val TAG_DEBUG_VIEW_EXTRA_KEY = "tagPings"
+        const val TAG_DEBUG_VIEW_EXTRA_KEY = "debugViewTag"
+        const val LEGACY_TAG_PINGS = "tagPings"
+
+        /**
+         * Tags all outgoing pings as debug pings to make them available for real-time validation.
+         * The value must match the pattern `[a-zA-Z0-9-]{1,20}`.
+         */
+        const val SOURCE_TAGS_KEY = "sourceTags"
     }
 
     // IMPORTANT: These activities are unsecured, and may be triggered by
@@ -49,6 +56,7 @@ class GleanDebugActivity : Activity() {
     /**
      * On creation of the debug activity, launch the requested command.
      */
+    @Suppress("ComplexMethod")
     override fun onCreate(savedInstanceState: Bundle?) {
         super.onCreate(savedInstanceState)
 
@@ -68,8 +76,10 @@ class GleanDebugActivity : Activity() {
         }
 
         // Make sure that at least one of the supported commands was used.
-        val supportedCommands =
-            listOf(SEND_PING_EXTRA_KEY, LOG_PINGS_EXTRA_KEY, TAG_DEBUG_VIEW_EXTRA_KEY)
+        val supportedCommands = listOf(
+            SEND_PING_EXTRA_KEY, LOG_PINGS_EXTRA_KEY,
+            TAG_DEBUG_VIEW_EXTRA_KEY, SOURCE_TAGS_KEY, LEGACY_TAG_PINGS
+        )
 
         // Enable debugging options and start the application.
         intent.extras?.let {
@@ -86,15 +96,28 @@ class GleanDebugActivity : Activity() {
             // Set the debug view tag, if the tag is invalid it won't be set
             debugViewTag?.let {
                 Glean.setDebugViewTag(debugViewTag)
+            } ?: run {
+                // If the 'debugViewTag' was not used, try to look for the legacy
+                // way of setting debug view tags. We should leave this block of
+                // code around at most until December 2020.
+                intent.getStringExtra(LEGACY_TAG_PINGS)?.let { legacyTag ->
+                    Glean.setDebugViewTag(legacyTag)
+                }
             }
 
             val logPings: Boolean? = intent.getBooleanExtra(LOG_PINGS_EXTRA_KEY, false)
             logPings?.let {
                 Glean.setLogPings(logPings)
             }
 
-            intent.getStringExtra(SEND_PING_EXTRA_KEY)?.let {
-                Glean.submitPingByName(it)
+            intent.getStringArrayExtra(SOURCE_TAGS_KEY)?.let { tags ->
+                Glean.setSourceTags(tags.toSet())
+            }
+
+            // Important: this should be applied as the last one, so that
+            // any other option will affect the ping submission as well.
+            intent.getStringExtra(SEND_PING_EXTRA_KEY)?.let { name ->
+                Glean.submitPingByName(name)
             }
         }
 

glean-core/android/src/main/java/mozilla/telemetry/glean/rust/LibGleanFFI.kt

@@ -564,6 +564,8 @@ internal interface LibGleanFFI : Library {
 
     fun glean_set_log_pings(value: Byte)
 
+    fun glean_set_source_tags(raw_tags: StringArray, raw_tags_count: Int): Byte
+
     // Misc
 
     fun glean_str_free(ptr: Pointer)

glean-core/android/src/test/java/mozilla/telemetry/glean/debug/GleanDebugActivityTest.kt

@@ -28,6 +28,8 @@ import mozilla.telemetry.glean.net.HeadersList
 import mozilla.telemetry.glean.net.PingUploader
 import mozilla.telemetry.glean.net.UploadResult
 import mozilla.telemetry.glean.net.HttpResponse
+import mozilla.telemetry.glean.private.NoReasonCodes
+import mozilla.telemetry.glean.private.PingType
 import mozilla.telemetry.glean.testing.GleanTestRule
 import org.junit.Rule
 import org.robolectric.Shadows.shadowOf
@@ -38,13 +40,20 @@ import java.util.concurrent.TimeUnit
  */
 private class TestPingTagClient(
     private val responseUrl: String = Configuration.DEFAULT_TELEMETRY_ENDPOINT,
-    private val debugHeaderValue: String? = null
+    private val debugHeaderValue: String? = null,
+    private val sourceTagsValue: Set<String>? = null
 ) : PingUploader {
     override fun upload(url: String, data: ByteArray, headers: HeadersList): UploadResult {
         assertTrue("URL must be redirected for tagged pings",
             url.startsWith(responseUrl))
-        assertEquals("Debug headers must match what the ping tag was set to",
-            debugHeaderValue, headers.find { it.first == "X-Debug-ID" }!!.second)
+        debugHeaderValue?.let {
+            assertEquals("The debug view header must match what the ping tag was set to",
+                debugHeaderValue, headers.find { it.first == "X-Debug-ID" }!!.second)
+        }
+        sourceTagsValue?.let {
+            assertEquals("The source tags header must match what the ping tag was set to",
+                sourceTagsValue.joinToString(","), headers.find { it.first == "X-Source-Tags" }!!.second)
+        }
 
         return HttpResponse(200)
     }
@@ -140,7 +149,7 @@ class GleanDebugActivityTest {
     }
 
     @Test
-    fun `tagPings filters ID's that don't match the pattern`() {
+    fun `debugViewTag filters ID's that don't match the pattern`() {
         val server = getMockWebServer()
 
         val context = ApplicationProvider.getApplicationContext<Context>()
@@ -189,35 +198,66 @@ class GleanDebugActivityTest {
     }
 
     @Test
-    fun `pings are correctly tagged using tagPings`() {
-        val pingTag = "test-debug-ID"
+    fun `pings are correctly tagged using legacy tagPings`() {
+        val pingTag = "legacy-debug-ID"
 
         // Use the test client in the Glean configuration
         val context = ApplicationProvider.getApplicationContext<Context>()
         resetGlean(context, Glean.configuration.copy(
             httpClient = TestPingTagClient(debugHeaderValue = pingTag)
         ))
 
-        // Put some metric data in the store, otherwise we won't get a ping out
-        // Define a 'booleanMetric' boolean metric, which will be stored in "store1"
-        val booleanMetric = BooleanMetricType(
-            disabled = false,
-            category = "telemetry",
-            lifetime = Lifetime.Application,
-            name = "boolean_metric",
-            sendInPings = listOf("metrics")
+        // Create a custom ping for testing. Since we're testing headers,
+        // it's fine for this to be empty.
+        val customPing = PingType<NoReasonCodes>(
+            name = "custom",
+            includeClientId = false,
+            sendIfEmpty = true,
+            reasonCodes = listOf()
         )
 
-        booleanMetric.set(true)
-        assertTrue(booleanMetric.testHasValue())
-
         // Set the extra values and start the intent.
         val intent = Intent(ApplicationProvider.getApplicationContext<Context>(),
             GleanDebugActivity::class.java)
         intent.putExtra(GleanDebugActivity.SEND_PING_EXTRA_KEY, "metrics")
-        intent.putExtra(GleanDebugActivity.TAG_DEBUG_VIEW_EXTRA_KEY, pingTag)
+        intent.putExtra(GleanDebugActivity.LEGACY_TAG_PINGS, pingTag)
+        launch<GleanDebugActivity>(intent)
+
+        customPing.submit()
+
+        // This will trigger the call to `fetch()` in the TestPingTagClient which is where the
+        // test assertions will occur
+        triggerWorkManager(context)
+    }
+
+    @Test
+    fun `pings are correctly tagged using sourceTags`() {
+        val testTags = setOf("tag1", "tag2")
+
+        // Use the test client in the Glean configuration
+        val context = ApplicationProvider.getApplicationContext<Context>()
+        resetGlean(context, Glean.configuration.copy(
+                httpClient = TestPingTagClient(sourceTagsValue = testTags)
+        ))
+
+        // Create a custom ping for testing. Since we're testing headers,
+        // it's fine for this to be empty.
+        val customPing = PingType<NoReasonCodes>(
+            name = "custom",
+            includeClientId = false,
+            sendIfEmpty = true,
+            reasonCodes = listOf()
+        )
+
+        // Set the extra values and start the intent.
+        val intent = Intent(ApplicationProvider.getApplicationContext<Context>(),
+                GleanDebugActivity::class.java)
+        intent.putExtra(GleanDebugActivity.SEND_PING_EXTRA_KEY, "metrics")
+        intent.putExtra(GleanDebugActivity.SOURCE_TAGS_KEY, testTags.toTypedArray())
         launch<GleanDebugActivity>(intent)
 
+        customPing.submit()
+
         // This will trigger the call to `fetch()` in the TestPingTagClient which is where the
         // test assertions will occur
         triggerWorkManager(context)

glean-core/ffi/glean.h

@@ -690,6 +690,8 @@ void glean_set_experiment_inactive(FfiStr experiment_id);
 
 void glean_set_log_pings(uint8_t value);
 
+uint8_t glean_set_source_tags(RawStringArray raw_tags, int32_t tags_count);
+
 void glean_set_upload_enabled(uint8_t flag);
 
 /**

glean-core/ffi/src/lib.rs

@@ -441,4 +441,12 @@ pub extern "C" fn glean_set_log_pings(value: u8) {
     with_glean_mut(|glean| Ok(glean.set_log_pings(value != 0)));
 }
 
+#[no_mangle]
+pub extern "C" fn glean_set_source_tags(raw_tags: RawStringArray, tags_count: i32) -> u8 {
+    with_glean_mut(|glean| {
+        let tags = from_raw_string_array(raw_tags, tags_count)?;
+        Ok(glean.set_source_tags(tags))
+    })
+}
+
 define_string_destructor!(glean_str_free);