Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1655477 Support X-Source-Tags header #1328

Merged
merged 1 commit into from
Aug 3, 2020
Merged

Bug 1655477 Support X-Source-Tags header #1328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/architecture/decoder_service_specification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ required group attributes {
optional string dnt // header from client
optional string x_pingsender_version // header from client
optional string x_debug_id // header from client
optional string x_source_tags // header from client
optional string user_agent_browser // from user_agent
optional string user_agent_browser_version // from user_agent
optional string user_agent_os // from user_agent
Expand Down
3 changes: 2 additions & 1 deletion ingestion-beam/src/main/java/com/mozilla/telemetry/decoder/AddMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ public class AddMetadata {

private static final String HEADER = "header";
private static final List<String> HEADER_ATTRIBUTES = ImmutableList //
.of(Attribute.DATE, Attribute.DNT, Attribute.X_PINGSENDER_VERSION, Attribute.X_DEBUG_ID);
.of(Attribute.DATE, Attribute.DNT, Attribute.X_PINGSENDER_VERSION, Attribute.X_DEBUG_ID,
Attribute.X_SOURCE_TAGS);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming here that we'll include x_debug_id as a simple string field in BigQuery, and potentially update user-facing views to transform the string to a list. This seems cleaner than trying to parse the field to a list within the pipeline.


private static final List<String> URI_ATTRIBUTES = ImmutableList //
.of(Attribute.URI, Attribute.APP_NAME, Attribute.APP_VERSION, Attribute.APP_UPDATE_CHANNEL,
Expand Down
10 changes: 6 additions & 4 deletions ingestion-beam/src/test/java/com/mozilla/telemetry/decoder/AddMetadataTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,18 +126,20 @@ public void testPutUserAgentAttributes() throws Exception {
public void testHeadersFromAttributes() throws Exception {
Map<String, String> attributes = ImmutableMap.of("dnt", "1", //
"sample_id", "18", //
"x_source_tags", "automation, perf", //
"x_debug_id", "mysession");
ObjectNode headers = AddMetadata.headersFromAttributes(attributes);
ObjectNode expected = mapToObjectNode(ImmutableMap.of("dnt", "1", "x_debug_id", "mysession"));
ObjectNode expected = mapToObjectNode(ImmutableMap.of("dnt", "1", "x_debug_id", "mysession",
"x_source_tags", "automation, perf"));
assertEquals(expected, headers);
}

@Test
public void testPutHeaderAttributes() throws Exception {
ObjectNode metadata = mapToObjectNode(
ImmutableMap.of("header", ImmutableMap.of("dnt", "1", "x_debug_id", "mysession")));
ObjectNode metadata = mapToObjectNode(ImmutableMap.of("header", ImmutableMap.of("dnt", "1",
"x_debug_id", "mysession", "x_source_tags", "automation, perf")));
Map<String, String> expected = ImmutableMap.of("dnt", "1", //
"x_debug_id", "mysession");
"x_debug_id", "mysession", "x_source_tags", "automation, perf");
Map<String, String> attributes = new HashMap<>();
AddMetadata.putHeaderAttributes(attributes, (metadata));
assertEquals(expected, attributes);
Expand Down
1 change: 1 addition & 0 deletions ingestion-core/src/main/java/com/mozilla/telemetry/ingestion/core/Constant.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ private Attribute() {
public static final String X_FORWARDED_FOR = "x_forwarded_for";
public static final String X_PINGSENDER_VERSION = "x_pingsender_version";
public static final String X_PIPELINE_PROXY = "x_pipeline_proxy";
public static final String X_SOURCE_TAGS = "x_source_tags";
}

public static class FieldName {
Expand Down
1 change: 1 addition & 0 deletions ingestion-edge/ingestion_edge/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ class Route:
"X-Pingsender-Version",
"X-Pipeline-Proxy",
"X-Debug-ID",
"X-Source-Tags",
]

METADATA_HEADERS = {
Expand Down