Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support to allow to track the resolution of DNS requests. #721

Merged
merged 16 commits into from
Sep 14, 2020
Merged

Added support to allow to track the resolution of DNS requests. #721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ed90f64
fixed missing logging output (Issue #258)
turban1988 Feb 10, 2019
e37186d
fixed missing logging output (Issue #258)
turban1988 Feb 10, 2019
a8837de
fixed missing logging output (Issue #258)
turban1988 Feb 10, 2019
f28b25a
fixed missing logging output (Issue #258)
turban1988 Feb 10, 2019
f29a271
resolving local conflicts
turban1988 Aug 4, 2020
1fe9dea
merge with latest upstream/master
turban1988 Aug 5, 2020
b410932
merge with latest upstream/master
turban1988 Aug 5, 2020
dfa970f
added DNS resolution support
turban1988 Aug 5, 2020
2278026
updated the demo and reverted one local change in teh HTTP instrument…
turban1988 Aug 5, 2020
0e307af
add dns instrumentation class
turban1988 Aug 5, 2020
12e58b4
removed old (unused?) files
turban1988 Aug 5, 2020
243c54b
added DNS resolution to the arrow/parquet schema
turban1988 Aug 17, 2020
561e80d
added export/import of the ResourceType arry from http-instrument.ts
turban1988 Aug 17, 2020
1c897b0
fixed some permission changes
turban1988 Aug 17, 2020
871edf4
Added test for dns instrument
Sep 11, 2020
e6e4d46
Merge pull request #1 from mozilla/dns_instrument
turban1988 Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions automation/DataAggregator/parquet_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@
]
PQ_SCHEMAS['navigations'] = pa.schema(fields)

# callstacks
fields = [
pa.field('visit_id', pa.int64(), nullable=False),
pa.field('request_id', pa.int64(), nullable=False),
Expand All @@ -197,8 +198,22 @@
]
PQ_SCHEMAS['callstacks'] = pa.schema(fields)

# incomplete_visits
fields = [
pa.field('visit_id', pa.int64(), nullable=False),
pa.field('instance_id', pa.uint32(), nullable=False)
]
PQ_SCHEMAS['incomplete_visits'] = pa.schema(fields)

# dns_responses
fields = [
pa.field('request_id', pa.int64(), nullable=False),
pa.field('crawl_id', pa.uint32(), nullable=False),
pa.field('visit_id', pa.int64(), nullable=False),
pa.field('hostname', pa.string()),
pa.field('addresses', pa.string()),
pa.field('canonical_name', pa.string()),
pa.field('is_TRR', pa.string()),
pa.field('time_stamp', pa.string(),nullable=False)
]
PQ_SCHEMAS['dns_responses'] = pa.schema(fields)
16 changes: 16 additions & 0 deletions automation/DataAggregator/schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,3 +227,19 @@ CREATE TABLE IF NOT EXISTS callstacks(
CREATE TABLE IF NOT EXISTS incomplete_visits (
visit_id INTEGER NOT NULL
);

/*
# DNS Requests
*/
CREATE TABLE IF NOT EXISTS dns_responses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
request_id INTEGER NOT NULL,
crawl_id INTEGER NOT NULL,
visit_id INTEGER NOT NULL,
hostname TEXT,
addresses TEXT,
used_address TEXT,
canonical_name TEXT,
is_TRR INTEGER,
time_stamp DATETIME NOT NULL
);
9 changes: 8 additions & 1 deletion automation/Extension/firefox/feature.js/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ import {
CookieInstrument,
JavascriptInstrument,
HttpInstrument,
NavigationInstrument
NavigationInstrument,
DnsInstrument
} from "openwpm-webext-instrumentation";

import * as loggingDB from "./loggingdb.js";
Expand Down Expand Up @@ -84,6 +85,12 @@ async function main() {
let callstackInstrument = new CallstackInstrument(loggingDB);
callstackInstrument.run(config['crawl_id']);
}

if (config['dns_instrument']) {
loggingDB.logDebug("DNS instrumentation enabled");
let dnsInstrument = new DnsInstrument(loggingDB);
dnsInstrument.run(config['crawl_id']);
}
}

main();
Expand Down
12 changes: 6 additions & 6 deletions automation/Extension/firefox/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion automation/Extension/firefox/src/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
"storage",
"alarms",
"downloads",
"tabs"
"tabs",
"dns"
],

"experiment_apis": {
Expand Down
Loading