dnst: document new fields

data-formats/df-002-dnst.md

@@ -26,11 +26,15 @@ code. See this directory's [README](README.md) for the basic concepts.
     "dial_id": 177171,
     "engine": "udp",
     "failure": "dns_nxdomain_error",
+    "getaddrinfo_error": 0,
     "hostname": "dns.googlex",
     "query_type": "A",
+    "raw_response": "",
+    "rcode": 0,
     "resolver_address": "8.8.8.8:53",
-    "resolver_hostname": "8.8.8.8",
-    "resolve_port": "53",
+    "resolver_hostname": null,
+    "resolve_port": null,
+    "t0": 1.007,
     "t": 1.114,
     "transaction_id": 1
 }
@@ -52,25 +56,41 @@ of available resolver engines.
 - `failure` (`string`; nullable): if there was an error, this field is
 a string indicating the error, otherwise it MUST be `null`.
 
-- `hostname`: (`string`): the hostname used in the DNS query, which MUST
-be reversed for PTR lookups like `1.0.0.127.in-addr.arpa`.
+- `getaddrinfo_error` (`int`; optional; since 2022-08-23): if the
+underlying engine used `getaddrinfo` and `getaddrinfo` failed, this
+is the system-dependent error code returned by `getaddrinfo`;
 
-- `query_type`: (`string`): a valid DNS query type (e.g. `MX`).
+- `hostname` (`string`): the hostname used in the DNS query, which MUST
+be reversed for PTR lookups like `1.0.0.127.in-addr.arpa.`.
 
-- `resolver_address`: (`string`; since 2019-12-29): more flexible way of
+- `query_type`: (`string`): a valid DNS query type (e.g. `MX`)—see the
+note below regarding representing the system/go resolver results;
+
+- `raw_response`: an optional base64 string containing the bytes of the
+response for the engines that allow us to observe it;
+
+- `rcode`: an optional int64 containing the DNS rcode inside the `raw_response`
+for the engines that allow us to observe it;
+
+- `resolver_address` (`string`; since 2019-12-29): more flexible way of
 specifying the resolver address that also allows for DoH, because it does
 not assume that the resolver is identified by an address, port tuple.
 
-- `resolver_hostname`: (`string`; optional; deprecated): legacy way to
+- `resolver_hostname` (`string`; optional; deprecated): legacy way to
 specify the resolver hostname when using a custom resolver. This is not
 used by ooni/probe-engine, which sets it to `null`.
 
-- `resolver_port`: (`string`; optional; deprecated): legacy way to
+- `resolver_port` (`string`; optional; deprecated): legacy way to
 specify the resolver hostname when using a custom resolver. This is not
 used by ooni/probe-engine, which sets it to `null`.
 
+- `t0` (`float`): number of seconds elapsed since `measurement_start_time`
+measured in the moment in which we started the operation (`t - t0` gives you
+the amount of time spent performing the given lookup);
+
 - `t` (`float`): number of seconds elapsed since `measurement_start_time`
-measured in the moment in which `failure` is determined.
+measured in the moment in which `failure` is determined (`t - t0` gives you
+the amount of time spent performing the given lookup);
 
 - `transaction_id` (`int`; optional; since 2020-01-11): if present, this is the
 ID of the HTTP transaction that caused this query.
@@ -96,6 +116,22 @@ line See [ooni/probe-cli#765](https://github.com/ooni/probe-cli/pull/765)
 and [ooni/probe-cli#766](https://github.com/ooni/probe-cli/pull/766) for
 more details about how we choose the resolver name.
 
+### Representing system/go resolver results
+
+As of 2022-08-23, resolutions performed by the system/go resolver are
+represented in two distinct ways:
+
+1. all mainline experiments artifically split a lookup into two entries, one
+for `A` and the other for `AAAA`;
+
+2. new step-by-step code (which eventually will become the default) represents a
+system/go lookup as a single `ANY` query containing all the `A`, `AAAA`, and
+`CNAME` records returned by `getaddrinfo` (or by the Go resolver).
+
+The latter approach is more correct in terms of representing which low
+level operations occurred, since a `getaddrinfo` is actually a single lookup
+and it's not faithful to reality to fake out two lookups.
+
 ## Answer
 
 ```JSON
@@ -173,10 +209,11 @@ not relevant to the DNS data format:
         "engine": "system",
         "failure": null,
         "hostname": "web.telegram.org",
-        "query_type": "A",
+        "query_type": "ANY",
         "resolver_hostname": null,
         "resolver_port": null,
         "resolver_address": "",
+        "t0": 1.001,
         "t": 1.114,
         "transaction_id": 1
       }