feat: http command #47

patrykcieszkowski · 2022-05-17T17:51:43Z

http handler
DNS resolver
unit tests

resolve jsdelivr/globalping#128

jimaek · 2022-05-24T19:35:06Z

I dont think the output has everything we need, do you have a sample?

This is what we need:

performance meta-data: dns lookup, tcp connection, tls handshake, time to first byte, download, total time
tls certificate meta-data like issuer, domain, issue and expiration date, valid bool...

patrykcieszkowski · 2022-05-24T19:42:57Z

do you have a sample?

heres the current output:

{
  "headers": {
    "connection": "close",
    "cache-control": "private",
    "content-type": "text/html; charset=utf-8",
    "content-encoding": "gzip",
    "strict-transport-security": "max-age=15552000",
    "x-frame-options": "SAMEORIGIN",
    "x-request-guid": "1706d2e1-63fd-49a8-b15b-b09b92c577e4",
    "feature-policy": "microphone 'none'; speaker 'none'",
    "content-security-policy": "upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com",
    "accept-ranges": "bytes",
    "date": "Tue, 24 May 2022 19:42:03 GMT",
    "via": "1.1 varnish",
    "x-served-by": "cache-lhr7383-LHR",
    "x-cache": "MISS",
    "x-cache-hits": "0",
    "x-timer": "S1653421323.041443,VS0,VE73",
    "vary": "Accept-Encoding,Fastly-SSL",
    "x-dns-prefetch-control": "off",
    "set-cookie": [
      "prov=82b27b3b-8094-d9b3-4d53-6140c4ab85f6; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly"
    ]
  },
  "rawHeaders": "Connection: close\ncache-control: private\ncontent-type: text/html; charset=utf-8\ncontent-encoding: gzip\nstrict-transport-security: max-age=15552000\nx-frame-options: SAMEORIGIN\nx-request-guid: 1706d2e1-63fd-49a8-b15b-b09b92c577e4\nfeature-policy: microphone 'none'; speaker 'none'\ncontent-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com\nAccept-Ranges: bytes\nDate: Tue, 24 May 2022 19:42:03 GMT\nVia: 1.1 varnish\nX-Served-By: cache-lhr7383-LHR\nX-Cache: MISS\nX-Cache-Hits: 0\nX-Timer: S1653421323.041443,VS0,VE73\nVary: Accept-Encoding,Fastly-SSL\nX-DNS-Prefetch-Control: off\nSet-Cookie: prov=82b27b3b-8094-d9b3-4d53-6140c4ab85f6; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly",
  "rawBody": "",
  "statusCode": 200,
  "responseTime": 451,
  "rawOutput": "HTTP/1.1 200\nConnection: close\ncache-control: private\ncontent-type: text/html; charset=utf-8\ncontent-encoding: gzip\nstrict-transport-security: max-age=15552000\nx-frame-options: SAMEORIGIN\nx-request-guid: 1706d2e1-63fd-49a8-b15b-b09b92c577e4\nfeature-policy: microphone 'none'; speaker 'none'\ncontent-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com\nAccept-Ranges: bytes\nDate: Tue, 24 May 2022 19:42:03 GMT\nVia: 1.1 varnish\nX-Served-By: cache-lhr7383-LHR\nX-Cache: MISS\nX-Cache-Hits: 0\nX-Timer: S1653421323.041443,VS0,VE73\nVary: Accept-Encoding,Fastly-SSL\nX-DNS-Prefetch-Control: off\nSet-Cookie: prov=82b27b3b-8094-d9b3-4d53-6140c4ab85f6; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly"
}

jimaek · 2022-05-24T19:51:56Z

Ok, yeah we need to expand the timings array to include everything else. And add a new one for HTTPS/HTTP2 requests about TLS

patrykcieszkowski · 2022-06-01T15:41:43Z

@jimaek

{
    "headers":
    {
        "location": "https://www.google.com/",
        "content-type": "text/html; charset=UTF-8",
        "date": "Wed, 01 Jun 2022 15:36:12 GMT",
        "expires": "Wed, 01 Jun 2022 15:36:12 GMT",
        "cache-control": "private, max-age=2592000",
        "server": "gws",
        "content-length": "220",
        "x-xss-protection": "0",
        "x-frame-options": "SAMEORIGIN",
        "set-cookie":
        [
            "CONSENT=PENDING+009; expires=Fri, 31-May-2024 15:36:12 GMT; path=/; domain=.google.com; Secure"
        ],
        "p3p": "CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"",
        "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
        "connection": "close"
    },
    "rawHeaders": "Location: https://www.google.com/\nContent-Type: text/html; charset=UTF-8\nDate: Wed, 01 Jun 2022 15:36:12 GMT\nExpires: Wed, 01 Jun 2022 15:36:12 GMT\nCache-Control: private, max-age=2592000\nServer: gws\nContent-Length: 220\nX-XSS-Protection: 0\nX-Frame-Options: SAMEORIGIN\nSet-Cookie: CONSENT=PENDING+009; expires=Fri, 31-May-2024 15:36:12 GMT; path=/; domain=.google.com; Secure\nP3P: CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"\nAlt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"\nConnection: close",
    "rawBody": "",
    "statusCode": 301,
    "timings":
    {
        "firstByte": 32,
        "dns": 27,
        "response": 128,
        "connect": 57,
        "total": 131,
        "download": 3
    },
    "tls":
    {
        "authorized": true,
        "createdAt": "May  9 08:32:32 2022 GMT",
        "expireAt": "Aug  1 08:32:31 2022 GMT",
        "issuer":
        {
            "C": "US",
            "O": "Google Trust Services LLC",
            "CN": "GTS CA 1C3"
        },
        "subject":
        {
            "CN": "*.google.com",
            "alt": "DNS:*.google.com, DNS:*.appengine.google.com, DNS:*.bdn.dev, DNS:*.cloud.google.com, DNS:*.crowdsource.google.com, DNS:*.datacompute.google.com, DNS:*.google.ca, DNS:*.google.cl, DNS:*.google.co.in, DNS:*.google.co.jp, DNS:*.google.co.uk, DNS:*.google.com.ar, DNS:*.google.com.au, DNS:*.google.com.br, DNS:*.google.com.co, DNS:*.google.com.mx, DNS:*.google.com.tr, DNS:*.google.com.vn, DNS:*.google.de, DNS:*.google.es, DNS:*.google.fr, DNS:*.google.hu, DNS:*.google.it, DNS:*.google.nl, DNS:*.google.pl, DNS:*.google.pt, DNS:*.googleadapis.com, DNS:*.googleapis.cn, DNS:*.googlevideo.com, DNS:*.gstatic.cn, DNS:*.gstatic-cn.com, DNS:googlecnapps.cn, DNS:*.googlecnapps.cn, DNS:googleapps-cn.com, DNS:*.googleapps-cn.com, DNS:gkecnapps.cn, DNS:*.gkecnapps.cn, DNS:googledownloads.cn, DNS:*.googledownloads.cn, DNS:recaptcha.net.cn, DNS:*.recaptcha.net.cn, DNS:recaptcha-cn.net, DNS:*.recaptcha-cn.net, DNS:widevine.cn, DNS:*.widevine.cn, DNS:ampproject.org.cn, DNS:*.ampproject.org.cn, DNS:ampproject.net.cn, DNS:*.ampproject.net.cn, DNS:google-analytics-cn.com, DNS:*.google-analytics-cn.com, DNS:googleadservices-cn.com, DNS:*.googleadservices-cn.com, DNS:googlevads-cn.com, DNS:*.googlevads-cn.com, DNS:googleapis-cn.com, DNS:*.googleapis-cn.com, DNS:googleoptimize-cn.com, DNS:*.googleoptimize-cn.com, DNS:doubleclick-cn.net, DNS:*.doubleclick-cn.net, DNS:*.fls.doubleclick-cn.net, DNS:*.g.doubleclick-cn.net, DNS:doubleclick.cn, DNS:*.doubleclick.cn, DNS:*.fls.doubleclick.cn, DNS:*.g.doubleclick.cn, DNS:dartsearch-cn.net, DNS:*.dartsearch-cn.net, DNS:googletraveladservices-cn.com, DNS:*.googletraveladservices-cn.com, DNS:googletagservices-cn.com, DNS:*.googletagservices-cn.com, DNS:googletagmanager-cn.com, DNS:*.googletagmanager-cn.com, DNS:googlesyndication-cn.com, DNS:*.googlesyndication-cn.com, DNS:*.safeframe.googlesyndication-cn.com, DNS:app-measurement-cn.com, DNS:*.app-measurement-cn.com, DNS:gvt1-cn.com, DNS:*.gvt1-cn.com, DNS:gvt2-cn.com, DNS:*.gvt2-cn.com, DNS:2mdn-cn.net, DNS:*.2mdn-cn.net, DNS:googleflights-cn.net, DNS:*.googleflights-cn.net, DNS:admob-cn.com, DNS:*.admob-cn.com, DNS:*.gstatic.com, DNS:*.metric.gstatic.com, DNS:*.gvt1.com, DNS:*.gcpcdn.gvt1.com, DNS:*.gvt2.com, DNS:*.gcp.gvt2.com, DNS:*.url.google.com, DNS:*.youtube-nocookie.com, DNS:*.ytimg.com, DNS:android.com, DNS:*.android.com, DNS:*.flash.android.com, DNS:g.cn, DNS:*.g.cn, DNS:g.co, DNS:*.g.co, DNS:goo.gl, DNS:www.goo.gl, DNS:google-analytics.com, DNS:*.google-analytics.com, DNS:google.com, DNS:googlecommerce.com, DNS:*.googlecommerce.com, DNS:ggpht.cn, DNS:*.ggpht.cn, DNS:urchin.com, DNS:*.urchin.com, DNS:youtu.be, DNS:youtube.com, DNS:*.youtube.com, DNS:youtubeeducation.com, DNS:*.youtubeeducation.com, DNS:youtubekids.com, DNS:*.youtubekids.com, DNS:yt.be, DNS:*.yt.be, DNS:android.clients.google.com, DNS:developer.android.google.cn, DNS:developers.android.google.cn, DNS:source.android.google.cn"
        }
    },
    "rawOutput": "HTTP/1.1 301\nLocation: https://www.google.com/\nContent-Type: text/html; charset=UTF-8\nDate: Wed, 01 Jun 2022 15:36:12 GMT\nExpires: Wed, 01 Jun 2022 15:36:12 GMT\nCache-Control: private, max-age=2592000\nServer: gws\nContent-Length: 220\nX-XSS-Protection: 0\nX-Frame-Options: SAMEORIGIN\nSet-Cookie: CONSENT=PENDING+009; expires=Fri, 31-May-2024 15:36:12 GMT; path=/; domain=.google.com; Secure\nP3P: CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"\nAlt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"\nConnection: close"
}

jimaek · 2022-06-01T15:56:46Z

And we include the private IPs protection without warming up the cache right? The output looks good to me, but I guess I will have to test it better in prod

patrykcieszkowski · 2022-06-01T16:03:20Z

And we include the private IPs protection without warming up the cache right?

We query the DNS, it responds with an array of records. We filter out all private IPs. If there's none left - it throws an error. If there are records left - we pick the first one.

ENODATA google.com

jimaek · 2022-06-01T16:43:31Z

Then the warm cache issue persists, doesn't it?
The solution I see to this is to time the DNS latency during that special DNS query and re-use that timing instead the one "got" provides.

patrykcieszkowski · 2022-06-01T16:48:35Z

Then the warm cache issue persists, doesn't it?

we explicitly disable cache on HTTP requests. DNS wise - I don't believe node DNS resolver caches anything.

jimaek · 2022-06-01T16:50:41Z

Linux caches the query on a system level as far as I know. Plus it gets cached by the resolver you hit. So you pre-warm it in 2 places.

patrykcieszkowski · 2022-06-01T16:52:10Z

Well, it does but how would you know whether the DNS result was cached or not?

jimaek · 2022-06-01T16:55:30Z

You dont know, you see it based on the results. If you select a less popular domain you will see that the first 1-2 queries are slow, e.g. 10ms and the ones after are fast, e.g. 3ms.

But the whole point of the service is to benchmark and test services to improve your performance. So if users cant get reliable readings then its a problem

jimaek · 2022-06-01T16:56:45Z

[root@ansible ~]# dig @8.8.8.8 www-jsdelivr-com-pr-400.onrender.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.9 <<>> @8.8.8.8 www-jsdelivr-com-pr-400.onrender.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9005
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www-jsdelivr-com-pr-400.onrender.com. IN A

;; ANSWER SECTION:
www-jsdelivr-com-pr-400.onrender.com. 300 IN CNAME www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net.
www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net. 300 IN A 216.24.57.3
www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net. 300 IN A 216.24.57.253

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Jun 01 16:56:08 UTC 2022
;; MSG SIZE  rcvd: 166

[root@ansible ~]# dig @8.8.8.8 www-jsdelivr-com-pr-400.onrender.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.9 <<>> @8.8.8.8 www-jsdelivr-com-pr-400.onrender.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1035
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www-jsdelivr-com-pr-400.onrender.com. IN A

;; ANSWER SECTION:
www-jsdelivr-com-pr-400.onrender.com. 291 IN CNAME www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net.
www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net. 291 IN A 216.24.57.253
www-jsdelivr-com-pr-400.onrender.com.cdn.cloudflare.net. 291 IN A 216.24.57.3

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Jun 01 16:56:11 UTC 2022
;; MSG SIZE  rcvd: 166

20ms no cache vs 6ms cached

patrykcieszkowski · 2022-06-06T13:35:44Z

But the problem is, we can't distinguish which reading is cached, and which wasn't. We could probably store the largest value, and update it accordingly on each reading, but that will also be just guess work on our end. It won't be reliable.

jimaek · 2022-06-06T17:26:03Z

Thats not the point, we shouldn't do anything to decide anything.
We need:

To stop making 2 DNS queries per test
If not possible then use the query time of the first query.

patrykcieszkowski · 2022-06-06T17:44:01Z

We're making one query.

jimaek · 2022-06-06T17:51:57Z

1 query in the custom implementation to block private IPs and 1 query to get the timing data. That's 2. Can you explain how it works in more detail?

patrykcieszkowski · 2022-06-06T17:55:53Z

1 query in the custom implementation to block private IPs and 1 query to get the timing data. That's 2. Can you explain how it works in more detail?

Wrong. There's just one DNS query, and its executed by GOT. We've replaced the original DNS method with our custom handler - the returned timing is calculated by got on it's original query.

github-actions · 2022-06-07T14:27:52Z

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

add: http command

47ed4a9

patrykcieszkowski mentioned this pull request May 17, 2022

feat: http command jsdelivr/globalping#128

Merged

3 tasks

patrykcieszkowski added 15 commits May 18, 2022 09:30

ref: unecessary method remap

d475531

add: dns lookup method + private IP check

6dfa2f6

ref: rename result publishing method name

431f4c3

add: dnslookup: support dnsServer override

1c2d9c9

ref: cleanup

acdd28b

ref: making deepsource happy

e9fc0c1

fix: dnslookup: support optional ipv6 (default ipv4)

c9fdf99

ref: prepare dnsLookup for unit tests

3307f45

ref: making eslint + deepsource happy

f6a1187

ref: making eslint + deepsource happy

6bfea68

fix: hard-coded limit

42b2175

add: unit test dns lookup (filter private ip)

6fd291b

Add: http unit tests

93b5167

add: httpCmd: support optional resolverFn argument

0e31020

fix: making deepsource happy

a68321f

patrykcieszkowski requested a review from zarianec May 19, 2022 11:55

patrykcieszkowski changed the title ~~wip: http command~~ feat: http command May 19, 2022

patrykcieszkowski added 3 commits May 19, 2022 19:07

ref: test: rename http dns helper filename

5bec837

add: http version +_response time data

74a361e

add: hide :status header

9b6a97e

ref: dns: use promises + add callbackify method

defcaad

patrykcieszkowski force-pushed the http-command branch from 4c0e00c to defcaad Compare May 27, 2022 02:45

patrykcieszkowski added 3 commits May 27, 2022 17:29

ref: move callbackify to lib/util.ts file

9e03b74

add: tls data to the response

47e3748

add: expand timings object

8fae952

patrykcieszkowski added 5 commits May 31, 2022 17:54

ref: http tests

fd81353

making linters happy

73eee30

making linters happy

7162f2d

making linters happy

9936e88

fix: schema: optional query obj

b74fded

jimaek merged commit 4b83346 into master Jun 7, 2022

jimaek deleted the http-command branch June 7, 2022 14:14

github-actions bot added the released label Jun 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: http command #47

feat: http command #47

patrykcieszkowski commented May 17, 2022 •

edited

Loading

jimaek commented May 24, 2022

patrykcieszkowski commented May 24, 2022 •

edited

Loading

jimaek commented May 24, 2022

patrykcieszkowski commented Jun 1, 2022 •

edited

Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022 •

edited

Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022

jimaek commented Jun 1, 2022 •

edited

Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 6, 2022

jimaek commented Jun 6, 2022

patrykcieszkowski commented Jun 6, 2022

jimaek commented Jun 6, 2022

patrykcieszkowski commented Jun 6, 2022

github-actions bot commented Jun 7, 2022

feat: http command #47

feat: http command #47

Conversation

patrykcieszkowski commented May 17, 2022 • edited Loading

jimaek commented May 24, 2022

patrykcieszkowski commented May 24, 2022 • edited Loading

jimaek commented May 24, 2022

patrykcieszkowski commented Jun 1, 2022 • edited Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022 • edited Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 1, 2022

jimaek commented Jun 1, 2022 • edited Loading

jimaek commented Jun 1, 2022

patrykcieszkowski commented Jun 6, 2022

jimaek commented Jun 6, 2022

patrykcieszkowski commented Jun 6, 2022

jimaek commented Jun 6, 2022

patrykcieszkowski commented Jun 6, 2022

github-actions bot commented Jun 7, 2022

patrykcieszkowski commented May 17, 2022 •

edited

Loading

patrykcieszkowski commented May 24, 2022 •

edited

Loading

patrykcieszkowski commented Jun 1, 2022 •

edited

Loading

patrykcieszkowski commented Jun 1, 2022 •

edited

Loading

jimaek commented Jun 1, 2022 •

edited

Loading