[Epic] New HTTP Adaptor & Utils

[josephjclark]

Epic PR for all the http and common work.

Closes #316 #320 #304

Changes

• New http helpers in common
• New http adaptor
• deprecated old common.http
• update template to showcase new pattern
• check and maybe improve docs
• remove axios, superagent and request across the repo
• Add linting for undeclared variables (and fix a whole bunch of stuff)

Overview

This PR has the broad aim of removing axios, superagent and request in favour of a single HTTP client - undici.

But we're not quite doing that yet as it's too painful.

As a starting point, we have added a new set of http helper functions (functions not operations!) into common, which you'll find in common/util/http. These are wrappers around undici. We intend that all new and existing adaptors migrate onto these utils.

The headline feature for this release is HTTP 6.0, which is the existing HTTP adaptor "rebased" onto the common helpers, with axios removed. This comes with loads of breaking changes, so there's a thorough migration guide in the http changelog. HTTP should now be a very lightweight adaptor which just wraps the common helpers.

The template has been changed to show usage of the new common helpers.

Finally, the docs output has been improved (internal links with upper case characters fixed; magic metadata removed; better documentation for http options added through a typedef) and linting has been strengthened (resulting in many small but critical patches on existing adaptors).

[josephjclark]

Added a cheeky fix for #304

[josephjclark]

@mtuchi caught an issue where parseXML was broken because expandReferences isn't imported.

This wasn't picked up in unit testing or linting, which is a bit scary.

I don't want to to write a parseXML unit test now, so I have added a linting rule for undeclared variables. This triggers a rash of patch fixes to various adaptors.

[josephjclark]

Running some tests with Mtuchi I think I really need to do slightly better error handling immediately.

The error message we get back right now does not include any message from the server - just the status code and status message.

If the server returns an error message, we need to include it in the error object

Edit: push a commit to return the response body and headers on the error object, and to log the error body when errors occur. This improves the feedback we're giving.

[stuartc]

If the server returns an error message, we need to include it in the error object

This is great! A lot of people will be very happy. I can't remember why (if any reason) we didn't do this before? I know there has like always been security considerations around this, like someone logging headers that have API keys in them.

Perhaps it's a non-issue now - but perhaps worth asking @taylordowns2000 why we previously cut error responses from HTTP requests to almost nothing.

[josephjclark]

@stuartc The error comments are more related to this branch than to main. I think the error reporting used to be mostly fine - the big problem was that the whole response object was logged, leaking all kinds of stuff and spamming the output.

The new adaptor builds an error object and returns hand-picked keys, which is a problem if we return the wrong keys.

I do want to enable access to the raw request, just in case, but there's a separate issue for that. I think we should do it soonish. Part of the problem is it's hard to test this stuff, and when you have a test implementation it's tempting to lean into it a bit too hard.