Support mocking SMTP

[kentcdodds]

I'm circling back on mswjs/msw#751

I would like to make my sendEmail function easier to migrate between different email providers and using the SMTP standard is the easiest way to do this. But then I have to resort to doing weird things to mock it (nodemailer specifically) out during development and testing.

Any chance MSW could support mocking these kinds of requests?

[kettanaito]

Hey, @kentcdodds.

I took a look at how Nodemailer works, and as I suspected, they tap into a raw socket connection via the net module of Node.js. The Interceptors currently operate on a higher level http.ClientRequest class to provision HTTP interception since net is generally too low-level to gain much control over the request instances. At that level, Node is operating with socket messages that don't have any context of the sender (was it an HTTP request? Was it an SMTP message?) or don't have enough context to lower our implementation logic to the sockets.

Here's a rough representation of those network layers:

http
  http.ClientRequest
    IncomingMessage / OutgoingMessage
      Socket
        write / on('data')

Also a useful ref for the future me.

The Socket layer seems to be the highest layer we can capture requests over SMTP and other non-HTTP protocols in Node.js. That's also how Nodemailer and other implementations do that since Node doesn't ship with a standard API to make SMTP requests easier as it does for HTTP requests.

My approach to interception is that it should be unified, as in I'm not eager to ship logic that's specific to certain libraries or even protocols. A good solution here would be to lower all the interception logic to the Socket layer, including the interception of HTTP requests. But, as I've mentioned, that can prove problematic in some use cases.

An incremental solution would be to try implementing something like a SocketInterceptor that is limited by design and may not support full HTTP interception like the ClientRequestInterceptor does. In turn, it could be utilized to implement SMTP interception since it is implemented on that layer and the consumer won't lose any context when tapping into it (it is dealing with raw sent/received messages).

I'm currently busy with the Fetch API primitives adoption in MSW and I'm excited it's reaching the finish line. I won't have any time to look into this feature in the nearest future, I'm afraid. This can be a great improvement for the upcoming releases though!

[kentcdodds]

Thanks for explaining all of that! If this becomes a high enough priority for me then I'll signal in here that I've started looking into it myself. Cheers!

[kettanaito]

I also recall that one technical challenge was to parse socket messages. At the Socket level, you are operating with plain text sent over the network so you receive chunks like

GET /resource HTTP1.1
Host: api.com
X-Header: Value

These strings have to be parsed to do request matching and turn the finished request message into a Fetch Request instance. Sadly, Node doesn't expose their parser publicly (it's written in C).

[kettanaito]

@kentcdodds, I've given the Socket-based interceptor a try in #375 and it mostly works. I've managed to work around the actual socket connection and let the socket think it's always connected, even if to a non-existing host. Responding with a mock message also works based on the test I wrote.

If you ever get to play around with this, take a look at that pull request, it's a good starting point.

[kentcdodds]

Wow! Very cool!

[kettanaito]

The main challenge at the moment is the import order. Since you're often using a dependency that imports net internally, the interceptor becomes sensitive to order:

// OK!
import './interceptor'
import { sendMail } from 'third-party' // imports already patched "net"

// Nope :( 
import { sendMail } from 'third-party' // imports and "remembers" unpatched "net"
import './interceptor'

Because of this, it cannot be applied lazily (i.e. in your beforeAll hook as it should be).

This must have a solution since it's not a problem for the http module and we already have an interceptor that works just fine with third-parties that depend on http.

This may also be only specific to native Node modules (e.g. http depending on net). I haven't tried this with third-party libraries like nodemailer).

[kettanaito]

When talking about consumable third-party packages like Nodemailer, the interception becomes more complex as a result of the implementation details of those packages. For example, Nodemailer can utilize three different protocols to implement email transfer and switch between them as it sees fit. A single interceptor cannot guarantee it will intercept an email sent with Nodemailer simply because of that.

This opens up a more interesting discussion about low-level interception and high-level usage needs. When you're using tools like Nodemailer, all you want to do is intercept that sent email and maybe mock the server's response. You don't want to think (or even know) how exactly that email transfer is being implemented under the hood—that's too low-level for you. But from the Interceptor's standpoint, it doesn't tackle specific tools but instead tries to provide an interception on the basis of common ground (e.g. let's patch http.ClientRequest so that any third-party using it would be intercepted). This principle makes sense but it doesn't scale to more complex third-party implementations.

This reminds me a lot about WebSockets and what makes their support challenging. No libraries are using the WebSocket standard API from the browser (at least by default). When talking about Node, there isn't even a standard API, to begin with, so every solution implements message transports as it sees fit. It's very hard to make the expectations from a tool like Interceptors and MSW predictable when the underlying (internal) implementation is unpredictable. This is the main reason why I revamped the architecture of the Interceptors to allow batched interceptors—compound interception mechanisms that can combine (and reuse) multiple low-level interceptors. For example, to intercept WebSocket transfer we have to

• Intercept WebSocket server handshake (an HTTP request).
• Then, intercept socket messages (actual data sent/received).

Which already gives us 2 different areas to work with. I think SMTP is very similar in this regard.

I suppose we can implement something like a SmtpInterceptor batched interceptor that would account for different protocols that third-parties may implement SMTP through. I just don't know how well this aligns with the low-level philosophy of this library and how that would integrate into MSW, where interceptors act as the source for incoming requests so you would always get SMTP interception logic even if you're not planning on intercepting anything related to that.

[kentcdodds]

That makes sense. Might be more work than it's worth 😬

[kettanaito]

With #515, mocking SMTP and other protocols in Node.js becomes a matter of introducing a respective parser. So, it's possible!