API mocking with MSW fails after upgrade to miniflare 2.2.0

[fbarthez]

Hey @mrbbot, thanks for all the work on miniflare, it's been a game-changer!
With miniflare 1, I've been using mock service worker to mock API responses. When attempting to upgrade to miniflare 2.2.0, my tests failed. I've no idea where to start looking for a solution, so I've put up a fairly minimal demo, hoping that'll make it easier to pinpoint the cause. HEAD is using miniflare 2.2.0, while the miniflare-1.4.1 branch shows what it looks like when it's working. Running npm test should show failing tests for HEAD and passing for the latter.

[mrbbot]

Hey! 👋 I think this is because instead of node-fetch, Miniflare 2 uses undici for its fetch implementation, which doesn't use the built-in http module, so bypasses Mock Service Worker's monkeypatching.

I'm not sure if this is you, but a similar question was recently asked on the Remix Discord server. The response there might be useful at least until Mock Service Worker is updated to work with undici: https://discord.com/channels/770287896669978684/777863813981536280/935974246339985438

[fbarthez]

Thanks! That'll get me back on track, probably using the built-in mocking feature of undici for now.

[SupremeTechnopriest]

Hey @fbarthez,

Were you ever able to get this to work? Im testing out the wrangler beta and Im not able to get the undici mocking to intercept fetch requests.

[fbarthez]

Hey @SupremeTechnopriest, no it's still on my todo list and will probably stay there for at least another week. ;/

[SupremeTechnopriest]

I kind of need to be able to mock the endpoints. I also really like the MSW API. I might bite the bullet and write the interceptor, but it might also be nice for jest-environment-miniflare to expose its own mocking framework. Maybe a wrapper around the undici ones. @mrbbot what do you think about starting a discussion around that?

[SupremeTechnopriest]

Seems like there are multiple modules that have undici as a dependency... miniflare, @minflare/core, @miniflare/http-server etc... Not sure which one is the one actually performing the request. I'm pretty sure you need to use the exported mocking functions from the instance of undici doing the request. The below test fails no matter where I import undici from.

import { MockAgent, setGlobalDispatcher } from '../../node_modules/@miniflare/core/node_modules/undici'

const agent = new MockAgent()
agent.disableNetConnect()

agent.get('http://foo.com')
  .intercept({
    path: '/',
    method: 'GET'
  })
  .reply(200, 'bar')

setGlobalDispatcher(agent)

describe('undiciMock', () => {
  test('Mocks response', async () => {
    const request = new Request('http://foo.com/')
    const res = await fetch(request)
    const text = await res?.text()
    expect(text).toBe('bar')
  })
})

If you have any insight as to what instance of undici we should be mocking that would be very helpful @mrbbot. Still think we should expose the mocking features of undici more cleanly.

Using jest-environment-miniflare with the default config.

[mrbbot]

Hey @SupremeTechnopriest! 👋 Unfortunately, you won't be able to use setGlobalDispatcher inside jest-environment-miniflare. Miniflare's undici version is imported outside the Jest environment in Node. Jest and Node don't share a module cache, so trying to import undici inside Jest will always give you a different version.

Currently, I think the best option for jest-environment-miniflare is to use Jest's function mocks to mock the global fetch function. Something like... (untested 😅)

const fetchSpy = jest.spyOn(globalThis, "fetch");
fetchSpy.mockImplementation((input, init) => {
  const req = new Request(input, init);
  const url = new URL(req.url);
  if (req.method === "GET" && url.pathname === "/") {
    return new Response("bar");
  }
  return new Response(null, { status: 404 });
});

...

jest.restoreAllMocks();

I'd definitiely like to start a discussion on this though. Maybe a global API like getMiniflareMockAgent() that returned a correctly-setup MockAgent?

[SupremeTechnopriest]

@mrbbot,

That's exactly what I was thinking! Miniflare has made my life soooooooo much better and this would be the icing on the cake. Let me know how I can help!

[SupremeTechnopriest]

Your fetch spy works by the way @mrbbot.

[SupremeTechnopriest]

Wrote a little warpper while we sort out the mock global.

export function MockFetch (base: string) {
  const mocks: IFetchMock[] = []
  const fetchSpy = jest.spyOn(globalThis, 'fetch')
  fetchSpy.mockImplementation(async (input: string | Request, init?: Request | RequestInit | undefined): Promise<Response> => {
    const request = new Request(input, init)

    for (const mock of mocks) {
      if (`${base}${mock.path}` === request.url && request.method === mock.method) {
        const resInit = {
          status: mock.status,
          statusText: mock.statusText,
          headers: {}
        }
        if (mock.headers) resInit.headers = mock.headers
        return new Response(mock.body, resInit)
      }
    }
    return new Response(null, { status: 404, statusText: 'Not Found' })
  })

  const intercept = (mock: IFetchMock) => {
    mocks.push(mock)
  }

  const restore = () => {
    mocks.length = 0
    jest.restoreAllMocks()
  }

  return { intercept, restore }
}

const { intercept, restore } = MockFetch('http://foo.com')

intercept({
  path: '/',
  method: 'GET',
  status: 200,
  statusText: 'OK',
  body: 'bar',
  headers: {
    'X-Foo': 'bar'
  }
})

afterAll(restore)

test('Mock', async () => {
  const request = new Request('http://foo.com/')
  const res = await fetch(request)
  const text = await res?.text()
  expect(text).toBe('bar')
  expect(res.headers.get('x-foo')).toBe('bar')
})

[ptim]

This was super helpful to me, as jest-fetch-mock doesn't return actual responses!

Here's the wrapper I'm using now:

// @test/helpers/MockAgent.ts

type RouteMatcher = (
  input: Request | string,
  init?: RequestInit | Request,
) => boolean

type Route = {
  matcher: RouteMatcher
  response: Response
}

/**
 * Returns a handler function composed of passed Handlers,
 * falling back to a 404 Response on no match.
 *
 * @see https://github.com/cloudflare/miniflare/issues/162#issuecomment-1059549646
 */
export function createRequestHandlers(routes: Route[] = []) {
  return async function handler(
    input: Request | string,
    init?: RequestInit | Request,
  ) {
    for (const { matcher, response } of routes) {
      if (matcher(input, init)) return response
    }

    return new Response('Not Found', { status: 404 })
  }
}

type CreateMatcherOptions = {
  method: string
}
/** Returns a RouteMatcher which evaluates matches based on `pathname` & `method` */
export function createMatcher(
  pathname: string,
  { method }: CreateMatcherOptions = { method: 'GET' },
): RouteMatcher {
  return function (input: Request | string, init?: RequestInit | Request) {
    const req = new Request(input, init)
    const url = new URL(req.url)
    return method === req.method && pathname === url.pathname
  }
}

export const get = (pathname: string) => createMatcher(pathname)
export const post = (pathname: string) =>
  createMatcher(pathname, { method: 'POST' })

/**
 * Mocks global fetch and returns a fetchSpy.
 *
 * ```ts
  // example GET matching on pathname /
  const fetchSpy = mockFetch('/', new Response('mocked'))

  // example POST /bar (using lower level matcher)
  const fetchSpy = mockFetch((input, init) => {
    const req = new Request(input, init)
    const url = new URL(req.url)
    return req.method === 'POST' && url.pathname === '/bar'
  }, new Response('mocked POST'))
 * ```
 */
export function mockFetch(
  pathnameOrMatcher: string | RouteMatcher,
  response: Response,
) {
  const fetchSpy = jest.spyOn(globalThis, 'fetch')
  const requestHandlers = createRequestHandlers([
    {
      matcher:
        typeof pathnameOrMatcher === 'string'
          ? get(pathnameOrMatcher)
          : pathnameOrMatcher,
      response,
    },
  ])
  fetchSpy.mockImplementation(requestHandlers)
  return fetchSpy
}

import { jest } from '@jest/globals'
import {
  basicRequestHandler,
  createRequestHandlers,
  get,
  mockFetch,
} from '@test/helpers/MockAgent'

afterEach(() => {
  jest.restoreAllMocks()
})

test('should provide mockFetch', async () => {
  const _fetchSpy = mockFetch('/', new Response('mocked'))

  const req = new Request('http://localhost/')
  const res = await fetch(req)
  expect(res.status).toBe(200)
  expect(await res.text()).toBe('mocked')
})

test('should mock different routes', async () => {
  let mockBarOnce = true

  const requestHandlers = createRequestHandlers([
    {
      matcher: get('/foo'),
      response: new Response('foo'),
    },
    {
      matcher: (input, init) => {
        const req = new Request(input, init)
        const url = new URL(req.url)
        const matches = req.method === 'GET' && url.pathname === '/bar'
        if (matches && mockBarOnce) {
          mockBarOnce = false
          return true
        }
        return false
      },
      response: new Response('bar'),
    },
  ])

  const fetchSpy = jest.spyOn(globalThis, 'fetch')
  fetchSpy.mockImplementation(requestHandlers)

  let req = new Request('http://localhost/')
  let res = await fetch(req)
  expect(res.status).toBe(404)

  req = new Request('http://localhost/foo')
  res = await fetch(req)
  expect(res.status).toBe(200)
  expect(await res.text()).toBe('foo')

  // mock GET /bar only once
  req = new Request('http://localhost/bar')
  res = await fetch(req)
  expect(res.status).toBe(200)
  expect(await res.text()).toBe('bar')

  // second call to GET /bar passes through
  req = new Request('http://localhost/bar')
  res = await fetch(req)
  expect(res.status).toBe(404)

  expect(fetchSpy.mock.calls).toHaveLength(4)
})

(updated)

[SupremeTechnopriest]

This looks nice! Good stop gap until the true undici mocks are exposed.

[aiji42]

I was hoping that perhaps this issue had been resolved with the release of 2.4.0 yesterday, but it has been postponed to 2.5.0.
I am very much looking forward to the next release 👀

[msaiducar]

Is there any progress on this issue?

[LukePammant]

@ptim and @SupremeTechnopriest if I'm not mistaken your mock fetch is only being used in the jest test code itself because you're calling fetch directly in the test. So any invokation to miniflare via something like miniflare.dispatchFetch will not use your mock. Is that right or am I misunderstanding?

I'd love to do a simple spyOn too but miniflare still uses the undici fetch since jest and miniflare don't share a module cache as @mrbbot pointed out.

[SupremeTechnopriest]

It should mock the fetch call for your module code as well. Are you seeing something different?

[LukePammant]

@SupremeTechnopriest yeah, if I add a mock implementation that simply throws an exception and then call my worker via miniflare.dispatchFetch the request goes through and calls the endpoint. Example that should just fail but works fine:

const fetchSpy = jest.spyOn(globalThis, 'fetch')
fetchSpy.mockImplementation(async (input: string | Request, init?: Request | RequestInit | undefined): Promise<Response> => {
    throw "No fetch for you"
})

it('shouldn throw', async () => {
     const miniflare = new Miniflare({
       // ... setup miniflare vars
     });
    miniflare.dispatchFetch("https://google.com");
});    

Maybe it's because I'm not using the new Modules syntax for CF Workers?

[msaiducar]

@SupremeTechnopriest and @LukePammant, I can confirm that mocking the globalThis.fetch doesn't work with miniflare.dispatchFetch. We have some fetch() calls to some external APIs inside our worker, we wanted to test those calls, but spying fetch was not possible. We used this library to workaround. We start a local HTTP server and use this server instead of external APIs.

[ptim]

Works for me if I call the handler directly, but not with mf.dispatchFetch:

Works:

// src/index.ts
export async function handleRequest(request: Request, env: Bindings) {
  const res = await fetch("http://not-a-real-domain.foo");
  return res;
}

import { handleRequest } from "@/index";
// ...

test("should provide mockFetch", async () => {
  const _fetchSpy = mockFetch(
    "http://not-a-real-domain.foo",
    new Response("mocked")
  );

  const env = getMiniflareBindings();
  const req = new Request("http://localhost/");
  const res = await handleRequest(req, env);

  expect(_fetchSpy).toHaveBeenCalledTimes(1);
  expect(res.status).toBe(200);
  expect(await res.text()).toBe("mocked");
});