[BUG] Cookies get assigned before fulfilling a response

[szmarczak]

Context:

• Playwright Version: 1.19.2
• Operating System: Linux
• Node.js version: v16.13.0
• Browser: Chromium, Firefox

Code Snippet

const playwright = require('playwright');

(async () => {
        const browser = await playwright.chrome.launchPersistentContext('', {
                headless: false,
        });

        await browser.route('**/*', async (route, request) => {
                try {
                        const response = await request.frame().page().request.fetch(route.request());
                        const responseHeaders = response.headers();

                        delete responseHeaders['set-cookie'];

                        console.log('route not yet fulfilled', (await browser.cookies()).length);

                        route.fulfill({
                                response,
                                headers: responseHeaders,
                        });
                } catch {
                        route.abort('failed');
                }
        });

        const page = await browser.newPage();
        await page.goto('https://www.google.com/');
})();

Describe the bug

Cookies get assigned despite the delete. Replacing set-cookie does not replace the original cookies.

Expected

route not yet fulfilled 0

Actual

route not yet fulfilled 1

[pavelfeldman]

Thank you for your report. I noticed that

1. we didn't do a great job at documenting the cookie jar difference, it is only mentioned here: https://playwright.dev/docs/api/class-apirequest.
2. our documentation for page.request is missing

This should work for you:

const { chromium, request } = require('playwright');

(async () => {
  // Don't use persistent context, it has cookies from previous sessions.
  const browser = await chromium.launch({ headless: false });
  // Create brand new context all the time.
  const context = await browser.newContext();
  await context.route('**/*', async route => {
    // If you use page.fetch, it'll use browser cookie jar, use new request context instead.
    // Create request context with standalone cookie jar.
    const requestContext = await request.newContext();
    const response = await requestContext.fetch(route.request());
    const responseHeaders = response.headers();
    delete responseHeaders['set-cookie'];
    console.log('route not yet fulfilled', (await context.cookies()).length);

    route.fulfill({
      response,
      headers: responseHeaders,
    });
  });
  const page = await context.newPage();
  await page.goto('https://www.apple.com/');
})();

[szmarczak]

Thanks for the help, really appreciate it! I'd like to use persistent context anyway due to caching. I saw this piece:

playwright/packages/playwright-core/src/client/playwright.ts

Line 47 in fb00991

readonly request: APIRequest;

so is await playwright.request.newContext() correct?

[pavelfeldman]

I'd like to use persistent context anyway due to caching.

Sure, whatever works for you. We would recommend to save/restore storageState, but I'm not sure what your scenario is.

so is await playwright.request.newContext() correct?

Yes, I'm doing it in my snippet above like this: const requestContext = await request.newContext();

[szmarczak]

I just figured out that context.fetch uses Node.js HTTP API under the hood. This is quite problematic, because it sometimes breaks in very edge cases (the HTTP parser is different, TLS ciphers are different, different API for HTTP2, etc.). You can compare the settings here. So we've come up with https://github.com/apify/got-scraping that tries to mimic the Chrome behavior closely.

Also I noticed that route.continue uses Fetch.continueRequest, but the issue here is that it still overrides the cookie header unless there are no cookies.