Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crawling context sendRequest cookies can't be easily overriden #2744

Open
1 task done
MatousMarik opened this issue Nov 13, 2024 · 0 comments
Open
1 task done

Crawling context sendRequest cookies can't be easily overriden #2744

MatousMarik opened this issue Nov 13, 2024 · 0 comments
Assignees
Labels
bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@MatousMarik
Copy link

MatousMarik commented Nov 13, 2024

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/core

Issue description

As shown in the API documentation but not explicitly stated anywhere, sendRequest takes an overrideOptions object that overrides options so that whatever you put into overrideOptions.headers.cookie or overrideOptions.cookieJar doesn't really matter if you won't also override overrideOptions.cookieJar.setCookie (or whatever internal function) to modify session cookies.

I think this should be explained somewhere here: https://crawlee.dev/docs/guides/got-scraping#cookiejar

My code: git@github.com:MatousMarik/send-request-bug.git
Platform run: https://console.apify.com/actors/tIFxT28Ja3LFuVjkA/source

code and output
sendRequest API
sendRequest cookieJar

Code sample

// changes in cheerio ts template
import { CookieJar } from 'tough-cookie';
const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxRequestsPerCrawl,
    requestHandler: async ({ request: { url }, sendRequest, session }) => {
        session?.setCookie('ahoj=svete', url);
        const cookieJar = new CookieJar();
        await cookieJar.setCookie(session!.getCookieString(url), url);
        await cookieJar.setCookie('hello=world', url);
        const response = await sendRequest({
            headers: {
                cookie: 'hi=world',
                Cookie: 'hello=world',
            },
            cookieJar,
        });
        console.log(response.request.options.headers);
        console.log('JAR:', await cookieJar.getCookieString(url));
    },
});

Package version

3.12.0

Node.js version

node@22.7.0, typescript@5.6.3

Operating system

Sequoia 15.1 (24B83)

Apify platform

  • Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

git@github.com:MatousMarik/send-request-bug.git
https://console.apify.com/actors/tIFxT28Ja3LFuVjkA/source

@MatousMarik MatousMarik added the bug Something isn't working. label Nov 13, 2024
@github-actions github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 13, 2024
@B4nan B4nan self-assigned this Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

2 participants