Is there a way to intercept HTTP requests made by Craft and modify the headers?

[jamesmacwhite]

I have a Feed Me feed which provides image URLs on a CDN path, the issue is the CDN is using Cloudflare and our server environment which happens to be Craft Cloud is being caught Cloudflare's bot/fingerprinting detection and the image URLs when being attempted to be downloaded are getting a 403.

I've tested locally and found by specifying a User-Agent the request is successful. Unfortunately, for asset downloading in Feed Me there is no way to add/modify HTTP headers with Curl or fopen. I raised an issue on the Feed Me GitHub: craftcms/feed-me#1538

I was however wondering if there any way to use an event/app.web.php to intercept requests and add/modify headers depending on a condition i.e. hostname as a workaround. Feed Me I don't think provides any events specifically for asset fetching, so it's not really able to manipulate it, so I was wondering if I could do it somehow at the higher level.

[brandonkelly]

You can modify the config that Craft uses to create Guzzle (HTTP) clients via config/guzzle.php:

<?php

return [
    'headers' => [
        'User-Agent' => '...',
    ],
];

Looks like it’s not documented (on it), but Feed Me supports setting a clientOptions array on your feed configs from config/feed-me.php, which will get merged into the main Guzzle config:

<?php

return [
    '*' => [
        'clientOptions' => [
            'headers' => [
                'User-Agent' => '...',
            ],
        ],
    ],
];

[jamesmacwhite]

Hi Brandon. I'm not sure these settings will apply in this case, because the AssetHelper class uses either fopen or curl, not Guzzle and wouldn't apply these settings. For the main Feed model with the Feed URL it does apply, but the asset fetching calls a different client I believe.