Skip to content

Commit

Permalink
feat: userAgent config
Browse files Browse the repository at this point in the history 
Fixes #243
  • Loading branch information
@harlan-zw
harlan-zw committed Oct 16, 2024
1 parent 1bb0095 commit c6c5f7d
Show file tree
Hide file tree
Showing 8 changed files with 60 additions and 33 deletions.
51 changes: 26 additions & 25 deletions docs/content/2.integrations/0.cli.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,32 +36,33 @@ See the [Configuration](#configuration) section for more details and the guides.

### CLI Options

| Options | |
|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `-v, --version` | Display version number. |
| `--site <url>` | Host URL to scan. |
| `--root <path>` | Define the project root. |
| `--config-file <path>` | Path to config file. |
| `--output-path <path>` | Path to save the contents of the client and reports to. |
| `--cache` | Enable the caching. |
| `--no-cache` | Disable the caching. |
| `--desktop` | Simulate device as desktop. |
| `--mobile` | Simulate device as mobile. |
| `--throttle` | Enable the throttling. |
| `--samples` | Specify the amount of samples to run. |
| `--sitemaps` | Comma separated list of sitemaps to use for scanning. |
| `--urls` | Specify explicit relative paths as a comma-separated list.<br>e.g. `unlighthouse --site unlighthouse.dev --urls /guide,/api,/config` |
| Options | |
|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `-v, --version` | Display version number. |
| `--site <url>` | Host URL to scan. |
| `--root <path>` | Define the project root. |
| `--config-file <path>` | Path to config file. |
| `--output-path <path>` | Path to save the contents of the client and reports to. |
| `--cache` | Enable the caching. |
| `--no-cache` | Disable the caching. |
| `--desktop` | Simulate device as desktop. |
| `--mobile` | Simulate device as mobile. |
| `--throttle` | Enable the throttling. |
| `--samples` | Specify the amount of samples to run. |
| `--sitemaps` | Comma separated list of sitemaps to use for scanning. |
| `--urls` | Specify explicit relative paths as a comma-separated list.<br>e.g. `unlighthouse --site unlighthouse.dev --urls /guide,/api,/config` |
| `--exclude-urls` | Specify relative paths (string or regex) to exclude from scanning as a comma-separated list. <br>e.g. `unlighthouse --site unlighthouse.dev --exclude-urls /guide/.*,/api/.*` |
| `--include-urls` | Specify relative paths (string or regex) to include as a comma-separated list. <br>e.g. `unlighthouse --site unlighthouse.dev --include-urls /guide/.*` |
| `--enable-javascript` | When inspecting the HTML wait for the javascript to execute. Useful for SPAs. |
| `--disable-javascript` | When inspecting the HTML, don't wait for the javascript to execute. |
| `--enable-i18n-pages` | Enable scanning pages which use x-default. |
| `--disable-i18n-pages` | Disable scanning pages which use x-default. |
| `--disable-dynamic-sampling` | Disable dynamic sampling of paths. |
| `--extra-headers` | Extra headers to send with the requests. |
| `--default-query-params` | Default query params to send with the requests. |
| `-d, --debug` | Debug. Enable debugging in the logger. |
| `-h, --help` | Display available CLI options |
| `--include-urls` | Specify relative paths (string or regex) to include as a comma-separated list. <br>e.g. `unlighthouse --site unlighthouse.dev --include-urls /guide/.*` |
| `--enable-javascript` | When inspecting the HTML wait for the javascript to execute. Useful for SPAs. |
| `--disable-javascript` | When inspecting the HTML, don't wait for the javascript to execute. |
| `--enable-i18n-pages` | Enable scanning pages which use x-default. |
| `--disable-i18n-pages` | Disable scanning pages which use x-default. |
| `--disable-dynamic-sampling` | Disable dynamic sampling of paths. |
| `--extra-headers` | Extra headers to send with the requests. |
| `--user-agent` | Provide a custom user agent for all network requests. |
| `--default-query-params` | Default query params to send with the requests. |
| `-d, --debug` | Debug. Enable debugging in the logger. |
| `-h, --help` | Display available CLI options |

### Config File

Expand Down
7 changes: 7 additions & 0 deletions docs/content/3.api/config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,13 @@ Provide cookies to be set for Axios and Puppeteer requests.

Provide extra headers to be set for Axios and Puppeteer requests.

### userAgent

- **Type:** `string`
- **Default:** `undefined`

Provide a custom user agent for all network requests.

### defaultQueryParams

- **Type:** ` false | QueryObject`
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/createCli.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ export default function createCli() {
cli.option('--mobile', 'Simulate device as mobile.')

cli.option('--site <site>', 'Host URL to scan.')
cli.option('--user-agent <user-agent>', 'Specify a top-level user agent all requests will use.')
cli.option('--router-prefix <site>', 'The URL path prefix for the client and API to run from.')
cli.option('--sitemaps <sitemaps>', 'Comma separated list of sitemaps to use for scanning. Providing these will override any in robots.txt.')
cli.option('--samples <samples>', 'Specify the amount of samples to run.')
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/types.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ export interface CliOptions {
disableSitemap?: boolean
disableDynamicSampling?: boolean
sitemaps?: string
userAgent?: string
}

export interface CiOptions extends CliOptions {
Expand Down
10 changes: 10 additions & 0 deletions packages/cli/src/util.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,16 @@ export function pickOptions(options: CiOptions | CliOptions): UserConfig {
})
}

if (options.userAgent) {
picked.extraHeaders = picked.extraHeaders || {}
picked.extraHeaders['User-Agent'] = options.userAgent
// set lighthouse
picked.lighthouseOptions = picked.lighthouseOptions || {}
picked.lighthouseOptions.emulatedUserAgent = options.userAgent
// pupeteer will respect userAgent
picked.userAgent = options.userAgent
}

if (options.defaultQueryParams) {
picked.defaultQueryParams = picked.defaultQueryParams || {}
options.defaultQueryParams.split(',').forEach((param) => {
Expand Down
3 changes: 3 additions & 0 deletions packages/core/src/puppeteer/util.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ export async function setupPage(page: Page) {
await page.setExtraHTTPHeaders(resolvedConfig.extraHeaders)
.catch(softErrorHandler('Failed to set extra headers'))
}
if (resolvedConfig.userAgent) {
await page.setUserAgent(resolvedConfig.userAgent)
}
await hooks.callHook('puppeteer:before-goto', page)
}
})
Expand Down
6 changes: 6 additions & 0 deletions packages/core/src/types.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,12 @@ export interface ResolvedUserConfig {
* @default false
*/
extraHeaders: false | Record<string, string>
/**
* The user agent to use for all requests. Uses default puppeteer / lighthouse user agent if not provided.
*
* @default false
*/
userAgent?: string
/**
* Query params to add to every request.
*
Expand Down
14 changes: 6 additions & 8 deletions packages/core/src/util.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,21 +146,19 @@ export async function createAxiosInstance(resolvedConfig: ResolvedUserConfig) {

axiosOptions.headers = axiosOptions.headers || {}
// this should always be set
axiosOptions.headers['User-Agent'] = resolvedConfig.lighthouseOptions.emulatedUserAgent || 'Unlighthouse'

if (resolvedConfig.cookies) {
axiosOptions.headers.Cookie = resolvedConfig.cookies
.map(cookie => `${cookie.name}=${cookie.value}`)
.join('; ')
}

if (resolvedConfig.extraHeaders) {
axiosOptions.headers = {
// fallback user agent, allow overriding
'User-Agent': 'Unlighthouse',
...resolvedConfig.extraHeaders,
...axiosOptions.headers,
}
const userAgent = resolvedConfig.userAgent || resolvedConfig.lighthouseOptions.emulatedUserAgent || 'Unlighthouse'
axiosOptions.headers = {
// fallback user agent, allow overriding
'User-Agent': userAgent,
...(resolvedConfig.extraHeaders || {}),
...axiosOptions.headers,
}

if (resolvedConfig.defaultQueryParams)
Expand Down

0 comments on commit c6c5f7d

Please sign in to comment.