feat: crawl URLs in `<meta>` tags #9900

LorisSigrist · 2023-05-10T17:09:18Z

As described in #5228

Up until now, urls in <meta> tags have not been crawled. This has made programmatic social-images tricky.

This PR crawls urls in <meta> tags, depending on if the have a name or property attribute that is whitelisted.
Not all meta-tags can contain urls, so a whitelist is needed to only crawl the ones that can. I added all the usual social-media tags to the whitelist.

This did require some minor reorganisation of prepublish/crawl.js, since this is the first time that more than one attribute needed to be evaluated to crawl single tag.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

It's really useful if your PR references an issue where it is discussed ahead of time. In many cases, features are absent for a reason. For large changes, please create an RFC: https://github.com/sveltejs/rfcs
This message body should clearly illustrate what problems it solves.
Ideally, include a test that fails without this PR but passes with it.

Tests

Run the tests with pnpm test and lint the project with pnpm lint and pnpm check

Changesets

If your PR makes a change that should be noted in one or more packages' changelogs, generate a changeset by running pnpm changeset and following the prompts. Changesets that add features should be minor and those that fix bugs should be patch. Please prefix changeset messages with feat:, fix:, or chore:.

…social tags

changeset-bot · 2023-05-10T17:09:22Z

🦋 Changeset detected

Latest commit: 774bc04

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@sveltejs/kit	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

.changeset/thirty-garlics-tan.md

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>

Rich-Harris

thank you! had a question about .trim().toLowerCase(), inline

Rich-Harris · 2023-05-16T21:57:35Z

packages/kit/src/core/postbuild/crawl.js

+					if (name && CRAWLABLE_META_NAME_ATTRS.has(name)) {
+						hrefs.push(resolve(base, content.trim().toLowerCase()));
+					}
+
+					if (property && CRAWLABLE_META_NAME_ATTRS.has(property.trim().toLowerCase())) {
+						hrefs.push(resolve(base, content));
+					}


why are we trimming/lowercasing the content attribute when name is present, but the property attribute when property is present?

in fact do we need to do any trimming or lowercasing at all? this doesn't happen with any other attribute, i'm curious why these would be different?

Whoops I left that in there accidentally 😅

At one point I was worried about the case where someone would leave some whitespace / wrongly capitalise some letters in an attribute, so during the refactor I trimmed and lowercased all attributes before the checks. After some experimenting, I then decided that it wasn't really required, and would make this PR dual-purpouse (addressing something that wasn't in the issue) so I removed it.

Must have missed that one, since it doesn't break anything. Sorry about that

* feat: Add a speedier script tag for prerendered redirects (sveltejs#9911) * feat: Add a script redirect to prerendered pages * changeset * Update .changeset/hungry-rocks-hunt.md Co-authored-by: Simon H <5968653+dummdidumm@users.noreply.github.com> * fix test * feat: Update string escaping * Update .changeset/hungry-rocks-hunt.md Co-authored-by: Conduitry <git@chor.date> --------- Co-authored-by: Simon H <5968653+dummdidumm@users.noreply.github.com> Co-authored-by: Conduitry <git@chor.date> * fix: avoid inlining raw/url CSS imports (sveltejs#9925) * fix: use transformRequest for CSS modules * just avoid raw or url * test and changeset * use parsed query * remove only * Update packages/kit/test/apps/basics/test/cross-platform/test.js * drive by test speed up * feat: prerender & analyse in worker rather than subprocess to support Deno (sveltejs#9919) * feat(fork): use workers * Create hot-actors-hope.md * Update packages/kit/src/utils/fork.js Co-authored-by: Rich Harris <hello@rich-harris.dev> --------- Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <hello@rich-harris.dev> * avoid using isMainThread, since it interacts poorly with vitest (sveltejs#9941) Co-authored-by: Rich Harris <git@rich-harris.dev> * chore: bump vite and devalue (sveltejs#9933) * bump vite and devalue * update templates * merge master * fix test --------- Co-authored-by: Rich Harris <git@rich-harris.dev> * chore: uvu -> vitest for create-svelte tests (sveltejs#9910) * chore: uvu -> vitest for create-svelte tests * format * concurrency (sveltejs#9921) * realised we werent typechecking this file, found some errors. fixed * Wait for beforeAll hook to complete * simplify * exclude create-svelte/template files from prettier, so that we can emit correctly formatted templates * remove unused file * Revert "exclude create-svelte/template files from prettier, so that we can emit correctly formatted templates" This reverts commit aa188d4. --------- Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <git@rich-harris.dev> * feat: unshadow `form` and `data` in `enhance` (sveltejs#9902) * feat: Un-shadow `data` and `form` in `enhance`, warn about future deprecation in dev * changeset * snek * Update .changeset/odd-crews-own.md Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> * Update packages/kit/test/apps/dev-only/package.json Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> * am not smart * still not smart * oops * oof * add deprecation notice --------- Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <git@rich-harris.dev> * fix: Set loader: { '.wasm': 'copy' } in esbuild config in `adapter-cloudflare-workers` (sveltejs#9940) * fix: Set loader: { '.wasm': 'copy' } in esbuild config in `adapter-cloudflare-workers` Copies WASM files in Cloudflare instead of trying to load them. Related to sveltejs#9909 * Create brave-peaches-buy.md * Update packages/adapter-cloudflare-workers/index.js * format --------- Co-authored-by: Rich Harris <git@rich-harris.dev> Co-authored-by: Rich Harris <hello@rich-harris.dev> * fix: Flaky test (sveltejs#9947) * fix: Flaky test * reuse locator * add semi * drive by test speed up * another classic * oh my god brain, y u so bad --------- Co-authored-by: gtmnayan <50981692+gtm-nayan@users.noreply.github.com> Co-authored-by: gtmnayan <gtmnayan@gmail.com> * remove envVarsInUse (sveltejs#9942) Co-authored-by: Rich Harris <git@rich-harris.dev> * fix: type `vitePlugin` in config (sveltejs#9946) * fix: type `vitePlugin` in config * changeset * fix: Set `loader: { '.wasm': 'copy' }` in esbuild config in `adapter-vercel` (sveltejs#9944) * fix: Enable wasm copy in adapter-vercel * Create olive-rings-eat.md --------- Co-authored-by: Rich Harris <richard.a.harris@gmail.com> * feat: crawl URLs in `<meta>` tags (sveltejs#9900) * Crawl social-image urls during prerender * Formatting & Linting * Format changeset & added exhaustive list of crawlable urls * Changed severity to minor as described in sveltejs#5228 * Added support for `property` attribute & limited valid names to just social tags * More tests * Better changeset message - I'm indecisive * Update .changeset/thirty-garlics-tan.md Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> * simplify * simplify * Removed redundant data-sanitation * DRY out --------- Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <git@rich-harris.dev> * Version Packages (sveltejs#9893) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat: support AWS via SST in adapter-auto (sveltejs#9874) * [feat] support AWS via SST in adapter-auto * Sync * Delete 95-adapter-aws-sst.md * Update .changeset/rotten-ducks-tan.md --------- Co-authored-by: Rich Harris <hello@rich-harris.dev> * Version Packages (sveltejs#9953) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * only cache response if response has cache-control header (sveltejs#9885) * only cache response if response has cache-control header * add changeset * Version Packages (sveltejs#9955) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix: ensure styles are loaded in dev mode for routes containing special characters (sveltejs#9894) * Fix loading styles for routes containing special characters in dev mode. SvelteKit doesn't decode special characters in pathnames when loading CSS modules in dev mode, resulting in an error: Internal server error: Failed to load url /src/routes/(special)/hinnap%C3%A4ring/+page.svelte?svelte=&type=style&lang.css=&inline= (resolved id: /src/routes/(special)/hinnap%C3%A4ring/+page.svelte?svelte&type=style&lang.css). Does the file exist? Actual path: /src/routes/(special)/hinnapäring/ Fix by using decodeURI on the url.pathname when loading CSS modules. * Create stale-houses-yell.md * decodeURL inside if * add test --------- Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <git@rich-harris.dev> * feat: Warn users when submitting forms with files but no `enctype="multipart/form-data"` (sveltejs#9888) * fix: Package name keeps me from filtering with pnpm * feat: Warn users when submitting a form containing a file without the correct enctype * changeset * Update packages/kit/src/runtime/app/forms.js Co-authored-by: gtmnayan <50981692+gtm-nayan@users.noreply.github.com> * style * moar style tweaks * better test skip * Update .changeset/tasty-llamas-relate.md Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> * DRY out * only warn once per submit * Update .changeset/tasty-llamas-relate.md Co-authored-by: Rich Harris <richard.a.harris@gmail.com> --------- Co-authored-by: gtmnayan <50981692+gtm-nayan@users.noreply.github.com> Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <git@rich-harris.dev> Co-authored-by: Rich Harris <richard.a.harris@gmail.com> * Version Packages (sveltejs#9957) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * security: Stop automatically adding URLs from server-side `load` `fetch` calls to dependencies (sveltejs#9945) * feat: Add `dangerZone` config * breaking: Don't implicitly track deps in server-side fetch * changeset * bein dumb * fix: Server load invalidation * fix: Write config for server * fix: test * unsurprisingly i am dumb * docs: Clarify difference between server `fetch` and universal `fetch` * tests * rename to trackServerFetches --------- Co-authored-by: Rich Harris <git@rich-harris.dev> * Version Packages (sveltejs#9963) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: S. Elliott Johnson <sejohnson@torchcloudconsulting.com> Co-authored-by: Simon H <5968653+dummdidumm@users.noreply.github.com> Co-authored-by: Conduitry <git@chor.date> Co-authored-by: gtmnayan <50981692+gtm-nayan@users.noreply.github.com> Co-authored-by: Fernando López Guevara <fernando.lguevara@gmail.com> Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com> Co-authored-by: Rich Harris <hello@rich-harris.dev> Co-authored-by: Rich Harris <richard.a.harris@gmail.com> Co-authored-by: Rich Harris <git@rich-harris.dev> Co-authored-by: Conner <conner@semicognitive.com> Co-authored-by: gtmnayan <gtmnayan@gmail.com> Co-authored-by: Loris Sigrist <43482866+LorisSigrist@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Frank <frank@sst.dev> Co-authored-by: Frank Dumont <fj.dumont@gmail.com> Co-authored-by: Reio Remma <reio@mrstuudio.ee>

LorisSigrist added 7 commits May 10, 2023 16:41

Crawl social-image urls during prerender

0e858c9

Formatting & Linting

9af1ddc

Format changeset & added exhaustive list of crawlable urls

494f1e9

Changed severity to minor as described in #5228

fc36b82

Added support for property attribute & limited valid names to just …

eeaead0

…social tags

More tests

fa08943

Better changeset message - I'm indecisive

15ff1b6

vercel bot deployed to Preview – kit May 10, 2023 17:09 View deployment

benmccann changed the title ~~feat: Crawl urls in <meta> tags~~ feat: crawl URLs in <meta> tags May 11, 2023

benmccann reviewed May 11, 2023

View reviewed changes

.changeset/thirty-garlics-tan.md Outdated Show resolved Hide resolved

Update .changeset/thirty-garlics-tan.md

9b72fe7

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>

vercel bot deployed to Preview – kit May 11, 2023 17:38 View deployment

Rich Harris added 2 commits May 16, 2023 17:48

simplify

7df6561

simplify

3ac548d

vercel bot deployed to Preview – kit May 16, 2023 21:55 View deployment

Rich-Harris reviewed May 16, 2023

View reviewed changes

Merge branch 'master' into LorisSigrist/master

7ad7abf

vercel bot deployed to Preview – kit May 16, 2023 21:59 View deployment

vercel bot deployed to Preview – kit-svelte-dev May 16, 2023 22:22 View deployment

Removed redundant data-sanitation

28cd18e

vercel bot deployed to Preview – kit May 17, 2023 09:10 View deployment

DRY out

774bc04

Rich-Harris approved these changes May 17, 2023

View reviewed changes

vercel bot deployed to Preview – kit May 17, 2023 11:41 View deployment

Rich-Harris merged commit ab9f577 into sveltejs:master May 17, 2023

github-actions bot mentioned this pull request May 17, 2023

Version Packages #9893

Merged

eltigerchino mentioned this pull request Aug 23, 2023

<meta property="og:image" content="..." is not being picked up by crawl #5228

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: crawl URLs in `<meta>` tags #9900

feat: crawl URLs in `<meta>` tags #9900

LorisSigrist commented May 10, 2023

changeset-bot bot commented May 10, 2023 •

edited

Loading

Rich-Harris left a comment

Rich-Harris May 16, 2023

LorisSigrist May 17, 2023

feat: crawl URLs in <meta> tags #9900

feat: crawl URLs in <meta> tags #9900

Conversation

LorisSigrist commented May 10, 2023

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

Tests

Changesets

changeset-bot bot commented May 10, 2023 • edited Loading

🦋 Changeset detected

Rich-Harris left a comment

Choose a reason for hiding this comment

Rich-Harris May 16, 2023

Choose a reason for hiding this comment

LorisSigrist May 17, 2023

Choose a reason for hiding this comment

feat: crawl URLs in `<meta>` tags #9900

feat: crawl URLs in `<meta>` tags #9900

changeset-bot bot commented May 10, 2023 •

edited

Loading