Content Layer (#11360)

* Empty commit * Changeset * feat: add Content Layer loader (#11334) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * fix: sync content layer in dev (#11365) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * Sync content layer in dev * feat: add typegen for loaders (#11358) * fix: watch for content layer changes (#11371) * fix: watch for content layer changes * Add test * feat: adds simple loader (#11386) * wip * Add simple loader * Fix type guard * Tighten loader schema * Add loader function to type * Reinstall vitest * feat: add glob loader (#11398) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * feat: add markdown rendering to content layer (#11440) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * wip: add markdown rendering to content layer * use cached entries * CLean up types * Instrument more of the build * Add digest helper * Add comments * Make image extraction work * feat: image support for content layer (#11469) * wip * wip * Add image to benchmark * Stub assets if missing * Resolve assets in data * Ignore virtual module * Format * rm log * Handle images when using cached data * Fix CCC * Add a comment * Changes from review * Format * Use relative paths for asset files * Pass all md props to getImage * Ensure dotastro dir exists * Fix tests * Changes from review * Don't use temp array in getcollection * Add error handling * Format * Handle paths that are already relative * Dedupe sync runs * Fix syncing in dev * Changes from review * Windows paths ftw * feat(content-layer): support references in content layer (#11494) * Support references in content layer * Fix utf8 rendering * Warn for invalid entries * Fix test * lol windows paths * Remove assertion * chore: fix content layer types (#11527) * Add experimental_content type * Fix import * Make data store methods generic * fix loader types * Lockfile * Clean content layer with `--force` (#11541) * Clearn content layer with `--force` * Add tests * Document --force flag * Fixes to content layer render types (#11558) * Lockfile * feat: use devalue to serialize content layer data (#11562) * feat: use devalue to serialize content layer data * Fix import * Use devalue stringify * Unused import * Propagate error messages correctly * Support --force flag in sync and dev (#11581) * Support --force flag in sync and dev * Fix test * Separate render function and merge content layer types (#11579) * Separate render function and merge content layer types * Changes from review * fix: clear content layer cache if config has changed (#11591) * fix: clear content layer cache if config has changed * Add test * Watch config * Change from review * fix: skip glob files in content dir (#11622) * fix: skip glob files in content dir * Changes from review * Log pattern * Refactor content layer into shared instance (#11625) * Refactor content layer into shared instance * Clean up when testing * Handle cleanup * fix: support filters in content layer getCollection (#11631) * Throw when using deprecated getEntryByX functions with content layer (#11637) * Updates to content layer types and jsdocs (#11643) * Add hot key to reload content layer (#11626) * Add hot key to reload content layer * Fix filename * Remove cli message * Update example * Change key to "s" * feat: handle simple mdx rendering (#11633) * feat: handle simple mdx rendering * cleanup * feedback * fix regression * remove log * flip condition * update tests * log collections to understand the error * let's try this alternative * try parallel test to understand the issue * chore: use a new fixture to fix tests * rebase and docs * fix regressions * remove old code * address feedback * rename param * log error * rebase * chore: try a different cache dir to solve the error test * fix invalidation of the module when there's no store available * address suggestion * run formatter * update lock file * Lint * Add experimental content layer flag (#11652) * Add experimental content layer flag * Syntax and format * Aside * Format * Reset content config between runs * Update fixture * Update terminology * Lint * wut * Normalize render function return value (#11663) * Add markdoc support to content layer (#11664) * Add markdoc support to content layer * Switch test to cheerio * Update benchmarks * update lock file * Update content layer flag docs (#11682) * Update content layer flag docs * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * More markdoc * Typo * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Add changeset for content layer experimental release (#11644) * Add changeset for content layer experimental release * Update changeset * Update .changeset/smooth-chicken-wash.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * feat: injectTypes (#11551) * feat: make inline config 1st arg * fix: run config done in sync * feat: start working on injectTypes * feat: write files * feat: adapt core features * feat: migrate db to injectTypes * feat: special db handling * feat: update settings instead of workarounds * fix: create dotAstroDir * feat: refactor sync tests * fix: path * fix: paths * chore: add comments * feat: overwrite content file if exists * chore: remove unused db env related code * feat: use dotAstroDir for settings * chore: simplify astro env sync * feat: use dotAstroDir for preferences * feat: handle db in integration api * chore: reorganize * feat: format * feat: add test * Discard changes to examples/basics/astro.config.mjs * Discard changes to examples/basics/package.json * Discard changes to pnpm-lock.yaml * chore: remove test files * feat: update examples dts * fix: dts * chore: changesets * fix: indentation * Apply suggestions from code review Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> * Apply suggestions from code review Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> * chore: format * Update packages/astro/src/integrations/hooks.ts * Update .changeset/mean-horses-kiss.md * feat: remove formatting * feat: handle fs errors * feat: remove astro:db special path handling * feat: add fs error * Update packages/astro/src/content/types-generator.ts * Update .changeset/mean-horses-kiss.md * Update errors-data.ts * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> * Add file generation and flag for content intellisense (#11639) * feat: add type to infer input type of collection * refactor: * feat: generate json schema for content too * feat: generate a manifest of all the collections * refactor: unnecessary type * fix: only add content collections to manifest * chore: changeset * fix: generate file URLs * fix: flag it properly * fix: save in lower case * docs: add jsdoc to experimental option * nit: move function out * fix: match vscode flag name * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update serious-pumas-run.md * test: add tests * Add content layer support * Apply suggestions from code review * fix: test * Update .changeset/serious-pumas-run.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Apply suggestions from code review * Remove check for json --------- Co-authored-by: Matt Kane <m@mk.gg> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * nit: use same filesystem error as injectTypes * fix: code component was missing support for meta string (#11605) * fix: code component was missing support for meta string Fixed #11604 * Create odd-buttons-pay.md * <Code>: add reference link for meta prop * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update .changeset/odd-buttons-pay.md * Update .changeset/odd-buttons-pay.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> * Deprecates exporting prerender with dynamic values (#11657) * wip * done i think * Add changeset * Use hook instead * Reorder hooks [skip ci] * Update .changeset/eleven-pens-glow.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Fix run * Fix link * Add link Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> * More accurate migration [skip ci] --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> * Use node parseArgs instead of yargs-parser and arg (#11645) * wip * done * Add changeset * Format * Update * Fix houston * Fix test * Fix test * [ci] format * resolve conflict --------- Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Florian Lefebvre <contact@florian-lefebvre.dev> Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> Co-authored-by: Julien Cayzac <jcayzac@users.noreply.github.com> Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com> Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> Co-authored-by: Bjorn Lu <ematipico@users.noreply.github.com>
withastro · Aug 14, 2024 · a79a8b0 · a79a8b0
1 parent 19a7259
commit a79a8b0
Show file tree

Hide file tree

Showing 155 changed files with 5,274 additions and 573 deletions.
diff --git a/.changeset/fresh-fans-study.md b/.changeset/fresh-fans-study.md
@@ -0,0 +1,18 @@
+---
+'@astrojs/db': minor
+---
+
+Changes how type generation works
+
+The generated `.d.ts` file is now at a new location:
+
+```diff
+- .astro/db-types.d.ts
++ .astro/integrations/astro_db/db.d.ts
+```
+
+The following line can now be removed from `src/env.d.ts`:
+
+```diff
+- /// <reference path="../.astro/db-types.d.ts" />
+```
diff --git a/.changeset/mean-horses-kiss.md b/.changeset/mean-horses-kiss.md
@@ -0,0 +1,35 @@
+---
+'astro': minor
+---
+
+Adds a new [`injectTypes()` utility](https://docs.astro.build/en/reference/integrations-reference/#injecttypes-options) to the Integration API and refactors how type generation works
+
+Use `injectTypes()` in the `astro:config:done` hook to inject types into your user's project by adding a new a `*.d.ts` file.
+
+The `filename` property will be used to generate a file at `/.astro/integrations/<normalized_integration_name>/<normalized_filename>.d.ts` and must end with `".d.ts"`.
+
+The `content` property will create the body of the file, and must be valid TypeScript.
+
+Additionally, `injectTypes()` returns a URL to the normalized path so you can overwrite its content later on, or manipulate it in any way you want.
+
+```js
+// my-integration/index.js
+export default {
+  name: 'my-integration',
+  'astro:config:done': ({ injectTypes }) => {
+    injectTypes({
+      filename: "types.d.ts",
+      content: "declare module 'virtual:my-integration' {}"
+    })
+  }
+};
+```
+
+Codegen has been refactored. Although `src/env.d.ts` will continue to work as is, we recommend you update it:
+
+```diff
+- /// <reference types="astro/client" />
++ /// <reference path="../.astro/types.d.ts" />
+- /// <reference path="../.astro/env.d.ts" />
+- /// <reference path="../.astro/actions.d.ts" />
+```
diff --git a/.changeset/serious-pumas-run.md b/.changeset/serious-pumas-run.md
@@ -0,0 +1,21 @@
+---
+'astro': minor
+---
+
+Adds support for Intellisense features (e.g. code completion, quick hints) for your content collection entries in compatible editors under the `experimental.contentIntellisense` flag.
+
+```js
+import { defineConfig } from 'astro';
+
+export default defineConfig({
+	experimental: {
+		contentIntellisense: true
+	}
+})
+```
+
+When enabled, this feature will generate and add JSON schemas to the `.astro` directory in your project. These files can be used by the Astro language server to provide Intellisense inside content files (`.md`, `.mdx`, `.mdoc`).
+
+Note that at this time, this also require enabling the `astro.content-intellisense` option in your editor, or passing the `contentIntellisense: true` initialization parameter to the Astro language server for editors using it directly.
+
+See the [experimental content Intellisense docs](https://docs.astro.build/en/reference/configuration-reference/#experimentalcontentintellisense) for more information updates as this feature develops.
diff --git a/.changeset/smooth-chicken-wash.md b/.changeset/smooth-chicken-wash.md
@@ -0,0 +1,107 @@
+---
+'astro': minor
+---
+
+Adds experimental support for the Content Layer API.
+
+The new Content Layer API builds upon content collections, taking them beyond local files in `src/content/` and allowing you to fetch content from anywhere, including remote APIs. These new collections work alongside your existing content collections, and you can migrate them to the new API at your own pace. There are significant improvements to performance with large collections of local files. 
+
+### Getting started
+
+To try out the new Content Layer API, enable it in your Astro config:
+
+```js
+import { defineConfig } from 'astro';
+
+export default defineConfig({
+  experimental: {
+    contentLayer: true
+  }
+})
+```
+
+You can then create collections in your `src/content/config.ts` using the Content Layer API.
+
+### Loading your content
+
+The core of the new Content Layer API is the loader, a function that fetches content from a source and caches it in a local data store. Astro 4.14 ships with built-in `glob()` and `file()` loaders to handle your local Markdown, MDX, Markdoc, and JSON files:
+
+```ts {3,7}
+// src/content/config.ts
+import { defineCollection, z } from 'astro:content';
+import { glob } from 'astro/loaders';
+
+const blog = defineCollection({
+  // The ID is a slug generated from the path of the file relative to `base`
+  loader: glob({ pattern: "**/*.md", base: "./src/data/blog" }),
+  schema: z.object({
+    title: z.string(),
+    description: z.string(),
+    publishDate: z.coerce.date(),
+  })
+});
+
+export const collections = { blog };
+```
+
+You can then query using the existing content collections functions, and enjoy a simplified `render()` function to display your content:
+
+```astro
+---
+import { getEntry, render } from 'astro:content';
+
+const post = await getEntry('blog', Astro.params.slug);
+
+const { Content } = await render(entry);
+---
+
+<Content />
+```
+
+### Creating a loader
+
+You're not restricted to the built-in loaders – we hope you'll try building your own. You can fetch content from anywhere and return an array of entries:
+
+```ts
+// src/content/config.ts
+const countries = defineCollection({
+  loader: async () => {
+    const response = await fetch("https://restcountries.com/v3.1/all");
+    const data = await response.json();
+    // Must return an array of entries with an id property,
+    // or an object with IDs as keys and entries as values
+    return data.map((country) => ({
+      id: country.cca3,
+      ...country,
+    }));
+  },
+  // optionally add a schema to validate the data and make it type-safe for users
+  // schema: z.object...
+});
+
+export const collections = { countries };
+```
+
+For more advanced loading logic, you can define an object loader. This allows incremental updates and conditional loading, and gives full access to the data store. It also allows a loader to define its own schema, including generating it dynamically based on the source API. See the [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md#loaders) for more details.
+
+### Sharing your loaders
+
+Loaders are better when they're shared. You can create a package that exports a loader and publish it to npm, and then anyone can use it on their site. We're excited to see what the community comes up with! To get started, [take a look at some examples](https://github.com/ascorbic/astro-loaders/). Here's how to load content using an RSS/Atom feed loader:
+
+```ts
+// src/content/config.ts
+import { defineCollection } from "astro:content";
+import { feedLoader } from "@ascorbic/feed-loader";
+
+const podcasts = defineCollection({
+  loader: feedLoader({
+    url: "https://feeds.99percentinvisible.org/99percentinvisible",
+  }),
+});
+
+export const collections = { podcasts };
+```
+
+### Learn more
+
+To find out more about using the Content Layer API, check out [the Content Layer RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md) and [share your feedback](https://github.com/withastro/roadmap/pull/982).
diff --git a/benchmark/bench/memory.js b/benchmark/bench/memory.js
@@ -18,7 +18,7 @@ export async function run(projectDir, outputFile) {
 	const outputFilePath = fileURLToPath(outputFile);
 
 	console.log('Building and benchmarking...');
-	await execaCommand(`node --expose-gc --max_old_space_size=256 ${astroBin} build`, {
+	await execaCommand(`node --expose-gc --max_old_space_size=10000 ${astroBin} build --silent`, {
 		cwd: root,
 		stdio: 'inherit',
 		env: {

diff --git a/benchmark/make-project/image.jpg b/benchmark/make-project/image.jpg
diff --git a/benchmark/make-project/markdown-cc1.js b/benchmark/make-project/markdown-cc1.js
@@ -0,0 +1,63 @@
+import fs from 'node:fs/promises';
+import { loremIpsumMd } from './_util.js';
+
+/**
+ * @param {URL} projectDir
+ */
+export async function run(projectDir) {
+	await fs.rm(projectDir, { recursive: true, force: true });
+	await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
+	await fs.mkdir(new URL('./src/content/blog', projectDir), { recursive: true });
+	await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./src/image.jpg', projectDir));
+
+	const promises = [];
+
+
+	for (let i = 0; i < 10000; i++) {
+		const content = `\
+# Article ${i}
+
+${loremIpsumMd}
+
+![image ${i}](../../image.jpg)
+
+
+`;
+		promises.push(
+			fs.writeFile(new URL(`./src/content/blog/article-${i}.md`, projectDir), content, 'utf-8')
+		);
+	}
+
+
+	await fs.writeFile(
+		new URL(`./src/pages/blog/[...slug].astro`, projectDir),
+		`\
+---
+import { getCollection } from 'astro:content';
+export async function getStaticPaths() {
+  const blogEntries = await getCollection('blog');
+  return blogEntries.map(entry => ({
+    params: { slug: entry.slug }, props: { entry },
+  }));
+}
+const { entry } = Astro.props;
+const { Content } = await entry.render();
+---
+<h1>{entry.data.title}</h1>
+<Content />
+`,
+		'utf-8'
+	);
+
+	await Promise.all(promises);
+
+	await fs.writeFile(
+		new URL('./astro.config.js', projectDir),
+		`\
+import { defineConfig } from 'astro/config';
+
+export default defineConfig({
+});`,
+		'utf-8'
+	);
+}
diff --git a/benchmark/make-project/markdown-cc2.js b/benchmark/make-project/markdown-cc2.js
@@ -0,0 +1,80 @@
+import fs from 'node:fs/promises';
+import { loremIpsumMd } from './_util.js';
+
+/**
+ * @param {URL} projectDir
+ */
+export async function run(projectDir) {
+	await fs.rm(projectDir, { recursive: true, force: true });
+	await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
+	await fs.mkdir(new URL('./data/blog', projectDir), { recursive: true });
+	await fs.mkdir(new URL('./src/content', projectDir), { recursive: true });
+	await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./image.jpg', projectDir));
+
+	const promises = [];
+
+	for (let i = 0; i < 10000; i++) {
+		const content = `\
+# Article ${i}
+
+${loremIpsumMd}
+
+![image ${i}](../../image.jpg)
+
+`;
+		promises.push(
+			fs.writeFile(new URL(`./data/blog/article-${i}.md`, projectDir), content, 'utf-8')
+		);
+	}
+
+	await fs.writeFile(
+		new URL(`./src/content/config.ts`, projectDir),
+		/*ts */ `
+		import { defineCollection, z } from 'astro:content';
+		import { glob } from 'astro/loaders';
+
+		const blog = defineCollection({
+			loader: glob({ pattern: '*', base: './data/blog' }),
+		});
+		
+		export const collections = { blog }
+
+		`
+	);
+
+	await fs.writeFile(
+		new URL(`./src/pages/blog/[...slug].astro`, projectDir),
+		`\
+---
+import { getCollection, render } from 'astro:content';
+export async function getStaticPaths() {
+  const blogEntries = await getCollection('blog');
+  return blogEntries.map(entry => ({
+    params: { slug: entry.id }, props: { entry },
+  }));
+}
+const { entry } = Astro.props;
+const { Content } = await render(entry);
+
+---
+<h1>{entry.data.title}</h1>
+<Content />
+`,
+		'utf-8'
+	);
+
+	await Promise.all(promises);
+
+	await fs.writeFile(
+		new URL('./astro.config.js', projectDir),
+		`\
+import { defineConfig } from 'astro/config';
+
+export default defineConfig({
+		experimental: {
+			contentLayer: true
+		}
+});`,
+		'utf-8'
+	);
+}