feat: add Content Layer loader

[ascorbic]

This is the work-in-progress branch implementing the Content Layer RFC.

Testing

There is an example site in the content-layer fixture, which shows the use of a custom loader and the file() loader. Currently the API for end user looks like this:

import { defineCollection, file, z } from 'astro:content';
import { loader } from '../loaders/post-loader.js';

const blog = defineCollection({
	type: "experimental_data",
	loader: loader({ url: "https://jsonplaceholder.typicode.com/posts" }),
});

const dogs = defineCollection({
	type: "experimental_data",
	// The built in `file` loader loads multiple items from a JSON file
	loader: file("_data/dogs.json"),
	schema: z.object({
		breed: z.string(),
		id: z.string(),
		size: z.string(),
		origin: z.string(),
		lifespan: z.string(),
		temperament: z.array(z.string())
	}),
})
export const collections = { blog, dogs };

The blog collection shows a custom loader. These can be defined like this:

import type { Loader } from 'astro:content';

export interface PostLoaderConfig {
	url: string;
}

export function loader(config:PostLoaderConfig): Loader {
	return {
		name: "post-loader",
		load: async ({
			store, meta, logger
		}) => {
			logger.info('Loading posts');

			const lastSynced = meta.get('lastSynced');

			// Don't sync more than once a minute
			if (lastSynced && (Date.now() - Number(lastSynced) < 1000 * 60)) {
					logger.info('Skipping sync');
					return;
			}

			const posts = await fetch(config.url)
				.then((res) => res.json());

			store.clear();

			for (const post of posts) {
				store.set(post.id, post);
			}
			meta.set('lastSynced', String(Date.now()));
		},
	};
}

The API is currently defined like this:

export interface Loader<S extends ZodSchema = ZodSchema> {
	/** Unique name of the loader, e.g. the npm package name */
	name: string;
	/** Do the actual loading of the data */
	load: (context: LoaderContext) => Promise<void>;
	/** Optionally, define the schema of the data. Overridden by user-defined schema */
	schema?: S | Promise<S> | (() => S | Promise<S>);
	// Not yet implemented
	render?: (entry: any, options: any) => any;
}

export interface LoaderContext {
	collection: string;
	/** A database abstraction to store the actual data */
	store: ScopedDataStore;
	/**  A simple KV store, designed for things like sync tokens */
	meta: MetaStore;
	logger: AstroIntegrationLogger;

	settings: AstroSettings;

	/** Validates and parses the data according to the schema */
	parseData<T extends Record<string, unknown> = Record<string, unknown>>(
		props: ParseDataOptions
	): T;
}

export interface ParseDataOptions {
	/** The ID of the entry. Unique per collection */
	id: string;
	/** The raw, unvalidated data of the entry */
	data: Record<string, unknown>;
	/** An optional file path, where the entry represents a local file */
	filePath?: string;
}

Currently there is no support for loading md or mdx files using glob(), or for defining renderers.

Querying is currently limited to getCollection and getDataEntryById, with no filter options.

To test this, install the experimental version with npm install astro@experimental--content-layer

Docs

[changeset-bot]

⚠️ No Changeset found

Latest commit: 6bb89bc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

This PR is blocked because it contains a minor changeset. A reviewer will merge this at the next release if approved.

[github-actions]

Snapshots have been released for the following packages:

• astro@experimental--content-layer

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--content-layer tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/db
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/lit
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/node
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vercel
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/web-vitals
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/studio
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info npm info @astrojs/upgrade
🦋  info astro is being published because our local version (0.0.0-content-layer-20240626163213) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 4.0.7 is already published on npm
🦋  warn create-astro is not being published because version 4.8.0 is already published on npm
🦋  warn @astrojs/db is not being published because version 0.11.7 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.4.0 is already published on npm
🦋  warn @astrojs/lit is not being published because version 4.3.0 is already published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.11.1 is already published on npm
🦋  warn @astrojs/mdx is not being published because version 3.1.2 is already published on npm
🦋  warn @astrojs/node is not being published because version 8.3.2 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.1.1 is already published on npm
🦋  warn @astrojs/preact is not being published because version 3.5.0 is already published on npm
🦋  warn @astrojs/react is not being published because version 3.6.0 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.1.6 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 4.4.0 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 5.6.0 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.1.0 is already published on npm
🦋  warn @astrojs/vercel is not being published because version 7.7.2 is already published on npm
🦋  warn @astrojs/vue is not being published because version 4.5.0 is already published on npm
🦋  warn @astrojs/web-vitals is not being published because version 0.2.1 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.4.1 is already published on npm
🦋  warn @astrojs/markdown-remark is not being published because version 5.1.1 is already published on npm
🦋  warn @astrojs/studio is not being published because version 0.1.1 is already published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.3.4 is already published on npm
🦋  warn @astrojs/upgrade is not being published because version 0.3.1 is already published on npm
🦋  info Publishing "astro" at "0.0.0-content-layer-20240626163213"
🦋  success packages published successfully:
🦋  astro@0.0.0-content-layer-20240626163213
🦋  Creating git tag...
🦋  New tag:  astro@0.0.0-content-layer-20240626163213

Build Log

> root@0.0.0 build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/db, @astrojs/internal-helpers, @astrojs/lit, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/studio, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/upgrade, @astrojs/vercel, @astrojs/vue, @astrojs/web-vitals, @benchmark/timer, astro, create-astro
• Running build in 29 packages
• Remote caching enabled
::group::@astrojs/upgrade:build
cache hit, suppressing logs bc4f3f091f583ec1
::endgroup::
::group::@astrojs/telemetry:build
cache hit, suppressing logs ec7499837d68ccd3
::endgroup::
::group::create-astro:build
cache hit, suppressing logs 935b58c36a25eda3
::endgroup::
::group::@astrojs/internal-helpers:build
cache hit, suppressing logs e8d97ce643c579cb
::endgroup::
::group::@astrojs/prism:build
cache hit, suppressing logs 82fa269e60a35beb
::endgroup::
::group::@astrojs/markdown-remark:build
cache hit, suppressing logs e04f3084b2a283e9
::endgroup::
::group::astro:build
cache miss, executing 31039041a22a123f

> astro@0.0.0-content-layer-20240626163213 build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" && tsc && pnpm run postbuild


> astro@0.0.0-content-layer-20240626163213 prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"


> astro@0.0.0-content-layer-20240626163213 postbuild /home/runner/work/astro/astro/packages/astro
> astro-scripts copy "src/**/*.astro" && astro-scripts copy "src/**/*.wasm"

::endgroup::
::group::@astrojs/studio:build
cache miss, executing cde0a60f1cbd58e7

> @astrojs/studio@0.1.1 build /home/runner/work/astro/astro/packages/studio
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/lit:build
cache miss, executing 590da9547eb9758e

> @astrojs/lit@4.3.0 build /home/runner/work/astro/astro/packages/integrations/lit
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing f563b6b14ecf2e5a

> @astrojs/partytown@2.1.1 build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing 5e7f7e8dd20771bf

> @astrojs/underscore-redirects@0.3.4 build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/underscore-redirects@0.3.4 postbuild /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::@astrojs/rss:build
cache miss, executing db00d8b44f868905

> @astrojs/rss@4.0.7 build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/node:build
cache miss, executing 5769374ccf98a18d

> @astrojs/node@8.3.2 build /home/runner/work/astro/astro/packages/integrations/node
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing d2a26a51130d540a

> @astrojs/tailwind@5.1.0 build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing 58aa9788bd9506c6

> @astrojs/mdx@3.1.2 build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing c530d6d95a7dce23

> @astrojs/markdoc@0.11.1 build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing 3fcbe38704a147bd

> @astrojs/svelte@5.6.0 build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing 7aa1aec0e160a401

> @astrojs/solid-js@4.4.0 build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing 8fbb526121d666d5

> @benchmark/timer@0.0.0 build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing e04d2cbdc0664d83

> @astrojs/react@3.6.0 build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing a7fddf863f5ab50a

> @astrojs/alpinejs@0.4.0 build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vercel:build
cache miss, executing d885a3be7842fbad

> @astrojs/vercel@7.7.2 build /home/runner/work/astro/astro/packages/integrations/vercel
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing a6531d751e1096eb

> @astrojs/preact@3.5.0 build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing 5ae4796e2344dae1

> @astrojs/vue@4.5.0 build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing e8c6fe714a7f43f4

> @astrojs/sitemap@3.1.6 build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/db:build
cache miss, executing b1fa98442d6f43ce

> @astrojs/db@0.11.7 build /home/runner/work/astro/astro/packages/db
> astro-scripts build "src/**/*.ts" && tsc && pnpm types:virtual


> @astrojs/db@0.11.7 types:virtual /home/runner/work/astro/astro/packages/db
> tsc -p ./tsconfig.virtual.json

::endgroup::
::group::@astrojs/web-vitals:build
cache miss, executing 046aeb953d322139

> @astrojs/web-vitals@0.2.1 build /home/runner/work/astro/astro/packages/integrations/web-vitals
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    27 successful, 27 total
Cached:    6 cached, 27 total
  Time:    51.34s 

[ematipico]

Let's get the party started!

Get out :D

[Suven]

For others finding their way to this PR and wanting to try out a recent version of this feature, the tag and method signature changed a little:

Instead of npm install astro@experimental--content-layer it now is npm install astro@experimental--contentlayer.

And instead of type: "experimental_data", it now is type: 'experimental_content',. The references and fixtures can be found here: https://github.com/withastro/astro/tree/content-layer/packages/astro/test/fixtures/content-layer