[draft RFC] Glob all files into an in-memory database, introducing "Astro.db"

[lukeocodes]

Below is a draft of an RFC. It's very specific to my usecase and I'd like some feedback/suggestions before I submit this as an RFC.

• Start Date: 2022-08-18
• Reference Issues:
• Implementation PR:

Summary

Glob all files from a specific directory (e.g. /content) and load them into an in-memory javascript document oriented database (e.g. LokiJS).

Example

Assuming all content in /content/blog had been ingested when running Astro, I see us using it like this when creating dynamic page routes.

export async function getStaticPaths() {
    const posts = await Astro.db(`blog/`)
        .where({
        { published: { $ne: false } },
        })
        .sortBy('published_at', 'desc')
        .fetch()

    return posts.map((post) => {
        return {
            params: {
                slug,
            },
            props: {
                post,
            },
        };
    });
}

Motivation

As we're building our new site, I've noticed an extraordinary amount of globs. We also sometimes need to import.glob.meta for content in files where we need the filename as well as the content (JSON files). This leads to a fair amount of quite repetitive code through the codebase.

For example, this is just from the blog microsite.

Loading all our blog posts:

const allPosts = await Astro.glob('../content/blog/posts/**/index.md')

All our authors and the filename (to use as a slug), which is also repeated for all our categories:

const authorFiles = import.meta.glob("../../content/blog/authors/*.json");

async function getAuthors() {
  let authors = [];
  for (const path in authorFiles) { // for every file
    const author = await authorFiles[path](); // import the file
    const slug = path.split("/").pop().replace(".json", " ").trim(); // get the filename
    authors.push({ ...(author as object), slug }); // create a new object with author details and filename
  }
  return authors;
}

const allAuthors = await getAuthors();

All unique post tags:

const allPosts = await Astro.glob('../content/blog/posts/**/index.md')

let postTags = []
for (const post of allPosts) {
  postTags.push(...post.frontmatter.tags) // manually create an array of all post tags (not unique yet)
}

Not an exhaustive list. We still load the posts for each category, each tag, each author, the author and category for each post display page, and more.

Detailed design

...wip...

Drawbacks

Why should we not do this? Please consider:

• Increased memory usage as we load all contents into a javascript document oriented database.
• Slower builds with the increased memory usage is possible.
• Change of direction from earlier design decisions, perhaps?

Alternatives

• Consuming the markdown content into a serverless endpoint.
• Building Astro/db as a plugin.
• Using an API-based Headless CMS entirely, and not storing markdown files with the Astro sites.

Adoption strategy

I believe a new top-level feature on the Astro class will limit the impact on users who do not plan to implement this feature. Something like Astro.db provides a clear and new interface to be able to query for any content that has been loaded into the database.

Please consider:

• If we implement this proposal, how will existing Astro developers adopt it?
• Is this a breaking change? Can we write a codemod?
• Can we provide a runtime adapter library for the original API it replaces?
• How will this affect other projects in the Astro ecosystem?

Unresolved questions

• When running the dev environment, how will changed files update their database-loaded selves?

[tanepiper]

+1 we also believe this is a step towards incremental building and would hope there would be adapters for this too (e.g. offload to SQLite, or some K/V store).

A prior art similar is how Obsidian has an internal database that can be queried using some basic SQL, a feature like this allows for Astro and framework components to get data via these queries (think Last X blog posts)

[lukeocodes]

Earlier versions of nuxt/content inspired this RFC. It used LokiJS to store javascript objects as documents in a NoSQL fashion, and even modelled the SDK on the Mongo pattern, with LokiJS having plans to implement Mongo DB compatibility in the future.

I think adapters is an absolutely fantastic idea for this, allowing larger sites to take the burden off Node by using a KVS