Skip to content

Commit

Permalink
feat(db-postgres, db-sqlite): drizzle schema generation (#9953)
Browse files Browse the repository at this point in the history 
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.

Example of generated file with the website template: 
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03

Video that shows the power:


https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236


We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
  pgTable,
  index,
  foreignKey,
  integer,
  text,
  varchar,
  jsonb,
  boolean,
  numeric,
  serial,
  timestamp,
  uniqueIndex,
  pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```


Fixes #4318

In the future we can also support types generation for mongoose / raw
mongodb results.
  • Loading branch information
@r1tsuu
r1tsuu authored Dec 19, 2024
1 parent ba0e7ae commit 23f1ed4
Show file tree
Hide file tree
Showing 39 changed files with 4,519 additions and 32 deletions.
64 changes: 58 additions & 6 deletions docs/database/postgres.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,21 +65,33 @@ export default buildConfig({
| `schemaName` (experimental) | A string for the postgres schema to use, defaults to 'public'. |
| `idType` | A string of 'serial', or 'uuid' that is used for the data type given to id columns. |
| `transactionOptions` | A PgTransactionConfig object for transactions, or set to `false` to disable using transactions. [More details](https://orm.drizzle.team/docs/transactions) |
| `disableCreateDatabase` | Pass `true` to disable auto database creation if it doesn't exist. Defaults to `false`. |
| `disableCreateDatabase` | Pass `true` to disable auto database creation if it doesn't exist. Defaults to `false`. |
| `localesSuffix` | A string appended to the end of table names for storing localized fields. Default is '_locales'. |
| `relationshipsSuffix` | A string appended to the end of table names for storing relationships. Default is '_rels'. |
| `versionsSuffix` | A string appended to the end of table names for storing versions. Defaults to '_v'. |
| `beforeSchemaInit` | Drizzle schema hook. Runs before the schema is built. [More Details](#beforeschemainit) |
| `afterSchemaInit` | Drizzle schema hook. Runs after the schema is built. [More Details](#afterschemainit) |
| `generateSchemaOutputFile` | Override generated schema from `payload generate:db-schema` file path. Defaults to `{CWD}/src/payload-generated.schema.ts` |

## Access to Drizzle

After Payload is initialized, this adapter will expose the full power of Drizzle to you for use if you need it.

You can access Drizzle as follows:
To ensure type-safety, you need to generate Drizzle schema first with:
```sh
npx payload generate:db-schema
```

```text
payload.db.drizzle
Then, you can access Drizzle as follows:
```ts
import { posts } from './payload-generated-schema'
// To avoid installing Drizzle, you can import everything that drizzle has from our re-export path.
import { eq, sql, and } from '@payloadcms/db-postgres/drizzle'

// Drizzle's Querying API: https://orm.drizzle.team/docs/rqb
const posts = await payload.db.drizzle.query.posts.findMany()
// Drizzle's Select API https://orm.drizzle.team/docs/select
const result = await payload.db.drizzle.select().from(posts).where(and(eq(posts.id, 50), sql`lower(${posts.title}) = 'example post title'`))
```

## Tables, relations, and enums
Expand Down Expand Up @@ -114,7 +126,7 @@ Runs before the schema is built. You can use this hook to extend your database s

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'
import { integer, pgTable, serial } from '@payloadcms/db-postgres/drizzle/pg-core'

postgresAdapter({
beforeSchemaInit: [
Expand Down Expand Up @@ -194,7 +206,7 @@ The following example adds the `extra_integer_column` column and a composite ind

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { index, integer } from '@payloadcms/db-postgres/drizzle/pg-core'
import { buildConfig } from 'payload'

export default buildConfig({
Expand Down Expand Up @@ -236,3 +248,43 @@ export default buildConfig({
})

```

### Note for generated schema:
Columns and tables, added in schema hooks won't be added to the generated via `payload generate:db-schema` Drizzle schema.
If you want them to be there, you either have to edit this file manually or mutate the internal Payload "raw" SQL schema in the `beforeSchemaInit`:

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'

postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
// Add a new table
schema.rawTables.myTable = {
name: 'my_table',
columns: [{
name: 'my_id',
type: 'serial',
primaryKey: true
}],
}

// Add a new column to generated by Payload table:
schema.rawTables.posts.columns.customColumn = {
name: 'custom_column',
// Note that Payload SQL doesn't support everything that Drizzle does.
type: 'integer',
notNull: true
}
// Add a new index to generated by Payload table:
schema.rawTables.posts.indexes.customColumnIdx = {
name: 'custom_column_idx',
unique: true,
on: ['custom_column']
}

return schema
},
],
})
```
89 changes: 71 additions & 18 deletions docs/database/sqlite.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,41 @@ export default buildConfig({

## Options

| Option | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `client` \* | [Client connection options](https://orm.drizzle.team/docs/get-started-sqlite#turso) that will be passed to `createClient` from `@libsql/client`. |
| `push` | Disable Drizzle's [`db push`](https://orm.drizzle.team/kit-docs/overview#prototyping-with-db-push) in development mode. By default, `push` is enabled for development mode only. |
| `migrationDir` | Customize the directory that migrations are stored. |
| `logger` | The instance of the logger to be passed to drizzle. By default Payload's will be used. |
| `idType` | A string of 'number', or 'uuid' that is used for the data type given to id columns. |
| `transactionOptions` | A SQLiteTransactionConfig object for transactions, or set to `false` to disable using transactions. [More details](https://orm.drizzle.team/docs/transactions) |
| `localesSuffix` | A string appended to the end of table names for storing localized fields. Default is '_locales'. |
| `relationshipsSuffix` | A string appended to the end of table names for storing relationships. Default is '_rels'. |
| `versionsSuffix` | A string appended to the end of table names for storing versions. Defaults to '_v'. |
| `beforeSchemaInit` | Drizzle schema hook. Runs before the schema is built. [More Details](#beforeschemainit) |
| `afterSchemaInit` | Drizzle schema hook. Runs after the schema is built. [More Details](#afterschemainit) |
| Option | Description |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `client` \* | [Client connection options](https://orm.drizzle.team/docs/get-started-sqlite#turso) that will be passed to `createClient` from `@libsql/client`. |
| `push` | Disable Drizzle's [`db push`](https://orm.drizzle.team/kit-docs/overview#prototyping-with-db-push) in development mode. By default, `push` is enabled for development mode only. |
| `migrationDir` | Customize the directory that migrations are stored. |
| `logger` | The instance of the logger to be passed to drizzle. By default Payload's will be used. |
| `idType` | A string of 'number', or 'uuid' that is used for the data type given to id columns. |
| `transactionOptions` | A SQLiteTransactionConfig object for transactions, or set to `false` to disable using transactions. [More details](https://orm.drizzle.team/docs/transactions) |
| `localesSuffix` | A string appended to the end of table names for storing localized fields. Default is '_locales'. |
| `relationshipsSuffix` | A string appended to the end of table names for storing relationships. Default is '_rels'. |
| `versionsSuffix` | A string appended to the end of table names for storing versions. Defaults to '_v'. |
| `beforeSchemaInit` | Drizzle schema hook. Runs before the schema is built. [More Details](#beforeschemainit) |
| `afterSchemaInit` | Drizzle schema hook. Runs after the schema is built. [More Details](#afterschemainit) |
| `generateSchemaOutputFile` | Override generated schema from `payload generate:db-schema` file path. Defaults to `{CWD}/src/payload-generated.schema.ts` |

## Access to Drizzle

After Payload is initialized, this adapter will expose the full power of Drizzle to you for use if you need it.

You can access Drizzle as follows:
To ensure type-safety, you need to generate Drizzle schema first with:
```sh
npx payload generate:db-schema
```

```text
payload.db.drizzle
Then, you can access Drizzle as follows:
```ts
// Import table from the generated file
import { posts } from './payload-generated-schema'
// To avoid installing Drizzle, you can import everything that drizzle has from our re-export path.
import { eq, sql, and } from '@payloadcms/db-sqlite/drizzle'

// Drizzle's Querying API: https://orm.drizzle.team/docs/rqb
const posts = await payload.db.drizzle.query.posts.findMany()
// Drizzle's Select API https://orm.drizzle.team/docs/select
const result = await payload.db.drizzle.select().from(posts).where(and(eq(posts.id, 50), sql`lower(${posts.title}) = 'example post title'`))
```

## Tables and relations
Expand Down Expand Up @@ -89,7 +102,7 @@ Runs before the schema is built. You can use this hook to extend your database s

```ts
import { sqliteAdapter } from '@payloadcms/db-sqlite'
import { integer, sqliteTable } from 'drizzle-orm/sqlite-core'
import { integer, sqliteTable } from '@payloadcms/db-sqlite/drizzle/sqlite-core'

sqliteAdapter({
beforeSchemaInit: [
Expand Down Expand Up @@ -169,7 +182,7 @@ The following example adds the `extra_integer_column` column and a composite ind

```ts
import { sqliteAdapter } from '@payloadcms/db-sqlite'
import { index, integer } from 'drizzle-orm/sqlite-core'
import { index, integer } from '@payloadcms/db-sqlite/drizzle/sqlite-core'
import { buildConfig } from 'payload'

export default buildConfig({
Expand Down Expand Up @@ -211,3 +224,43 @@ export default buildConfig({
})

```

### Note for generated schema:
Columns and tables, added in schema hooks won't be added to the generated via `payload generate:db-schema` Drizzle schema.
If you want them to be there, you either have to edit this file manually or mutate the internal Payload "raw" SQL schema in the `beforeSchemaInit`:

```ts
import { sqliteAdapter } from '@payloadcms/db-sqlite'

sqliteAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
// Add a new table
schema.rawTables.myTable = {
name: 'my_table',
columns: [{
name: 'my_id',
type: 'integer',
primaryKey: true
}],
}

// Add a new column to generated by Payload table:
schema.rawTables.posts.columns.customColumn = {
name: 'custom_column',
// Note that Payload SQL doesn't support everything that Drizzle does.
type: 'integer',
notNull: true
}
// Add a new index to generated by Payload table:
schema.rawTables.posts.indexes.customColumnIdx = {
name: 'custom_column_idx',
unique: true,
on: ['custom_column']
}

return schema
},
],
})
```
1 change: 1 addition & 0 deletions package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
"clean:build": "node ./scripts/delete-recursively.js 'media/' '**/dist/' '**/.cache/' '**/.next/' '**/.turbo/' '**/tsconfig.tsbuildinfo' '**/payload*.tgz' '**/meta_*.json'",
"clean:cache": "node ./scripts/delete-recursively.js node_modules/.cache! packages/payload/node_modules/.cache! .next/*",
"dev": "cross-env NODE_OPTIONS=--no-deprecation tsx ./test/dev.ts",
"dev:generate-db-schema": "pnpm runts ./test/generateDatabaseSchema.ts",
"dev:generate-graphql-schema": "pnpm runts ./test/generateGraphQLSchema.ts",
"dev:generate-importmap": "pnpm runts ./test/generateImportMap.ts",
"dev:generate-types": "pnpm runts ./test/generateTypes.ts",
Expand Down
40 changes: 40 additions & 0 deletions packages/db-postgres/package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,26 @@
"import": "./src/exports/migration-utils.ts",
"types": "./src/exports/migration-utils.ts",
"default": "./src/exports/migration-utils.ts"
},
"./drizzle": {
"import": "./src/drizzle-proxy/index.ts",
"types": "./src/drizzle-proxy/index.ts",
"default": "./src/drizzle-proxy/index.ts"
},
"./drizzle/pg-core": {
"import": "./src/drizzle-proxy/pg-core.ts",
"types": "./src/drizzle-proxy/pg-core.ts",
"default": "./src/drizzle-proxy/pg-core.ts"
},
"./drizzle/node-postgres": {
"import": "./src/drizzle-proxy/node-postgres.ts",
"types": "./src/drizzle-proxy/node-postgres.ts",
"default": "./src/drizzle-proxy/node-postgres.ts"
},
"./drizzle/relations": {
"import": "./src/drizzle-proxy/relations.ts",
"types": "./src/drizzle-proxy/relations.ts",
"default": "./src/drizzle-proxy/relations.ts"
}
},
"main": "./src/index.ts",
Expand Down Expand Up @@ -90,6 +110,26 @@
"import": "./dist/exports/migration-utils.js",
"types": "./dist/exports/migration-utils.d.ts",
"default": "./dist/exports/migration-utils.js"
},
"./drizzle": {
"import": "./dist/drizzle-proxy/index.js",
"types": "./dist/drizzle-proxy/index.d.ts",
"default": "./dist/drizzle-proxy/index.js"
},
"./drizzle/pg-core": {
"import": "./dist/drizzle-proxy/pg-core.js",
"types": "./dist/drizzle-proxy/pg-core.d.ts",
"default": "./dist/drizzle-proxy/pg-core.js"
},
"./drizzle/node-postgres": {
"import": "./dist/drizzle-proxy/node-postgres.js",
"types": "./dist/drizzle-proxy/node-postgres.d.ts",
"default": "./dist/drizzle-proxy/node-postgres.js"
},
"./drizzle/relations": {
"import": "./dist/drizzle-proxy/relations.js",
"types": "./dist/drizzle-proxy/relations.d.ts",
"default": "./dist/drizzle-proxy/relations.js"
}
},
"main": "./dist/index.js",
Expand Down
1 change: 1 addition & 0 deletions packages/db-postgres/src/drizzle-proxy/index.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from 'drizzle-orm'
1 change: 1 addition & 0 deletions packages/db-postgres/src/drizzle-proxy/node-postgres.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from 'drizzle-orm/node-postgres'
1 change: 1 addition & 0 deletions packages/db-postgres/src/drizzle-proxy/pg-core.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from 'drizzle-orm/pg-core'
1 change: 1 addition & 0 deletions packages/db-postgres/src/drizzle-proxy/relations.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from 'drizzle-orm/relations'
11 changes: 11 additions & 0 deletions packages/db-postgres/src/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
create,
createGlobal,
createGlobalVersion,
createSchemaGenerator,
createVersion,
deleteMany,
deleteOne,
Expand All @@ -36,6 +37,7 @@ import {
updateVersion,
} from '@payloadcms/drizzle'
import {
columnToCodeConverter,
countDistinct,
createDatabase,
createExtensions,
Expand Down Expand Up @@ -106,6 +108,14 @@ export function postgresAdapter(args: Args): DatabaseAdapterObj<PostgresAdapter>
json: true,
},
fieldConstraints: {},
generateSchema: createSchemaGenerator({
columnToCodeConverter,
corePackageSuffix: 'pg-core',
defaultOutputFile: args.generateSchemaOutputFile,
enumImport: 'pgEnum',
schemaImport: 'pgSchema',
tableImport: 'pgTable',
}),
idType: postgresIDType,
initializing,
localesSuffix: args.localesSuffix || '_locales',
Expand Down Expand Up @@ -188,4 +198,5 @@ export function postgresAdapter(args: Args): DatabaseAdapterObj<PostgresAdapter>
}

export type { MigrateDownArgs, MigrateUpArgs } from '@payloadcms/drizzle/postgres'
export { geometryColumn } from '@payloadcms/drizzle/postgres'
export { sql } from 'drizzle-orm'
15 changes: 14 additions & 1 deletion packages/db-postgres/src/types.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import type {
} from '@payloadcms/drizzle/postgres'
import type { DrizzleAdapter } from '@payloadcms/drizzle/types'
import type { DrizzleConfig } from 'drizzle-orm'
import type { NodePgDatabase } from 'drizzle-orm/node-postgres'
import type { PgSchema, PgTableFn, PgTransactionConfig } from 'drizzle-orm/pg-core'
import type { Pool, PoolConfig } from 'pg'

Expand All @@ -30,6 +31,8 @@ export type Args = {
*/
disableCreateDatabase?: boolean
extensions?: string[]
/** Generated schema from payload generate:db-schema file path */
generateSchemaOutputFile?: string
idType?: 'serial' | 'uuid'
localesSuffix?: string
logger?: DrizzleConfig['logger']
Expand All @@ -52,7 +55,17 @@ export type Args = {
versionsSuffix?: string
}

export interface GeneratedDatabaseSchema {
schemaUntyped: Record<string, unknown>
}

type ResolveSchemaType<T> = 'schema' extends keyof T
? T['schema']
: GeneratedDatabaseSchema['schemaUntyped']

type Drizzle = NodePgDatabase<ResolveSchemaType<GeneratedDatabaseSchema>>
export type PostgresAdapter = {
drizzle: Drizzle
pool: Pool
poolOptions: PoolConfig
} & BasePostgresAdapter
Expand All @@ -65,7 +78,7 @@ declare module 'payload' {

beforeSchemaInit: PostgresSchemaHook[]
beginTransaction: (options?: PgTransactionConfig) => Promise<null | number | string>
drizzle: PostgresDB
drizzle: Drizzle
enums: Record<string, GenericEnum>
extensions: Record<string, boolean>
/**
Expand Down
Loading

0 comments on commit 23f1ed4

Please sign in to comment.