Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mvp lint #41

Merged
merged 6 commits into from
May 13, 2024
Merged

Mvp lint #41

merged 6 commits into from
May 13, 2024

Conversation

kaaveland
Copy link
Owner

First stab at #37, expecting to add a frontend/middle layer for this later this week. This recognizes:

  • SET [LOCAL] lock_timeout = ...
  • ALTER TABLE ... -- but doesn't recognize most variants yet
    • Setting new type
    • Setting to not null
  • CREATE INDEX [CONCURRENTLY]
  • CREATE TABLE
  • CREATE TABLE AS

It gives us the tools to build the MVP lints:

  • Locking a relation visible to other transactions without lock timeout
  • Creating an index nonconcurrently
    • Although only directly, implicit indexes from adding primary key, unique constraint etc not picked up yet
  • Setting a column to not null

kaaveland
kaaveland commented May 12, 2024
View reviewed changes
src/lints/ast.rs Outdated Show resolved Hide resolved
@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-12T21:46:21.127773070+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-12T22:15:42.840546789+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-12T22:35:24.091672950+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-12T22:37:26.105435052+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland
Add eugene lint and lints for most existing hints
064a0ea 
- Setting column type to json
- Altering column type
- Exclusion constraints
- VALID constraints
- Set to not null
- Implicit unique index

This also refactors `src/hints.rs` to extract `hint_data.rs` and share
common facts between hints & lints.
@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-13T21:40:35.222857935+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-13T21:41:04.101845150+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

@kaaveland kaaveland merged commit 6196a66 into main May 13, 2024
10 checks passed
@kaaveland kaaveland deleted the mvp-lint branch May 13, 2024 21:54
@kaaveland
Copy link
Owner Author

Eugene 🔒 trace report of add_authors.sql

This is a human readable lock tracing and migration report generated by eugene to assist you in writing safer database migration scripts.

Here are some tips for reading it:

  • A lock is called dangerous ❌ if it will cause concurrent queries to wait for the migration to complete
  • You read that right, once a lock is acquired, it is only released at the end of the script
  • Eugene will tell you what kinds of queries dangerous locks would block in a summary
  • Hints can sometimes help you avoid dangerous locks, or hold them for a shorter time
  • It is hard to avoid dangerous locks, but we should minimize time spent while holding them
  • Sometimes seemingly fast migration scripts cause long outages because of lock queues, here is an example scenario

There is a summary section for the entire script at the start of the report and then a section for each statement in the script, that goes over the state of the database at the time the script was executed, as well as effects or hints specific to that particular statement

Overall Summary

Started at Total duration (ms) Number of dangerous locks
2024-05-13T21:54:57.804702451+00:00 4 2 ❌

All locks found

Schema Object Mode Relkind OID Safe Duration held (ms)
public books AccessExclusiveLock Table 16416 ❌ 1
public books AccessShareLock Table 16416 ✅ 1
public books ShareRowExclusiveLock Table 16416 ❌ 1
public books_pkey AccessShareLock Index 16422 ✅ 1

Dangerous locks found

  • AccessExclusiveLock would block the following operations on public.books:
    • SELECT
    • FOR UPDATE
    • FOR NO KEY UPDATE
    • FOR SHARE
    • FOR KEY SHARE
    • UPDATE
    • DELETE
    • INSERT
    • MERGE
  • ShareRowExclusiveLock would block the following operations on public.books:
    • UPDATE
    • DELETE
    • INSERT
    • MERGE

Statement number 1 for 3 ms

SQL

create table authors(id serial primary key, name text not null)

Locks at start

No locks held at the start of this statement.

New locks taken

No new locks taken by this statement.

Statement number 2 for 0 ms

SQL

alter table books alter column title set not null

Locks at start

No locks held at the start of this statement.

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

Hints

Validating table with a new NOT NULL column

ID: E2

A column was changed from NULL to NOT NULL. This blocks all table access until all rows are validated. A safer way is: Add a CHECK constraint as NOT VALID, validate it later, then make the column NOT NULL.

The column title in the table public.books was changed to NOT NULL. If there is a CHECK (title IS NOT NULL) constraint on public.books, this is safe. Splitting this kind of change into 3 steps can make it safe:

  1. Add a CHECK (title IS NOT NULL) NOT VALID; constraint on public.books.
  2. Validate the constraint in a later transaction, with ALTER TABLE public.books VALIDATE CONSTRAINT ....
  3. Make the column NOT NULL

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took AccessExclusiveLock on the Table public.books without a timeout. It blocks SELECT, FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE, UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 3 for 0 ms

SQL

alter table books add column author_id integer not null

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Statement number 4 for 1 ms

SQL

alter table books add foreign key (author_id) references authors(id)

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌

New locks taken

Schema Object Mode Relkind OID Safe
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

Hints

Validating table with a new constraint

ID: E1

A new constraint was added and it is already VALID. This blocks all table access until all rows are validated. A safer way is: Add the constraint as NOT VALID and validate it with ALTER TABLE ... VALIDATE CONSTRAINT later.

A new constraint books_author_id_fkey of type FOREIGN KEY was added to the table public.books as VALID. Constraints that are NOT VALID can be made VALID by ALTER TABLE public.books VALIDATE CONSTRAINT books_author_id_fkey which takes a lesser lock.

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Taking dangerous lock without timeout

ID: E9

A lock that would block many common operations was taken without a timeout. This can block all other operations on the table indefinitely if any other transaction holds a conflicting lock while idle in transaction or active. A safer way is: Run SET LOCAL lock_timeout = '2s'; before the statement and retry the migration if necessary.

The statement took ShareRowExclusiveLock on the Table public.books without a timeout. It blocks UPDATE, DELETE, INSERT, MERGE while waiting to acquire the lock.

Statement number 5 for 0 ms

SQL

select * from books

Locks at start

Schema Object Mode Relkind OID Safe
public books AccessExclusiveLock Table 16416 ❌
public books AccessShareLock Table 16416 ✅
public books ShareRowExclusiveLock Table 16416 ❌
public books_pkey AccessShareLock Index 16422 ✅

New locks taken

No new locks taken by this statement.

Hints

Running more statements after taking AccessExclusiveLock

ID: E4

A transaction that holds an AccessExclusiveLock started a new statement. This blocks all access to the table for the duration of this statement. A safer way is: Run this statement in a new transaction.

The statement is running while holding an AccessExclusiveLock on the Table public.books, blocking all other transactions from accessing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kaaveland