This gem allows to observe database locks generated by a Rails application. By default, locks data is not persisted anywhere in the PostgreSQL logs, so the only way to monitor it is via analyzing the transient state of the pg_locks
metadata table. pg-locks-monitor
is a simple tool that makes this process quick to implement and adjust to each app's individual requirements.
You can read this blogpost for more detailed info on database locks in the Rails apps.
Shameless plug: pg-locks-monitor is just one of the tools that I use when conducting Rails performance audits. Check out my offer if you need help with optimizing your application.
PgLocksMonitor
class provides a snapshot!
method, which notifies selected channels about database locks that match configured criteria.
Start by adding the gem to your Gemfile:
gem "pg-locks-monitor"
Then run bundle install
and rake pg_locks_monitor:init
. It creates a default configuration file with the following settings:
config/initializers/pg_locks_monitor.rb
PgLocksMonitor.configure do |config|
config.locks_limit = 5
config.monitor_locks = true
config.locks_min_duration_ms = 200
config.monitor_blocking = true
config.blocking_min_duration_ms = 100
config.notify_logs = true
config.notify_slack = false
config.slack_webhook_url = ""
config.slack_channel = ""
config.notifier_class = PgLocksMonitor::DefaultNotifier
config.locks_filter_proc = ->(lock) { true }
config.blocking_filter_proc = ->(lock) { true }
end
locks_limit
- specify the max number of locks to report in a single notificationmonitor_locks
- observe database locks even if they don't conflict with a different SQL querylocks_min_duration_ms
- notify about locks that execeed this duration threshold in millisecondsmonitor_blocking
- observe database locks which cause other SQL query to wait from them to releaseblocking_min_duration_ms
- notify about blocking locks that execeed this duration threshold in millisecondsnotify_logs
- send notifications about detected locks usingRails.logger.info
methodnotify_slack
- send notifications about detected locks to the configured Slack channelslack_webhook_url
- webhook necessary for Slack notification to workslack_channel
- the name of the target Slack channelnotifier_class
- customizable notifier classlocks_filter_proc
- configurable filter to exclude locks based on any custom logicblocking_filter_proc
- configurable filter to exclude blocking locks based on any custom logic
Before enabling a recurring invocation of the snapshot!
method, it's recommended to first manually trigger the notification to test the configured channels.
You can generate an "artificial" blocking lock and observe it by running the following code in the Rails console:
user = User.last
Thread.new do
User.transaction do
user.update(email: "email-#{SecureRandom.hex(2)}@example.com")
sleep 5
raise ActiveRecord::Rollback
end
end
Thread.new do
User.transaction do
user.update(email: "email-#{SecureRandom.hex(2)}@example.com")
sleep 5
raise ActiveRecord::Rollback
end
end
sleep 0.5
PgLocksMonitor.snapshot!
Please remember to adjust the update operation to match your app's schema.
As a result of running the above snippet, you should receive a notification about the acquired blocking database lock.
Received notifications contain data helpful in debugging the cause of long lasting-locks.
And here's a sample blocking lock notification:
[
{
# PID of the process which was blocking another query
"blocking_pid": 29,
# SQL query blocking other SQL query
"blocking_statement": "UPDATE \"users\" SET \"updated_at\" = $1 WHERE \"users\".\"id\" = $2 from/sidekiq_job:UserUpdater/",
# the duration of blocking SQL query
"blocking_duration": "PT0.972116S",
# app that triggered the blocking SQL query
"blocking_sql_app": "bin/sidekiq",
# PID of the process which was blocked by another query
"blocked_pid": 30,
# SQL query blocked by other SQL query
"blocked_statement": "UPDATE \"users\" SET \"last_active_at\" = $1 WHERE \"users\".\"id\" = $2 from/controller_with_namespace:UsersController,action:update/",
# the duration of the blocked SQL query
"blocked_duration": "PT0.483309S",
# app that triggered the blocked SQL query
"blocked_sql_app": "bin/puma"
}
]
This sample blocking notification shows than a query originating from the UserUpdater
Sidekiq job is blocking an update operation on the same user for the UsersController#update
action. Remember to configure the marginalia gem to enable these helpful query source annotations.
Here's a sample lock notification:
[
{
# PID of the process which acquired the lock
"pid": 50,
# name of affected table/index
"relname": "users",
# ID of the source transaction
"transactionid": null,
# bool indicating if the lock is already granted
"granted": true,
# mode of the acquired lock
"mode": "RowExclusiveLock",
# SQL query which acquired the lock
"query_snippet": "UPDATE \"users\" SET \"updated_at\" = $1 WHERE \"users\".\"id\" = $2 from/sidekiq_job:UserUpdater/",
# age of the lock
"age": "PT0.94945S",
# app that acquired the lock
"application": "bin/sidekiq",
# ID of the database where lock was acquired
"database": "84759327",
# type of the acquired lock
"locktype": "relation",
},
This gem is intended to be used via a recurring background job, but it is agnostic to the background job provider. Here's a sample Sidekiq implementation:
app/jobs/pg_locks_monitor_job.rb
require 'pg-locks-monitor'
class PgLocksMonitoringJob
include Sidekiq::Worker
sidekiq_options retry: false
def perform
PgLocksMonitor.snapshot!
ensure
if ENV["PG_LOCKS_MONITOR_ENABLED"]
PgLocksMonitoringJob.perform_in(1.minute)
end
end
end
Remember to schedule this job when your app starts:
config/pg_locks_monitor.rb
#...
if ENV["PG_LOCKS_MONITOR_ENABLED"]
PgLocksMonitoringJob.perform
end
A background job that schedules itself is not the cleanest pattern. So alternatively you can use sidekiq-cron, whenever or clockwork gems to trigger the PgLocksMonitor.snapshot!
invocation periodically.
A recommended frequency of invocation depends on your app's traffic. From my experience, even 1 minute apart snapshots can provide a lot of valuable data, but it all depends on how often the locks are occurring in your Rails application.
You can modify locks_filter_proc
and blocking_filter_proc
to exclude locks from getting reported. For example, here's how you can report only locks that originated from the Puma server process:
config/initializers/pg_locks_monitor.rb
PgLocksMonitor.configure do |config|
# ...
config.locks_filter_proc = -> (lock) {
lock.fetch("application").downcase.include?("puma")
}
end
or exclude blocking locks which are affecting only the Sidekiq process:
config/initializers/pg_locks_monitor.rb
PgLocksMonitor.configure do |config|
# ...
config.blocking_filter_proc = -> (lock) {
!lock.fetch("blocked_sql_app").downcase.include?("sidekiq")
}
end
Please beware that configuring these procs does not overwrite the min duration settings, i.e., locks_min_duration_ms
and blocking_min_duration_ms
.
PgLocksMonitor::DefaultNotifier
supports sending lock notifications with Rails.logger
or to a Slack channel. If you want to use different notification channels you can define your custom notifier like that:
config/initializers/pg_locks_monitor.rb
class PgLocksEmailNotifier
def self.call(locks_data)
LocksMailer.with(locks_data: locks_data).notification.deliver_now
end
end
PgLocksMonitor.configure do |config|
# ...
config.notifier_class = PgLocksEmailNotifier
end