request_migrations

Make breaking API changes without breaking things! Use request_migrations to craft backwards-compatible migrations for API requests, responses, and more. Read the blog post.

This gem was extracted from Keygen and is being used in production to serve millions of API requests per day.

Sponsored by:

A fair source software licensing and distribution API.

Links:

• Installing request_migrations
• Supported Ruby versions
• RubyDoc
• Usage
  • Response migrations
  • Request migrations
  • Data migrations
  • Routing constraints
  • Configuration
  • Version formats
• Testing
• Tips and tricks
• Examples
• Credits
• Contributing
• License

Installation

Add this line to your application's Gemfile:

gem 'request_migrations'

And then execute:

$ bundle

Or install it yourself as:

$ gem install request_migrations

Supported Rubies

request_migrations supports Ruby 3.1 and above. We encourage you to upgrade if you're on an older version. Ruby 3 provides a lot of great features, like better pattern matching and a new shorthand hash syntax.

Documentation

You can find the documentation on RubyDoc.

We're working on improving the docs.

Features

• Define migrations for migrating a response between versions.
• Define migrations for migrating a request between versions.
• Define migrations for applying data migrations.
• Define version-based routing constraints.
• It's fast.

Usage

Use request_migrations to make backwards-incompatible changes in your code, while providing a backwards-compatible interface for clients on older API versions. What exactly does that mean? Well, let's demonstrate!

Let's assume that we provide an API service, which has /users CRUD resources.

Let's also assume we start with the following User model:

class User
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :name, :string
end

After awhile, we realize our User model's combined name attribute is not working too well, and we want to change it to first_name and last_name.

So we write a database migration that changes our User model:

class User
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :first_name, :string
  attribute :last_name, :string
end

But what about the API consumers who were relying on name? We just broke our API contract with them! To resolve this, let's create our first request migration.

We recommend that migrations be stored under app/migrations/.

class CombineNamesForUserMigration < RequestMigrations::Migration
  # Provide a useful description of the change
  description %(transforms a user's first and last name to a combined name attribute)

  # Migrate inputs that contain a user. The migration should mutate
  # the input, whatever that may be.
  migrate if: -> data { data in type: 'user' } do |data|
    first_name = data.delete(:first_name)
    last_name  = data.delete(:last_name)

    data[:name] = "#{first_name} #{last_name}"
  end

  # Migrate the response. This is where you provide the migration input.
  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users',
                                                                 action: 'show' } do |res|
    data = JSON.parse(res.body, symbolize_names: true)

    # Call our migrate definition above
    migrate!(data)

    res.body = JSON.generate(data)
  end
end

As you can see, with pattern matching, it makes creating migrations for certain resources simple. Here, we've defined a migration that only runs for the users#show resource, and only when the response is successful. In addition, the data is only migrated when the response body contains a user.

Next, we'll need to configure request_migrations via an initializer under initializers/request_migrations.rb:

RequestMigrations.configure do |config|
  # Define a resolver to determine the target version. Here, you can perform
  # a lookup on the current user using request parameters, or simply use
  # a header like we are here, defaulting to the latest version.
  config.request_version_resolver = -> request {
    request.headers.fetch('Foo-Version') { config.current_version }
  }

  # Define the latest version of our application.
  config.current_version = '1.1'

  # Define previous versions and their migrations, in descending order.
  config.versions = {
    '1.0' => %i[combine_names_for_user_migration],
  }
end

Lastly, you'll want to update your application controller so that migrations are applied:

class ApplicationController < ActionController::API
  include RequestMigrations::Controller::Migrations

  # Optionally rescue from requests for unsupported versions
  rescue_from RequestMigrations::UnsupportedVersionError, with: -> {
    render(
      json: { error: 'unsupported API version requested', code: 'INVALID_API_VERSION' },
      status: :bad_request,
    )
  }
end

Now, when an API client provides a Foo-Version: 1.0 header, they'll receive a response containing the combined name attribute.

Response migrations

We covered this above, but response migrations define a change to a response. You define a response migration by using the response class method.

class RemoveVowelsMigration < RequestMigrations::Migration
  description %(in the past, we had a bug that removed all vowels, and some clients rely on that behavior)

  response if: -> res { res.request.params in action: 'index' | 'show' | 'create' | 'update' } do |res|
    body = JSON.parse(res.body, symbolize_names: true)

    # Mutate the response body by removing all vowels
    body.deep_transform_values! { _1.gsub(/[aeiou]/, '') }

    res.body = JSON.generate(body)
  end
end

The response method accepts an :if keyword, which should be a lambda that evaluates to a boolean, which determines whether or not the migration should be applied. An ActionDispatch::Response will be yielded, the current response (calls controller#response).

The gem makes no assumption on a response's content type or what the migration will do. You could, for example, migrate the response body, or mutate the headers, or even change the response's status code.

The response method can be used multiple times per-migration.

Request migrations

Request migrations define a change on a request. For example, modifying a request's headers. You define a response migration by using the request class method.

class AssumeContentTypeMigration < RequestMigrations::Migration
  description %(in the past, we assumed all requests were JSON, but that has since changed)

  # Migrate the request, adding an assumed content type to all requests.
  request do |req|
    req.headers['Content-Type'] = 'application/json'
  end
end

The request method accepts an :if keyword, which should be a lambda that evaluates to a boolean, which determines whether or not the migration should be applied. An ActionDispatch::Request object will be yielded, the current request (calls controller#request).

Again, like with response migrations, the gem makes no assumption on what a migration does. A migration could mutate a request's params, or mutate headers. It's up to you, all it does is provide the request.

Request migrations should avoid using the migrate method.

The request method can be used multiple times.

Data migrations

In our first scenario, where we combined our user's name attributes, we defined our migration using the migrate class method. At this point, you may be wondering why we did that, since we didn't use that method for the 2 previous request and response migrations above.

Well, it comes down to support for data migrations (as well as offering a nice interface for pattern matching inputs). Let's go back to our first example, CombineNamesForUserMigration.

class CombineNamesForUserMigration < RequestMigrations::Migration
  # Provide a useful description of the change
  description %(transforms a user's first and last name to a combined name attribute)

  # Migrate inputs that contain a user. The migration should mutate
  # the input, whatever that may be.
  migrate if: -> data { data in type: 'user' } do |data|
    first_name = data.delete(:first_name)
    last_name  = data.delete(:last_name)

    data[:name] = "#{first_name} #{last_name}"
  end

  # Migrate the response. This is where you provide the migration input.
  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users' | 'api/v1/me',
                                                                 action: 'show' } do |res|
    data = JSON.parse(res.body, symbolize_names: true)

    # Call our migrate definition above
    migrate!(data)

    res.body = JSON.generate(data)
  end
end

What if we had a webhook system that we also needed to apply these migrations to? Well, we can use a data migration here, via the Migrator class:

class WebhookWorker
  def perform(event, endpoint, data)
    # ...

    # Migrate event data from latest version to endpoint's configured version
    current_version = RequestMigrations.config.current_version
    target_version  = endpoint.api_version
    migrator        = RequestMigrations::Migrator.new(
      from: current_version,
      to: target_version,
    )

    # Migrate the event data (tries to apply all matching migrations)
    migrator.migrate!(data:)

    # ...

    event.send!(data)
  end
end

This will apply the block defined in migrate onto our data. With that, we've successfully applied a migration to both our API responses, as well as to the webhook events we send. In this case, if our event data matches our expected data shape, e.g. type: 'user', then the migration will be applied.

In addition to data migrations, this allows for easier testing.

The migrate method can be used multiple times per-migration to e.g. match and migrate on different shapes of data. For a JSON:API app, for example, you could migrate on data: [*] and includes: [*].

Routing constraints

When you want to encourage API clients to upgrade, you can utilize a routing version_constraint to define routes only available for certain versions.

You can also utilize routing constraints to remove an API endpoint entirely.

Rails.application.routes.draw do
  # This endpoint is only available for version 1.1 and above
  version_constraint '>= 1.1' do
    resources :some_shiny_new_resource
  end

  # Remove this endpoint for any version below 1.1
  version_constraint '< 1.1' do
    scope module: :v1x0 do
      resources :a_deprecated_resource
    end
  end
end

Currently, routing constraints only work for the :semver version format. (PRs welcome!)

Configuration

RequestMigrations.configure do |config|
  # Define a resolver to determine the target version. Here, you can perform
  # a lookup on the current user using request parameters, or simply use
  # a header like we are here, defaulting to the latest version.
  config.request_version_resolver = -> request {
    request.headers.fetch('Foo-Version') { config.current_version }
  }

  # Define the accepted version format. Default is :semver.
  config.version_format = :semver

  # Define the latest version of our application.
  config.current_version = '1.2'

  # Define previous versions and their migrations, in descending order.
  # Should be a hash, where the key is the version and the value is an
  # array of migration symbols or classes.
  config.versions = {
    '1.1' => %i[
      has_one_author_to_has_many_for_posts_migration
      has_one_author_to_has_many_for_post_migration
    ],
    '1.0' => %i[
      combine_names_for_users_migration
      combine_names_for_user_migration
    ],
  }

  # Use a custom logger. Supports ActiveSupport::TaggedLogging.
  config.logger = Rails.logger
end

Version formats

By default, request_migrations uses a :semver version format, but it can be configured to instead use one of the following, set via config.version_format=.

Format
:semver	Use semantic versions, e.g. 1.0, 1.1, and 2.0.
:date	Use date versions, e.g. 2020-09-02, 2021-01-01.
:integer	Use integer versions, e.g. 1, 2, and 3.
:float	Use float versions, e.g. 1.0, 1.1, and 2.0.
:string	Use string versions, e.g. a, b, and z.

All versions will be sorted according to the format's type.

Testing

Using data migrations allows for easier testing of migrations. For example, using Rspec:

describe CombineNamesForUserMigration do
  before do
    RequestMigrations.configure do |config|
      config.current_version = '1.1'
      config.versions        = {
        '1.0' => [CombineNamesForUserMigration],
      }
    end
  end

  it 'should migrate user name attributes' do
    migrator = RequestMigrations::Migrator.new(from: '1.1', to: '1.0')
    data     = serialize(
      create(:user, first_name: 'John', last_name: 'Doe'),
    )

    expect(data).to include(type: 'user', first_name: 'John', last_name: 'Doe')
    expect(data).to_not include(name: anything)

    migrator.migrate!(data:)

    expect(data).to include(type: 'user', name: 'John Doe')
    expect(data).to_not include(first_name: 'John', last_name: 'Doe')
  end
end

To avoid polluting the global configuration, you can use RequestMigrations::Testing within your application's spec/rails_helper.rb, or a similar spec helper:

require 'request_migrations/testing'

Rspec.configure do |config|
  config.before :each do
    RequestMigrations::Testing.setup!
  end

  config.after :each do
    RequestMigrations::Testing.teardown!
  end
end

This will setup a new test configuration, and then restore the previous global configuration after each spec.

Tips and tricks

Over the years, we're learned a thing or two about versioning an API. We'll share tips here.

Use pattern matching

Pattern matching really cleans up the :if conditions, and overall makes migrations more readable.

class AddUsernameAttributeToUsersMigration < RequestMigrations::Migration
  description %(adds username attributes to a collection of users)

  migrate if: -> body { body in data: [*] } do |body|
    case body
    in data: [*, { type: 'users', attributes: { ** } }, *]
      body[:data].each do |user|
        case user
        in type: 'users', attributes: { email: }
          user[:attributes][:username] = email
        else
        end
      end
    else
    end
  end

  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users',
                                                                 action: 'index' } do |res|
    body = JSON.parse(res.body, symbolize_names: true)

    migrate!(body)

    res.body = JSON.generate(body)
  end
end

Just be sure to remember your else block when case pattern matching. :)

Route helpers

If you need to use route helpers in a migration, include them in your migration:

class SomeMigration < RequestMigrations::Migration
  include Rails.application.routes.url_helpers
end

Separate by shape

Define separate migrations for different input shapes, e.g. define a migration for an #index to migrate an array of objects, and define another migration that handles the singular object from #show, #create and #update. This will help keep your migrations readable.

For example, for a singular user response:

class CombineNamesForUserMigration < RequestMigrations::Migration
  description %(transforms a user's first and last name to a combined name attribute)

  migrate if: -> data { data in type: 'user' } do |data|
    first_name = data.delete(:first_name)
    last_name  = data.delete(:last_name)

    data[:name] = "#{first_name} #{last_name}"
  end

  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users',
                                                                 action: 'show' } do |res|
    data = JSON.parse(res.body, symbolize_names: true)

    migrate!(data)

    res.body = JSON.generate(data)
  end
end

And for a response containing a collection of users:

class CombineNamesForUsersMigration < RequestMigrations::Migration
  description %(transforms a collection of users' first and last names to a combined name attribute)

  migrate if: -> data { data in [*, { type: 'user' }, *] do |data|
    data.each do |record|
      case record
      in type: 'user', first_name:, last_name:
        record[:name] = "#{first_name} #{last_name}"

        record.delete(:first_name)
        record.delete(:last_name)
      else
      end
    end
  end

  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users',
                                                                 action: 'index' } do |res|
    data = JSON.parse(res.body, symbolize_names: true)

    migrate!(data)

    res.body = JSON.generate(data)
  end
end

Note that the migrate method now migrates an array input, and matches on the #index route.

Always check response status

Always check a response's status. You don't want to unintentionally apply migrations to error responses.

class SomeMigration < RequestMigrations::Migration
  response if: -> res { res.successful? } do |res|
    # ...
  end
end

Also mind 204 No Content, since the response body will be nil.

Don't match on URL pattern

Don't match on URL pattern. Instead, use response.request.params to access the request params in a response migration, and use the :controller and :action params to determine route.

class SomeMigration < RequestMigrations::Migration
  # Bad
  response if: -> res { res.request.path.matches?(/^\/v1\/posts$/) }

  # Good
  response if: -> res { res.request.params in controller: 'api/v1/posts', action: 'index' }
end

Namespace deprecated controllers

When you need to entirely change a controller or service class, use a V1x0::UsersController-style namespace to keep the old deprecated classes tidy.

class V1x0::UsersController
  def foo
    # Some old foo action
  end
end

Avoid migrate for request migrations

Avoid using migrate for request migrations. If you do, then data migrations, e.g. for webhooks, will attempt to apply the request migrations. This may erroneously produce bad output, or even undo a response migration. Instead, keep all request migration logic, e.g. transforming params, inside of the request block.

class SomeMigration < RequestMigrations::Migration
  # Bad (side-effects for data migrations)
  migrate do |params|
    params[:foo] = params.delete(:bar)
  end

  request do |req|
    migrate!(req.params)
  end

  # Good
  request do |req|
    req.params[:foo] = req.params.delete(:bar)
  end
end

Avoid routing contraints

Avoid using routing version constraints that remove functionality. They can be a headache during upgrades. Consider only making additive changes. Instead, consider removing or hiding the documentation for old or deprecated endpoints, to limit any new usage.

Rails.application.routes.draw do
  resources :users do
    # Iffy
    version_constraint '< 1.1' do
      resources :posts
    end

    # Good
    scope module: :v1x0 do
      resources :posts
    end
  end
end

Avoid n+1s

Avoid introducing n+1 queries in your migrations. Try to utilize the current data you have to perform more meaningful queries, returning only the data needed for the migration.

class AddRecentPostToUsersMigration < RequestMigrations::Migration
  description %(adds :recent_post association to a collection of users)

  # Bad (n+1)
  migrate if: -> data { data in [*, { type: 'user' }, *] do |data|
    data.each do |record|
      case record
      in type: 'user', id:
        recent_post = Post.reorder(created_at: :desc)
                          .find_by(user_id: id)

        record[:recent_post] = recent_post&.id
      else
      end
    end
  end

  # Good
  migrate if: -> data { data in [*, { type: 'user' }, *] do |data|
    user_ids = data.collect { _1[:id] }
    post_ids = Post.select(:id, :user_id)
                   .distinct_on(:user_id)
                   .where(user_id: user_ids)
                   .reorder(created_at: :desc)
                   .group_by(&:user_id)

    data.each do |record|
      case record
      in type: 'user', id: user_id
        record[:recent_post] = post_ids[user_id]&.id
      else
      end
    end
  end

  response if: -> res { res.successful? && res.request.params in controller: 'api/v1/users',
                                                                 action: 'index' } do |res|
    data = JSON.parse(res.body, symbolize_names: true)

    migrate!(data)

    res.body = JSON.generate(data)
  end
end

Instead of potentially tens or hundreds of queries, we make a single purposeful query to get the data we need in order to complete the migration.

---

Have a tip of your own? Open a pull request!

Examples

Below are some real-world examples of request migrations:

• Migrations: https://github.com/keygen-sh/keygen-api/tree/master/app/migrations
• Tests: https://github.com/keygen-sh/keygen-api/tree/master/spec/migrations

Is it any good?

Yes.

Credits

Credit goes to Stripe for inspiring the high-level migration strategy. Intercom has another good post on the topic.

Contributing

If you have an idea, or have discovered a bug, please open an issue or create a pull request.

License

The gem is available as open source under the terms of the MIT License.