extending_sprockets.md

Extending Sprockets

Sprockets can use custom processors, compressors, and directives. This document is intended for library authors who want to extend Sprockets functionality.

Contents

• Types of Extensions
  • Processors
  • Transformers
  • Compressors
  • Exporters
• Extension Interface
  • Extension Keys
  • Extension Metadata Keys
  • Register Mime Type
• Adding Directives to your Extension
• Adding ERB Support to your Extension
• Supporting All Versions of Sprockets in Processors
  • Registering All Versions of Sprockets in Processors

Types of Extensions

Sprockets supports a few different ways to extend functionality.

• processors
• transformers
• compressors
• exporters

For a detailed explanation of each see the respective sections below.

Sprockets ships with a number of built in processors, transformers and compressors. You can see all the defaults registered in lib/sprockets.rb.

Processors

There are two types of processors: preprocessors and postprocessors.

A preprocessor is called as an asset is loaded. Generally preprocessors take in a raw file and convert its contents. For example the DirectiveProcessor is the processor that is responsible for reading in the Sprockets directives such as:

//= require "foo"

It will then load in the foo file and add its contents to the original file.

A postprocessor is called after all of the transformers run. Generally postprocessors do a final transformation to an asset before it is ready for bundling in the asset pipeline.

For example, imagine that you like to write JavaScript without semicolons. However, your project uses a CoffeeScript library. Since CoffeeScript adds semicolons during compilation, you might make a processor that removes all ; from JavaScript files since CoffeeScript compiles to JavaScript. A simple implementation of this could be a proc:

remove_semicolons_processor = -> (input) {
  data = input[:data].gsub(";", "")
  { data: data }
}

When you register this processor as a postprocessor your CoffeeScript library will first be compiled to JavaScript, then post-processed by this processor, thus removing all semicolons from the output.

Without postprocessors, you would have to ensure that the CoffeeScript transformer is run before your processor so you would have to insert it after the CoffeeScript transformer much like two Rack middlewares that have an order-dependence on one another. With postprocessors, you can register your processor as a postprocessor and not need to worry about ordering it after the CoffeeScript transformer.

Transformers

A transformer takes one asset and converts it into another asset. For example the CoffeeScriptProcessor is what takes a file with a .coffee file extension and returns a .js file.

Transformers

Like a preprocessor, a transformer will return the contents of the file in a :data key, and any other metadata.

You can register a transformer like this:

Sprockets.register_transformer 'text/coffeescript', 'application/javascript', CoffeeScriptProcessor

The first argument is the mime type of the file that the processor accepts. The second argument is the mime type that it generates, and the last argument is the object that responds to call.

For example, say you wanted to have Sprockets process a .html file, and output the mime type as application/javascript. To accomplish this, you would do the following:

Sprockets.register_transformer 'text/html', 'application/javascript', MyTemplateProcessor

Compressors

A compressor takes in an asset, and returns a smaller version of that asset. For example the uglifier compressor takes in a JavaScript file, it then removes the whitespace and applies other space saving techniques and returns a smaller JavaScript source.

Compressors must respond to call and return a :data key in a hash similar to a processor.

You can register a compressor like this:

Sprockets.register_compressor 'application/javascript', :uglify, UglifierCompressor

Registering a compressor allows it to be used later. After registering a compressor you can activate it using the js_compressor= or css_compressor= method on the environment or Sprockets global.

Sprockets.register_compressor 'text/css', :my_css, MyCssCompressor
Sprockets.css_compressor = :my_css

Compressors only operate on JavaScript and CSS. If you want to compress a different type of asset, use a processor (see "Processors" above) to process the asset.

Exporters

An exporter takes a compiled asset and writes it to disk. The default exporters are FileExporter which writes an asset's compiled source to disk and ZlibExporter that will produce a .gz file extension.

You can write your own exporter:

register_exporter '*/*', Sprockets::Exporters::ZlibExporter

First argument is the mime type of files that the exporter will operate on. For your convenience, a Sprockets::Exporters::Base class is provided for you to inherit from. Details about the required interface for an exporter are in that class.

Your exporter gets initialized once for each asset to be exported by sprockets with the following keyword arguments

• asset (Instance of Sprockets::Asset)
• environment (Instance of Sprockets::Environment)
• directory (Instance of String)

A setup method is called right after the exporter is initialized. Your exporter is expected implement a skip? method. If this method returns true then sprockets will skip your exporter and move to the next one. An instance of Sprockets::Logger is passed into this method that can be used to indicate to the user what is happening. This method is called synchronously.

The work of writing the new asset to disk is performed in the call method. This method is potentially called in a new thread and should not mutate any global state. A write method is provided that takes a filename to be written to (full path) and yields an IO object.

Extension Interface

A processor is expected to respond to call() and it accepts a hash of file contents. It is expected to return a hash that includes a :data key. The value returned in the :data key will be used as the contents for the file. Any other keys returned will be stored in the :metadata hash of an asset. For example, this result:

class HelloWorldProcessor
  def call(input)
    return { data: input[:data] + "\n'hello world'" }
  end
end

Would append the string 'hello world' on a new line to any asset.

For Sprockets to call the processor it must be registered. If we wanted this processor to be called with any JavaScript files we would use the JavaScript mime type application/javascript and pass in the object that responds to call, in this case an instance of the HelloWorldProcessor class:

Sprockets.register_preprocessor('application/javascript', HelloWorldProcessor.new)

To register a processor as a postprocessor instead of a preprocessor, invoke the register_postprocessor command instead:

Sprockets.register_postprocessor('application/javascript', HelloWorldProcessor.new)

Extension Keys

The call interface returns a hash. In different versions of Sprockets there may be different keys. This doc is for Sprockets 4 (master). You can see these values being passed into processors in the Sprockets loader.

• :data - [String] Contains the contents of the file being passed to the your processor.

Example:

'var foo = "bar"'

• :uri - [String] A full URI to the asset, may include custom Sprockets params.

Example:

"file:///Users/richardschneeman/Documents/projects/sprockets/test/fixtures/default/application.coffee?type=application/javascript",

• :filename - [String] Full path to asset on disk.

Example:

"/Users/richardschneeman/Documents/projects/sprockets/test/fixtures/default/gallery.js\"

• :load_path - [String] The load path that was used to find the asset.

Example:

"/Users/richardschneeman/Documents/projects/sprockets/test/fixtures/default"

• :name - [String] The name of the file being loaded without extension.

Example

"gallery"

• :content_type - [String] The corresponding mime content type of the asset.

Example:

"application/javascript"

• :metadata - [Hash] Extra data, see the "metadata" section.

Example:

# See metadata section for more info
{
  dependencies: [].to_set
  map: {
    # ...
  }
}

• :cache - [Sprockets::Cache] A cache object you can use to store and retrieve intermediate objects. You can use Cache#get, Cache#set and Cache#fetch api. Refer to method docs for more info. If using paths for the key or contents, use Sprockets::Environment#compress_from_root and Sprockets::Environment#expand_from_root as the location of of your files absolute path will change.

• :environment [Sprockets::Environment] Now you have direct access to all 105 methods that Sprockets uses! Use carefully, we may consider limiting this in the future. If you have feedback on what methods you need or use please say hi, Open an issue and let the Sprockets team know.

Extension Metadata Keys

While you can store arbitrary keys in the metadata returned by your extension, there are some with special meaning and uses inside of Sprockets. More may be added in the future.

Anything you add to the metadata will be stored in the Sprockets cache for the asset.

• map: This key contains a source map for the asset.

A source map is a way to tell a browser how to map a generated file to an original for example if you write a CoffeeScript file, Sprockets will generate a JavaScript file which is what the browser will see. If you need to debug this javascript file it helps if you know where the in your original CoffeeScript file the generated JavaScript code came from. The source map tells the browser how to map from a generated file to an original.

Sprockets expects this map to follow the source map spec.

• charset: This key contains the mime charset for an asset.

A charset is the encoding for text based assets. If you do not specify a charset then one will be automatically assigned by sprockets based on the encoding type of the contents returned in the :data key. Normally you want that, the only time you don't want that is if you're working with binary data, or data you don't want to be compressed. If Sprockets sees a charset then it will think that the contents of the file are text and can be compressed via GZIP. You can avoid this by setting the field manually.

return { data: data, charset: nil }

WIP the format of the source map may be subject to change before 4.0 is released. Currently it takes a :original and :generated key which each hold an array of line and column numbers. Line numbers are 1 indexed column numbers are 0 indexed. The first character of a file will always be [1,0].

• required: A Set of String Asset URIs that the Bundle processor should concatenate together.

• stubbed: A Set of String Asset URIs that will be omitted from the :required set.

• links: A Set of String Asset URIs that should be compiled along with this asset.

• dependencies: A Set of String Cache URIs that should be monitored for caching.

Register Mime Types

You can add new mime-types to Sprockets by using the register_mime_type method. For example:

Sprockets.register_mime_type 'application/json', extensions: ['.json'], charset: :unicode
Sprockets.register_mime_type 'application/ruby', extensions: ['.rb']

The first method is the mime-type of the object. The :extensions key passed to the second argument contains an array of all the extensions for the given mime type. An optional charset can be registered. This should only be used for text based mimetypes.

Your extension may have multiple parts for example some people use .js.coffee when this file type is used, we do not wish it to process as javascript first and then as coffee script so we register this entire extension:

Sprockets.register_mime_type 'text/coffeescript', extensions: ['.coffee', '.js.coffee']

Adding Directives to your Extension

If you are writing a transformer you likely also want it to understand Sprockets directives (such as the ability to require other files). You don't have to add this functionality in manually, instead you can register a custom instance of DirectiveProcessor to run with your asset.

The DirectiveProcessor can be initialized with a :comments key that holds an array of characters that denote a comment. For example the coffeescript language uses # as a comment. We could apply directive support to coffeescript files by registering:

Sprockets.register_preprocessor 'text/coffeescript', DirectiveProcessor.new(comments: ["#", ["###", "###"]])

Adding ERB Support to your Extension

In Sprockets 4 file types are no longer "chainable" this means that if you wanted to use a .coffee.erb that it must be registered to sprockets explicitly. This is different from previous versions of sprockets where you would have to register only a .erb processor and then a .coffee processor and sprockets would chain them (first running erb then coffee).

The reason for the change is to have more explicit behavior. It helps sprockets know to do the right thing, decreases magic, and increases speed. It also means that as a library maintainer you must tell sprockets all the extensions you want your project to work with. Going with the coffee script example:

Sprockets.register_mime_type('text/coffeescript+ruby', extensions: ['.coffee.erb'])
Sprockets.register_transformer('text/coffeescript+ruby', 'text/coffeescript', ::Sprockets::CoffeeScriptProcessor)

Supporting All Versions of Sprockets in Processors

If you are extending sprockets you may want to support all current major versions of sprockets (2, 3, and 4). The processor interface was deprecated from Sprockets 2 and a legacy shim was put into Sprockets 3. Now that Sprockets 4 is out that shim no longer is active, so you'll need to update your gem to either only use the new interface or use both interfaces. For example:

# Sprockets 2, 3 & 4 interface

class MySprocketsExtension
  def initialize(filename, &block)
    @filename = filename
    @source   = block.call
  end

  def render(context, empty_hash_wtf)
    self.class.run(@filename, @source, context)
  end

  def self.run(filename, source, context)
    source + "/* Hello From my sprockets extension */"
  end

  def self.call(input)
    filename = input[:filename]
    source   = input[:data]
    context  = input[:environment].context_class.new(input)

    result = run(filename, source, context)
    context.metadata.merge(data: result)
  end
end

require 'sprockets/processing'
extend Sprockets::Processing

register_preprocessor 'text/css', MySprocketsExtension

This extension is registered to add a comment /* Hello From my sprockets extension */ to the end of a text/css (.css) file.

To understand why all of this is needed, we need to look at the different interfaces for Sprockets 2, 3, and 4.

Sprockets 2 let you register a processor with a class that would be instantiated and the method render called on it you can view the calling code in Sprockets 2.x. It may have looked like this:

# Sprockets 2.x interface

class MySprocketsExtension
  def initialize(filename, &block)
    # code
  end

  def render(variable, empty_hash_wtf)
    # code
  end
end

require 'sprockets/processing'
extend Sprockets::Processing

register_preprocessor 'text/css', MySprocketsExtension

You can also pass a block to both 2.x and 3.x+ processor interfaces, however the number of args the method takes has changed so it's very hard to do that method and support multiple processor interfaces.

This Sprockets 2.x render interface is deprecated, instead you should use the call interface which was introduced in Sprockets 3.x. Whatever you pass to the processor can have a call method, it does not need to be a class.

# Sprockets 3.x+ interface

module MySprocketsExtension
  def self.call(input)
    # code
    result = input[:data] # :data key holds source
    { data: result }
  end
end

require 'sprockets/processing'
extend Sprockets::Processing

register_preprocessor 'text/css', MySprocketsExtension

So if you want 2, 3, and 4 to work you can pass in a class that also has a call method on it as well as a render instance method. To see how this can be done you can reference this autoprefixer-rails pull request.

This way you're passing in an object that responds to 3.x's call interface and 2.x's new.render interface. In generally we're recommending people not use Sprockets 2.x and that they upgrade to Sprockets 3+. If it's easier on you, you can rev a major version and only support the new interface.

There are new hash keys introduced in Sprockets 4. The :source_path key contains the file that will hold the sourcemap of the current asset.

# Sprockets 3.x+ interface

module MySprocketsExtension
  def self.call(input)
    file_where_source_map_will_end_up = input[:source_path]
    # code
    { data: result }
  end
end

If your application is making serious modifications to the source file (an example could be a coffee script file generating JS will be significantly different) then you'll want to calculate and return an appropriate map key in the metadata hash. See the "metadata" section for more info on doing this.

Once you've written your processor to run on all 3 versions of Sprockets you will need to register it. This is covered next.

Registering All Versions of Sprockets in Processors

In Sprockets 2 and 3 the way you registered a processor was via register_engine. Unfortunately they have different method signatures. In Sprockets 4 you must first explicitly register a mime type and use the appropriate processor directive i.e. register_transformer, register_preprocessor, etc. To register a processor for all 3 versions of sprockets you could do it like this:

# Sprockets 2, 3, and 4

if env.respond_to?(:register_transformer)
  env.register_mime_type 'text/css', extensions: ['.css'], charset: :css
  env.register_preprocessor 'text/css', MySprocketsExtension
elsif env.respond_to?(:register_engine)
  args = ['.css', MySprocketsExtension]
  args << { mime_type: 'text/css', silence_deprecation: true } if Sprockets::VERSION.start_with?("3")
  env.register_engine(*args)
end

To understand why this is all needed, we can break down the parts. First is how you register an "engine" with Sprockets 3:

# Sprockets 3
env.register_engine '.css', MySprocketsExtension, mime_type: 'text/css', silence_deprecation: true

The use of register_engine is deprecated in Sprockets 3 and you will get a deprecation warning about it's use. We can pass in silence_deprecation: true to let Sprockets know that the interface is going away, only do this on code you know works with Sprockets 4.

To get the register_engine code working with Sprockets 2 we have to do some version detection since Sprockets 2 will error if you try to pass a hash into it:

# Sprockets 2 & 3
args = ['.css', MySprocketsExtension]
args << { mime_type: 'text/css', silence_deprecation: true } if Sprockets::VERSION.start_with?("3")
env.register_engine(*args)

Next we see how to do it with Sprockets 4 when register_engine is removed:

# Sprockets 4
env.register_mime_type 'text/css', extensions: ['.css'], charset: :css
env.register_preprocessor 'text/css', MySprocketsExtension

In reality you wouldn't need to add a mime type for JS and CSS assets since they're already there by default but you will for other filetypes for example .coffee or .scss. The above register_mime_type example is used for example purposes and wouldn't be required.

Sprockets 4 will not chain asset extensions so .coffee.erb is explicitly registered in addition to .coffee. If your application introduces a new mime/extension combo it will be responsible for registering all combinations.

WIP

This guide is a work in progress. There are many different groups of people who interact with Sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into Sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using Sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.

These guides live in guides.