Document how to generate and host source maps for production bug tracking

[dimroc]

Goal: Give bug trackers like bugsnag access to sprockets generated source maps.

Hi all, I'm trying to have my bug tracker use source maps in production and staging. This references the discussion happening in #410 (comment).

I have successfully generated the source maps using the link directive in the manifest.js:

//= link source.js
//= link source.js.map

I am ok including sourceMapURL= in the final minified js file (despite it not being best practice). However, I've been unable to get this to happen for production builds, only debug. Maybe this is straightforward and I'm missing an option. Is it a simple configuration in an initializer? There is already this configuration for debug:

https://github.com/rails/sprockets/blob/master/lib/sprockets.rb#L106-L108

  register_pipeline :debug do
    [SourceMapCommentProcessor]
  end

I would prefer to just host the source map file and have my bug tracker pick it up from a url.

That being said, @vincentwoo mentioned that he POSTs it up to sentry.io. How do you know how to pair the source file with the map file if they have different digests? Do you have your own processor?

Thanks for any help.

System configuration

• Sprockets version 4 beta 5
• Ruby version 2.4.1 on rails 5.0.6

[plrthink]

same issues for me now, I'm doing the exact same thing with you besides I want to upload to Sentry. But I also have no idea how to add the source map URL to the generated file.

any updates? any suggestions would be grateful.

[schneems]

There's not a way to do it at the moment without enabling all debug features. You can set config.assets.debug = true. However that might not totally work.

I think it will map your sources correctly, but since those sources weren't compiled they won't be visible on the server. You'll still get the correct backtrace, and can use that to view your own sources locally though.

[emiellohr]

I don't get it. Can someone please explain? I've upgraded my Rails project to Sprockets 4, just to get source maps in production. Instead I got sourcemaps in development? How does this work? Do I need an additional initializer or config setting?

[cllns]

@emiellohr I imagine you don't need help anymore, but I just worked on this today. Hoping this will help others trying to do this too.

To config/environments/production.rb, add:

config.assets.debug = true
config.assets.resolve_with = %i[manifest]

The second part you need to specify because by default, Rails (sprockets?) only tries :manifest if debug = false, which it's clearly not in our case. The other value that can be in that array is environment but that doesn't work for me, even as a backup, and I'm using :manifest anyway. If you don't do this, Rails will generate non-digested pathnames, which won't work, since sprockets still digests them when they're compiled.

debug = true for production feels weird, but it's not so bad. Even in development, it still concatenates the file (this is different than the normal behavior, where debug = true typically means that the files are unconcatenated and unminified) no matter what. (I'm guessing this is manifest.js behavior.)

Also in config/environments/development.rb, config.assets.debug is probably already true (if it's not, make it true). surprisingly here, you need to add a JS compressor in order to get the source maps to work in development:

config.assets.js_compressor = :uglifier

( I guess you could set this in config/initializers/assets.rb or elsewhere, if you wanted, but that means the :test environment would compress JS, which will probably not handle sourcemaps as elegantly as the browser).

Then, in your app/assets/config/manifest.js, add a corresponding .map file for each .js file you have. For example:

//= link application.js                                                                                        
//= link application.js.map

Now, when you deploy, the map will be compiled too. Letting people see your JS is a minor security problem, but, crucially, there's no link to it in your JS files, so it's very hard to find.

The reason development knows where your source map is, is that the following line is added to the bottom of your minified JS. Thankfully, this does not happen in production.

//# sourceMappingURL=application.js-d54377f4bfe13c83f772d0a7c353127a0d7388afe67fcca1344b5cdac0370c1c.map

If you're concerned about it still being available (though hard to find), you could manually delete the compiled .map file from your compiled assets as a part of your build process.

The above notes might be wrong, it's just my experience from working on this today. Hopefully it helps someone :)

(I still need to work on modifying my build process to send the source map to my error logging service, Rollbar. That should be relatively simple, since they're being generated already.)

[supairish]

@cllns Thanks for sharing that info!

[seanlinsley]

Thanks for the tips @cllns.

Has anyone managed to get sourceMappingURL comments to show up in production?

[seanlinsley]

The commit that introduces sourceMappingURL sets pipeline: :debug in its test case, but that doesn't appear to be set anywhere else in the Sprockets source code.

The only place I can find it is in the sprockets-rails gem. javascript_include_tag calls this:

def find_debug_asset(path)
  if asset = find_asset(path, pipeline: :debug)
    raise_unless_precompiled_asset asset.logical_path.sub('.debug', '')
    asset
  end
end

If I'm understanding correctly, this means that Sprockets only adds sourceMappingURL when assets are dynamically compiled. While I understand orgs wanting not to expose their unminified source, it's security through obscurity (meaning it's not adding any real security). It seems I'm in the minority, and Sprockets seems to already pretty far along in the beta process, so at most we should add a configuration option to enable it for static compilation.

I'd be happy to open a PR for this, but I'm not sure exactly what code needs to change. default_source_map.rb and source_map_utils.rb seem like candidates, but I'd appreciate guidance

[seanlinsley]

I was hoping I could just do this, but it results in infinite recursion:

env.register_bundle_processor 'application/javascript',
  Sprockets::AddSourceMapCommentToAssetProcessor

And it doesn't appear that bundle processors have access to the compiled file paths. So perhaps there should be a concept of a bundle postprocessor?

[seanlinsley]

I couldn't find a good place to patch Sprockets, so I ended up writing an extension to rake assets:precompile. It mostly works (though the source files for CoffeeScript code can't be found).

https://gist.github.com/seanlinsley/ab0fb9fbc063e9812629be8cb6a92f2f

The application I'm working on previously relied on having undigested assets for some JS libraries to use, which is a good thing b/c it turns out that the source maps that are generated link to undigested asset paths. Without the undigesting code, every file in the dev tools would be empty like this:

[seanlinsley]

@cllns mentioned this:

surprisingly here, you need to add a JS compressor in order to get the source maps to work in development

which appears to be true, though oddly breakpoints in a debugger don't stay in the right location like they do when the assets are statically compiled

[ajaps]

Hey @cllns were you able to send the source map to Rollbar? I'm having the same issue.

[cllns]

@framky007 I was able to send the source maps to Rollbar, but not in the way Rollbar expected (so, they didn't work in Rollbar's interface)

[cdesch]

@cllns Were you able to find a way to generate the js source maps for rollbar?

[ajaps]

For those with similar issues, I was able to resolve it using the following steps

1. upgrade sprockets to v4 beta 7 . gem 'sprockets', '~> 4.0.0.beta7'

2. create a manifest file

 //= link application.js.map
 //= link_directory ../javascripts .js
 //= link_directory ../stylesheets .css

The above should generate source map when you run this command. bundle exec rake assets:precompile RAILS_ENV=production

To send source map to an error reporting service(in this case Rollbar)

From Rollbar's documentation, the preferred method is via their API

3. create a rake task that sends the generated source map via the API - this should be done before deployment

sourcemap.rake

namespace :assets do 
  LOCAL_SOURCEMAP_PATH = `search/get the generated sourcemap path`

  def sourcemap_to_rollbar
    puts HTTP.post("https://api.rollbar.com/api/1/sourcemap",
      form: {
        access_token: ROLLBAR_TOKEN,
        version: VERSION_IDENTIFIER,
        minified_url: MINIFIED_URL_PATH,
        source_map: HTTP::FormData::File.new(LOCAL_SOURCEMAP_PATH)
      }
    )
  end

task :precompile do
    if Rails.env.production?
      sourcemap_to_rollbar
    end
  end

Here I'm using https://github.com/httprb/http for making request from Ruby

Note: The above rake task executes after the default rake assets:precompile runs, it doesn't replace the default rails task

Alternately, If you don't mind putting the sourcemap url in the minified JS --->

def sourcemap_to_rollbar
  source_map = Dir.glob("#{'public/assets'}/**/*application-*.map") //get source map path
  minfied_file = Dir.glob("#{'public/assets'}/**/*application-*.js")  //get minifies js

  minfied_file.open(file, "a+"){|f| f << "\n //# sourceMappingURL=" + source_map } //place url path at the bottom of the minified JS
end

[adamcrown]

Any update on this?

I'm trying to get source maps working in production as well. I was able to get them generated by following @cllns's instructions above. But I'm also not able to get it to add a sourceMappingURL comment.

Now that source maps in production by default seems to be the official Rails position, can we expect some movement on this?

[jarthod]

Sorry to bump this but we're also in the same situation, we want source maps in production (like DHH). After waiting years for sprockets to support this we were very happy to see that sprockets 4 officially added support (thanks 🙇), but then when trying to upgrade we noticed there's actually no way to use it in production... (without brittle hacks mentioned above).

I totally understand that there may be a majority still considering this a bad practice and thus keeping it disabled by default in production seem ok. But there could at least be an option to enable it for people who want to, no?

Thanks!

[nanaya]

I got as far as this (without modifying existing sprockets pipeline, based on comments above):

lib/tasks/sourcemap.rake

namespace :assets do
  def append_sourcemap
    assets = JSON.parse(File.read(Dir[Rails.root.join('public/assets/.sprockets-manifest-*.json').to_s][0]))['assets']

    assets.each do |name, digested|
      ext = File.extname(name)
      next unless ['.css', '.js'].include? ext

      map_digested = assets["#{name}.map"]
      next unless map_digested

      file = Rails.root.join("public/assets/#{digested}")
      mapping_string = "sourceMappingURL=#{map_digested}"

      mapping_string = case ext
        when '.css' then "/*# #{mapping_string} */"
        when '.js' then "//# #{mapping_string}"
      end

      next if file.readlines[-1].include? mapping_string

      file.open('a+') { |f| f << "\n#{mapping_string}" }
    end
  end

  task precompile: :environment do
    append_sourcemap
  end
end

This will append sourcemap to generated js files assuming they also have corresponding .map file.

The problem is the .map files generated by sprockets don't include actual source. It needs to be extended to include sourcesContent for every objects which contain sources in it.

As for esbuild output, here's my build.js script:

build.js

#!/usr/bin/env node

import babel from '@babel/core'
import { createHash } from 'crypto'
import esbuild from 'esbuild'
import coffeeScriptPlugin from 'esbuild-coffeescript'
import fsPromises from 'fs/promises'

const outfileName = 'application.jsout'
const outfileEsbuild = `tmp/${outfileName}`
const outfileBabel = `app/assets/builds/${outfileName}`

const plugins = [
  coffeeScriptPlugin({
    bare: true,
    inlineMap: true
  }),
  {
    name: 'babel',
    setup (build) {
      build.onEnd(async () => {
        const options = {
          minified: true,
          presets: [
            ['@babel/preset-env']
          ],
          sourceMaps: true
        }
        const outEsbuild = await fsPromises.readFile(outfileEsbuild)
        const result = await babel.transformAsync(outEsbuild, options)
        result.map.sources = result.map.sources
          // CoffeeScript sourcemap and Esbuild sourcemap combined generates duplicated source paths
          .map((path) => path.replace(/\.\.\/app\/javascript(\/.+)?\/app\/javascript\//, '../app/javascript/'))
        const resultMap = JSON.stringify(result.map)
        const resultMapHash = createHash('sha256').update(resultMap).digest('hex')

        return Promise.all([
          // add hash so it matches sprocket output
          fsPromises.writeFile(outfileBabel, `${result.code}\n//# sourceMappingURL=${outfileName}-${resultMapHash}.map`),
          fsPromises.writeFile(`${outfileBabel}.map`, JSON.stringify(result.map))
        ])
      })
    }
  },
  {
    name: 'analyze',
    setup (build) {
      build.onEnd(async (result) => {
        if (options.analyze) {
          const analyzeResult = await esbuild.analyzeMetafile(result.metafile)

          console.log(analyzeResult)
        }
      })
    }
  },
]

const args = process.argv.slice(2)
const options = {
  watch: args.includes('--watch'),
  analyze: args.includes('--analyze')
}

esbuild.build({
  bundle: true,
  entryPoints: ['app/javascript/application.coffee'],
  metafile: options.analyze,
  nodePaths: ['app/javascript'],
  outfile: outfileEsbuild,
  plugins,
  resolveExtensions: ['.coffee', '.js'],
  sourcemap: 'inline',
  watch: options.watch
})

The idea is to output the file to extension not recognized by sprockets so the files are not processed further. The map file in the comment point to digested filename as output by sprockets (with config.assets.version set to nil for predictable hash)

(should be slightly simpler if not using babel and coffeescript)