Generate source maps and configure digest bypass

[brenogazzola]

Depends on rails/sprockets#714

Updates Webpack's config to make it hash any chunk it generates. This causes Sprockets to skip fingerprinting the chunks, and therefore not breaking Webpack features that rely on knowing the chunk's exact name, such as lazy loading:

async bootstrap () {
  const { default: Trix } = await import(/* webpackChunkName: "trix" */ '../../chunks/trix')
  return Trix
}

@dhh I've tested this one by creating a new project from the current Rails main, and forcing bundler to use the sprockets PR we were discussing, then applying this configuration, and running on production mode. Worked great.

../rails/railties/exe/rails new mega --dev -j webpack

gem "sprockets", :git => 'git://github.com/FestaLab/sprockets.git', :branch => 'skip-digested-files'

[brenogazzola]

I had to remove the the LimitChunkCountPlugin otherwise chunking would not work. To avoid excessive number of chunks I've applied cacheGroups directives which prevents webpack from extracting code that is shared between multiple entrypoints. This should keep chunk count down. It's the exact configuration I've been running in production for over a year now.

[dhh]

Why do we need to keep the chunk count low? Seems like an optimization someone should just consider whether they need or not if it's a problem. Don't think we need that as a default.

[dhh]

(Just that we're trying to keep an absolute minimal config, just enough to make things work. Then someone can tweak as they see fit from there.)

[brenogazzola]

I added that because the current file has a plugin to keep chunk count at 1. I thought it was a priority. I’ll remove an retest to ensure everything is working, then update.

[dhh]

Ah, no, I only kept it at one chunk so we wouldn't have to deal with the issue with digested chunks not working 👍

[brenogazzola]

Removed and tested. It's ok. We don't need to try to keep chunk count low.

[dhh]

Sweet. Will merge once we've pushed out a new sprockets release. Just waiting for a review from @rafaelfranca on that.

[theodorton]

Are you able to get this to work as expected in development as well with config.assets.debug = true and config.assets.digest = true? I believe sprockets will be confused when attempting to serve the digested asset in development. It will happily serve up the entry point (because it's not hashed).

Or is the idea to not split chunks in development?

[brenogazzola]

Hu... I kept testing it only in production mode with precompile and missed this. Sprockets is really having trouble finding the hashed in dev. We will either need to disable chunking in dev or I'll have to open another PR for sprockets to (I think) fix the resolver.

Now if only I could figure out what is routing /assets to sprockets so I can go down this rabbit hole...

[brenogazzola]

An easier option would be to allow chunking in dev, but disable hashing. The digest and compile options have no impact since the one doing the compilation in development is the yarn process, not sprockets (at least on my quick test where setting digest true did not add the digest to the files created in app/assets/build).

[rmacklin]

@brenogazzola Do you have a public repository or gist with a testing app for this issue? I'm interested in looking at some approaches to solving this and other use cases, and if there isn't already a public repository perhaps we could sync up on creating one.

[theodorton]

Another approach I tried out was to reverse the order of the name and the hash [hash]-[name].js. While it doesn't look pretty, it works fine in development and it only requires an override in Procfile.dev to work with esbuild. Setting config.assets.digest = true in development should also make it resolve assets similarly to production.

[brenogazzola]

@rmacklin Here you go: lazy-loading-jsbundling.

I've created this with:

rails/railties/exe/rails new lazy-loading-jsbundling --dev -j webpack 

Then I switched rails, sprockets and jsbudling-rails in the Gemfile to use their github repos. I added the clipboard package to package.json and created a clipboard_controller that lazy loads the package.

If you go to localhost:3000 you will see a page that initializes the controller and tries to load the chunk.

Dynamic compiling (development):

1. Run ./bin/dev
2. Open localhost:3000
3. Check the network tab to see that clipboard chunk was not found

Precompiled (production):

1. Run ./bin/rails assets:precompile
2. Run ./bin/rails s
3. Open localhost:3000
4. Check the network tab to see that clipboard chunk was found

From what I learned about sprockets when submitting my PR, the precompiled version is relying on the manifest file which knows exactly where clipboard.js is, while the dynamic compiling is relying on something else and that is what needs fixing.

[brenogazzola]

Ok, the one to blame here is the server.rb file in Sprockets:
https://github.com/rails/sprockets/blob/cddf9fb841eece80276a1ccaee1e018a356547a0/lib/sprockets/server.rb#L45-L47

It is intentionally stripping away the fingerprint in the requested file path. I went through the history and got this:
https://github.com/rails/sprockets/blob/a2a792f78931550d813666e819061b9e8baefba9/lib/sprockets/trail.rb#L90-L111

From the comments it seems the original developers were not sure how valuable this behavior was. But since it was intentionally added, I don't think we can modify this without someone with considerably more experience in sprockets weighting in.

[brenogazzola]

Like I mentioned before, a possible fix is disabling hashing in development

const environment = process.env.NODE_ENV;

module.exports = {
  output: {
    chunkFilename: environment === 'production' ? '[name]-[contenthash].js' : '[name].js',
  }
}

This however means that to get the correct behavior in production it's necessary to have NODE_ENV=production set. It's the default in Heroku, and should not be a problem for those who are deploying by ourselves since we are already setting RACK_ENV=production and RAILS_ENV=production in our servers.

[theodorton]

From the comments it seems the original developers were not sure how valuable this behavior was. But since it was intentionally added, I don't think we can modify this without someone with considerably more experience in sprockets weighting in.

What about adding a configuration option for that and have jsbundling-rails append it to config/initializers/assets.rb on initialization? Then it could be marked as deprecated and dropped in the next major version release of sprockets.

[brenogazzola]

I tested commenting out the gsub, but that causes a few tests in test_server.rb to break and I'm frankly a bit afraid of changing anything here without having the context of why the current behavior is this.

Disabling hashing in dev is enough to fix the problem without risking unintentionally breaking something in existing apps.

[theodorton]

@brenogazzola I've made an attempt to solve this in Sprockets: rails/sprockets#717

This approach works fine in my environment, using the same esbuild configuration for development and production.

[brenogazzola]

Ah, nice. I like that solution. Let's hope the maintainer are happy with it too so we can get all this merged.

[brenogazzola]

@theodorton can you test this on my sample app? I'm not sure if I'm doing something wrong, but now that your PR is merged its breaking on javascript_include_tag "application". I've also tried to manually access assets/clipboard-[generated_hash].js and it's not finding it.

Make sure you do rm -rf app/assets/builds tmp public/assets between tests. I'm not sure why, but at some point sprockets starts using the manifest and everything works, but before that things are breaking.

What I'm seeing is that for application my break points in server.rb are not reached before they return with not found. This is odd, because the version without your PR does reach the breakpoints

For clipboard.js, server.rb reached your code, buuuuuuuuuuuut...

Location: sprockets-56b968690d63/lib/sprockets/resolve.rb

|    22:     #
|    23:     # The String Asset URI is returned or nil if no results are found.
|    24:     def resolve(path, load_paths: config[:paths], accept: nil, pipeline: nil, base_path: nil)
|    25:       paths = load_paths
|    26:
| => 27:       if valid_asset_uri?(path)
|    28:         uri, deps = resolve_asset_uri(path)
|    29:       elsif absolute_path?(path)
|    30:         filename, type, deps = resolve_absolute_path(paths, path, accept)
|    31:       elsif relative_path?(path)

At this point the builddirectory is not in the load_paths, so it cannot find the file, even if it knows it should look for the version with the digest in the name.

[brenogazzola]

I'm actually wondering how sprockets, when we are not messing with digests, knows to search the builds folder. I'll take a look when I have more time.

[theodorton]

Don't rm -rf app/assets/builds - I believe the asset pipeline actually depends on that folder to be present to work correctly. I had a lot of weird behavior happen when that folder was deleted. Not completely sure though, I assume @dhh could confirm.

[brenogazzola]

Oooops. You are right. Now that I'm deleting the files instead of the folder everything works. 🚀

[brenogazzola]

Adjusted the chunk name to be [name]-[contenthash].digested.js, as Sprockets/Propshaft automatic detection of the hash was causing false positives.

[brenogazzola]

Added sourcemaps.

[thanosbellos]

@theodorton can you test this on my sample app? I'm not sure if I'm doing something wrong, but now that your PR is merged its breaking on javascript_include_tag "application". I've also tried to manually access assets/clipboard-[generated_hash].js and it's not finding it.

Make sure you do rm -rf app/assets/builds tmp public/assets between tests. I'm not sure why, but at some point sprockets starts using the manifest and everything works, but before that things are breaking.

What I'm seeing is that for application my break points in server.rb are not reached before they return with not found. This is odd, because the version without your PR does reach the breakpoints

For clipboard.js, server.rb reached your code, buuuuuuuuuuuut...

Location: sprockets-56b968690d63/lib/sprockets/resolve.rb

|    22:     #
|    23:     # The String Asset URI is returned or nil if no results are found.
|    24:     def resolve(path, load_paths: config[:paths], accept: nil, pipeline: nil, base_path: nil)
|    25:       paths = load_paths
|    26:
| => 27:       if valid_asset_uri?(path)
|    28:         uri, deps = resolve_asset_uri(path)
|    29:       elsif absolute_path?(path)
|    30:         filename, type, deps = resolve_absolute_path(paths, path, accept)
|    31:       elsif relative_path?(path)

At this point the builddirectory is not in the load_paths, so it cannot find the file, even if it knows it should look for the version with the digest in the name.

Hello.

I am using your latest sprockets pull request and i am trying to try esbuild source maps on development mode.

require('esbuild').build({
  entryPoints: ['app/javascript/application.js'],
  bundle: true,
  outdir: 'app/assets/builds',
  sourcemap: true,
  // splitting: true,
  // format: 'esm',
  // chunkNames: '[name]-[hash]',
  entryNames: '[name]-[hash].digested',
  minify: !(process.env.NODE_ENV === 'development'),
  watch: watch
}).catch(() => process.exit(1))

i keep getting asset not found exception from this line
<%= javascript_include_tag "application", debug: false, "data-turbo-track": "reload", defer: true %>
in my application.html.erb

I have cleared tmp/cache and i have been really careful not to delete the 'app/assets/builds/' folder. Did you do anything else in order to resolve it?

[brenogazzola]

Yeah, esbuild is not compatible with our current solution. By setting the entryNames you are telling it to apply the hash to all files generated, instead of just chunks and sourcemaps, which is what we are doing with webpack.

So your application file is not application but application-askdjasd87a6s876sadf76a or something like that. I'm going to fix that on Propshaft next week, but in Sprockets it's going to be a lot more complicated...

[theodorton]

@thanosbellos we still need to have hexadecimal digests in esbuild as well, because the regex checking for a digest in propshaft does not support the Base32 hashes currently produced.. See evanw/esbuild#1613

As a workaround, since you're not using splitting, you can just omit the hash from esbuild and rely on propshaft producing the digest for you. Or do you have a specific use case in mind for having the hash in the entry file name?

[thanosbellos]

Thanks for your answers to both of you.

@theodorton tbh i am more or less experimenting with the new tools and esbuild. i don't have something specific in mind. More or less i would love to have splitting and source maps. Based on your comments and some search, this is what i have got working so far and i think is similar to the webpack proposed solution.

Keep in mind that i do use the sprockets version with @brenogazzola latest pull request that does permit esbuild like hash names.

require('esbuild').build({
  entryPoints: ['app/javascript/application.js', 'app/javascript/admin.js'],
  chunkNames: '[name]-[hash].digested',
  bundle: true,
  outdir: 'app/assets/builds',
  splitting: true,
  format: 'esm',
  watch: watch
}).catch(() => process.exit(1))

Unfortunately i can't really use propshaft right now. Although i am looking forward to it. I have not migrated to rails 7 alpha yet.

[brenogazzola]

Right now if you need sourcemaps, the only working options are:
1 - JS in sprockets, with debug = true
2 - Webpacker
3 - Jsbundling (webpack) + my PR

If you are already in jsbundling I'd suggest staying in webpack for now while we try to solve the esbuild problem.

[brenogazzola]

I think if we changed the digest bypass to only search the -digest part we could get esbuild to be compatible without any more major changes in propshaft or sprockets. But then we risk false positives (?). I think we need @dhh to weight in on this.

[dhh]

We can't have a solution where "javascript-something.js" is not digested in production. So need to figure something else out with that. Then other option, especially for esbuild, is just to output ES6 directly. Then you don't even need a source map.

[dhh]

Ended up making sprockets capable of dealing with source map comments created by transpilers directly via 3.4.0, so just leaned on that for #50.

This work is still valuable re: chunks, but let's treat that as a separate PR.

Also, have to replicate the source map handling I've added to sprockets-rails in Propshaft. Do you want to take a look at that @brenogazzola?