FileStore cache patch in v3.6 causes directory name collisions in Windows

[atruskie]

The changes introduced in bc40f15 add a namespace directory to the digest key.

Unfortunately, on Windows the namespace folder causes conflicts due to case sensitivity. For example: lc, Lc, lC all resolve to the same folder when creating the directory. When it comes time to resolve the asset in the folder the correct file cannot be found.

Steps to reproduce:

1. Use sprockets v3.6 on a case-insensitive file system
2. Clear the tmp folder for the application (a rails application in my case)
3. Start the application (thin start in my case)
   1. sprockets starts to generate a bunch of cache files (for example /home/vagrant/baw-server/tmp/cache/assets/sprockets/v3.0/k5/k5Tf66eynpbdWEXajNU1EtcNgCTk8C201A0Vy-aBzgg.cache
4. Try and load a page that depends on a sprockets resource

This exception occurs:

ActionView::Template::Error (Not a directory @ rb_sysopen - /home/vagrant/baw-server/tmp/cache/assets/sprockets/v3.0/K5/K5HCuh0G2rUW6bW13cTaz7GiHa1K1gCReNVyrOUHixw.cache.5855600.7132.945301):
    14:     / Le HTML5 shim, for IE6-8 support of HTML elements
    15:     /[if lt IE 9]
    16:       = javascript_include_tag '//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js'
    17:     = stylesheet_link_tag (content_for?(:stylesheet_link_tag_name) ? content_for(:stylesheet_link_tag_name) : 'application'), media: 'all'
    18:     %link(href="images/apple-touch-icon-144x144.png" rel="apple-touch-icon-precomposed" sizes="144x144")
    19:     %link(href="images/apple-touch-icon-114x114.png" rel="apple-touch-icon-precomposed" sizes="114x114")
    20:     %link(href="images/apple-touch-icon-72x72.png" rel="apple-touch-icon-precomposed" sizes="72x72")
  app/views/layouts/application.html.haml:17:in `_app_views_layouts_application_html_haml__1709115770063515875_49642660'
  app/controllers/public_controller.rb:39:in `index'

It appears that when sprockets asked for the K5 directory to be created, it returned with no side effect because the directory already existed (as k5). Then when actually creating the file, the ENOTDIR error was raised because the directory didn't actually exist.

Rolling back to version 3.5.2 fixes the issue for me.

I suspect similar problems may have been possible before the bc40f15 patch, however, such a large collision space (10^105) means the problem didn't occur. Once you account for case-insensitive operation, there's only 1,444 combinations available in a 2 character sequence.

Other details:

• I'm using Vagrant

  • host: Windows 10, NTFS file system
  • guest: Ubuntu 14.04.4 LTS
  • synced folder using nfs

• I've tried to isolate the problem by testing the following:

  • mkdir K5 also fails (does not create the K5 directory, does not error) in Ubuntu

  • mkdir.exe fails in Windows with mkdir.exe: cannot create directory K5': File exists`

  • FileUtils.mkdir_p on Ubuntu, in nfs mounted directory, gives (only one folder created and has casing from first command)

    2.2.4 :002 > require 'fileutils'
    => true
    2.2.4 :003 > FileUtils.mkdir_p 'K5'
    => ["K5"]
    2.2.4 :004 > FileUtils.mkdir_p 'k5'
    => ["k5"]
    

  • FileUtils.mkdir_p on Windows gives (only one folder created and has casing from first command)

    irb(main):001:0> require 'fileutils'
    => true
    irb(main):010:0> FileUtils.mkdir_p 'K5'
    => ["K5"]
    irb(main):011:0> FileUtils.mkdir_p 'k5'
    => ["k5"]
    

  I'm not sure what the best solution is, but the only thing that comes to mind is to downcase the namespace before using it.

[schneems]

cc @rafaelfranca

[jeremy]

+downcase. Affects OS X too.
On Tue, Apr 26, 2016 at 07:56 Richard Schneeman notifications@github.com
wrote:

cc @rafaelfranca https://github.com/rafaelfranca

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#283 (comment)

[schneems]

Is there a hashing algorithm that produces lowercase characters only in their results string? Splitting the string and down-casing on every cache lookup would be expensive :(

[jeremy]

Touching the filesystem has got to be an order of magnitude more expensive than downcasing a couple chars. We're already splitting the string to get the dir name, huh?

[schneems]

The current implementation can have collisions for the whole asset, would be nice to not have to do that.

Not sure about your filesystem comment, we're already having to read in the contents to create the digest, was suggesting maybe there exists a different digest algorithm that produces strings without varying case.

[schneems]

For example Digest::SHA256.hexdigest only uses hex, so it never generates uppercase characters.

[jeremy]

Gotcha. Speaking to:

Splitting the string and down-casing on every cache lookup would be expensive

to suggest that we're already splitting the string and that downcasing is inexpensive 😊

[schneems]

So it looks like the filenames are already using hex

require 'securerandom'
string = SecureRandom.hex
Digest::SHA256.hexdigest(string) == (Digest::SHA256.new << string).digest.unpack('H*'.freeze).first
# => true

I need to look into using hexdigest method only, as it's faster than the current implementation.

The reported problem is coming from the cache blobs and not from filename digests. Still looking at where those are coming from. Maybe we could use hexdigest there too

[konung]

Just adding to the error report + possible temporary solution.

My dev setup is a Debian box in Vagrant on Windows 7 host with winnfsd plugin for vagrant to simulate NFS. I'm getting the same error. Temporary solution for me is to explicitly set sprockets to version 3.5.2 in Gemfile.

[schneems]

I guess I don't know how we can be making file paths with upper and lowercase characters as hexdigest wouldn't allow it. Maybe something else is happening here?

[satishchhibba]

(from Schneems: This comment is not related to current ticket)

I am using Rails 5.0.0. RC1 with sprockets 3.1.1 and have this same problem.
It got worse over days where 50+% of the time I would get this error (in development mode) on any request -- success rate was below 20-30% and I got very sick of it.

Env. is Debian .. and there is no separate client.
I looked inside tmp/assets and Sprockets 3.0 directory was littered with hashes with case-conflict tag.
just a partial row of the directory ..

     --  IH (Case Conflict)  Mv (Case Conflict)  Rb (Case Conflict)  Wd (Case Conflict)

On Debian there should be no issue with case conflicts .. I suppose the 2 char hash is broken.
Switched to Sprocket 3.0.4 and behavior WAS same

Cleaning the tmp seems to have fixed it for now -- but I am sure this is not the fix .. and I don't know enough but wasn't development mode supposed to be "raw" where everything is reloaded ..

It seems not be the case anymore -- I guess the whole platform is so bloated now that it can't move without crutches :-)

[schneems]

The behavior the original ticket describes was not in Sprockets 3.1.1

Can you open a new issue and give us an example app that reproduces the behavior. Also please try with the latest 3.6 version we won't apply patches against an older minor version of Sprockets.

[satishchhibba]

(from Schneems: This comment is not related to current ticket)

Sorry, I didn't check this for the past 2 weeks
I wasn't being very careful in my post earlier- I should have said sprocket-rails 3.1.1

This may be the wrong thread for this issue as it manifests on sprocket-rails -- though I believe that this cache issue is probably core sprocket and not rails specific.

This week, I created a fresh VM with debian with 4.16 kernel and ruby 2.3 rails 5.0 + postgres and bootstrap and it is showing exactly the same behavior.

Can you make a suggestion on creating an app -- on github? my new code is really small but the DB is 100MB and I don't have a test suit that you could use :-(
( I am moving an old application from rails 3.2 mysql to rails 5.0 pg)

[schneems]

Can you make a suggestion on creating an app -- on github

Run $ rails new then add what ever files you need, put sprockets 3.6.x in the gemfile, once you reproduced the problem go to github.com make a new project push your example app to the github project, when you're done open a new issue and link to the github project.

Please quit responding on this thread, it is detracting from the original issue.

[schneems]

Is there still an issue here with sprockets 4?

[satishchhibba]

I haven't upgraded to Sprockets 4 .. Our production schedule is very tight and no upgrades are doable before Feb-2018. If I see the problem I will report. Sorry about the delay . -SC

…

On Fri, Nov 17, 2017 at 10:38 AM, Richard Schneeman < ***@***.***> wrote: Is there still an issue here with sprockets 4? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#283 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGXe6qlNLLcFnAluL-uKWChWHavw6RNQks5s3aiPgaJpZM4IPmnE> .

-- -satish

[bistline]

I'm seeing this issue as well when running inside Docker for MacOSX using Rails 5.2.0 and Sprockets 3.7.2. I cannot load any pages in development - I get the following error:
File exists @ dir_s_mkdir - /home/app/webapp/tmp/cache/assets/sprockets/v3.0/[directory name]. My Docker image is based off of the phusion passenger image: https://github.com/phusion/passenger-docker

Precompiling assets will temporarily resolve the issue, but this isn't a tractable solution for development work. If I backdate to an image that was based off of Rails 4.2, I don't have this problem. So there's something now with the combination of Rails 5 & sprockets 3.7 that is at fault. I updated to 3.7.2 because of CVE-2018-3760, so I can't backdate my version.

My Gemfile.lock: https://github.com/broadinstitute/single_cell_portal_core/blob/development/Gemfile.lock

Any suggestions?

[nateleavitt]

I'm seeing the exact same problem as @bistline with the same images.

EDIT: Just found out I had some gems conflicting with each other in my Gemfile.lock file. I regenerated it and that solved my problem.

[Smolations]

Re: @bistline and others

This is not only an issue with this particular gem, but I have seen issues with node packages and poorly-named imports as well. I dev on OS X, and to mitigate the problem, I created a new partition that is case-sensitive, journaled (it's actually one of the options when partitioning) and that seems to resolve the issues and prevent any future ones. This may not be applicable depending on IT restrictions (if it's a work computer) but I've found it to be a better overall solution as opposed to modifying app dependencies.

In any case, I ran into this exact problem after doing a rake tmp:cache:clear and then trying to run the app again locally, so I'm gonna get my partition on once IT gives me the go-ahead. 😋

[nicwillemse]

Same problem here, the issue seems to be introduced with v 3.6.0, downgrading to 3.5.2 seems to fix the issue.

[pheuko]

Like @Smolations commented, looks like this is related with how Windows manages filenames.

Enabling case sensitivity on /tmp folder fixed the issue for me:

fsutil.exe file SetCaseSensitiveInfo C:\folder\path enable

[pixeltrix]

@nicwillemse literally also just stumbled onto this due to trying something out in Docker Desktop for Windows and the issue is that a base64 digest is case sensitive so all versions since 372301c could cause a conflict. However the namespace change in 3.6 greatly increases the chances of that occurring since it's just two characters.

I'm working around the problem for now by mounting a docker volume as a custom cache path for Sprockets, e.g:

# config/initializers/assets.rb

# Allow overriding of the sprockets cache path
Rails.application.config.assets.configure do |env|
  env.cache = Sprockets::Cache::FileStore.new(
    ENV.fetch("SPROCKETS_CACHE", "#{env.root}/tmp/cache/assets"),
    Rails.application.config.assets.cache_limit,
    env.logger
  )
end

# docker-compose.yml
web:
  environment:
    - BUNDLE_PATH=/bundle
    - SPROCKETS_CACHE=/cache
  volumes:
    - .:/app
    - bundle:/bundle
    - cache:/cache

volumes:
  bundle:
  cache:

Full commit here

[nicwillemse]

Thanks @pheuko - but this unfortunately will not work as the attributes are not inherited in the folder structure, see https://devblogs.microsoft.com/commandline/per-directory-case-sensitivity-and-wsl/ for more detail.

Like @Smolations commented, looks like this is related with how Windows manages filenames.

Enabling case sensitivity on /tmp folder fixed the issue for me:

fsutil.exe file SetCaseSensitiveInfo C:\folder\path enable

[nicwillemse]

Thanks @pixeltrix - will give it a try, but looks to be a good solution.

[A-Mozeak]

@nicwillemse just updating here to note that Per-Directory Case Sensitivity for WSL has been improved to pass the attributes to newly created subdirectories. In my case I made a new local directory with the case sensitive attribute, then re-cloned my repo into it:

mkdir case_sensitive_folder
setfattr -n system.wsl_case_sensitive -v 1 case_sensitive_folder
git clone git@github.com:myrepo case_sensitive_folder

The solution from @pixeltrix is probably the better option for different or more containerized projects, but in my case the above solution was ideal.

Thanks @pheuko - but this unfortunately will not work as the attributes are not inherited in the folder structure, see
https://devblogs.microsoft.com/commandline/per-directory-case-sensitivity-and-wsl/ for more detail.

[DannyBen]

Thank you for this fix, it works - but this is really putting a damper on my rails experience.

Since this is going on for so long, wouldn't it at least be simple enough to rescue this error and ignore it? Since the folder already exists?

I am also on Vagrant Ubuntu guest on a Windows host. Up until now, I symlinked my tmp folder so that it actually resides outside of the Windows host, but I am unable to symlink anymore for some reason.

[skepsys]

Have had the same issue while building assets on Win10 with Docker (on Hyper-V):
Errno::EEXIST: File exists @ dir_s_mkdir - /var/www/html/tmp/cache/assets/sprockets/v4.0.0/xb
And indeed it was related to filename case sensitivity on Windows. For example, sprockets was trying to create xb folder when xB folder was already there:

As a solution I just switched Docker to WSL support mode (WSL v2 enabled), pruned and rebuild all project images and used WSL console instead of Powershell. The issue has gone away.