Issues with concurrent bundle installs with shared gem cache

[stefansedich]

I am attempting to use a shared gem cache with a docker-compose setup, a sample project is available here: https://github.com/stefansedich/bundle-test which easily replicates the issue by simply running run.sh.

The error is consistent and one of the two containers will fail to compile the native extensions for in this case the pg gem I assume due to the concurrent installs attempting to create the Makefile.

What is the best way to handle this scenario, my current solution is to wrap my bundle install in a script that will use flock to limit concurrent installs.

[indirect]

@stefansedich thank you for reporting this!

@segiddins ugh this scenario did not even occur to me 😭 I'm thinking process lock file a la git?

[stefansedich]

FYI my current solution using a install wrapper script does the trick!

#!/bin/bash

set -e

(
  flock 200

  bundle install
) 200>/.gems/install.lock

[stefansedich]

@indirect if a process lock is the desired solution and everyone is happy with that I am happy to tackle it and submit a PR for it.

[indirect]

@stefansedich I think that's the way to go for now—we're not designed to allow multiple concurrent processes. Maybe a future goal can be optimizing that beyond a process lock.

[stefansedich]

Sounds good, from my initial look it looks like Installer#run is the best place to put the lock, does that sound right to you?

[indirect]

That sounds right to me. You'll also want to check the update command and make sure it shares the lock. 👍🏻

[segiddins]

We could also do everything in temporary directories and move stuff since moves are atomic

[stefansedich]

@segiddins does it make more sense to just wait? otherwise in this example I compile PG twice in parallel, if we just lock and wait by the time the second install can continue PG should already be compiled and ready to use?

[segiddins]

You've given the install command twice, so I don't think that installing twice is out of the question. In addition, in my experience proper multi-process coordination is a lot harder than just sticking with atomic ops

[deivid-rodriguez]

This was apparently fixed by rubygems/bundler#5852, but never closed.