Parallel execution on self-hosted runners

[reichhartd]

We run this action on a self-hosted ubuntu and mac machine. If the action is run dedicatedly, everything runs smoothly. However, we have several runners installed on the machine, so ruby/setup-ruby action can be run in parallel. This leads to errors because they all access the same ruby version in /opt/hostedtoolcache/

Among others we got the following:

Run ruby/setup-ruby@v1
Error: Command failed: rm -rf "/Users/runner/hostedtoolcache/Ruby/3.0.1/x64"
rm: fts_read: No such file or directory

Installing Bundler
  Using Bundler 2.2.20 from Gemfile.lock BUNDLED WITH 2.2.20
  /opt/hostedtoolcache/Ruby/3.0.1/x64/bin/gem install bundler -v 2.2.20 --no-document
  `/opt/hostedtoolcache/Ruby/3.0.1/x64/lib/ruby/gems/3.0.0/gems/bundler-2.2.20/exe/bundle` does not exist, maybe `gem pristine bundler` will fix it?
  `/opt/hostedtoolcache/Ruby/3.0.1/x64/lib/ruby/gems/3.0.0/gems/bundler-2.2.20/exe/bundler` does not exist, maybe `gem pristine bundler` will fix it?
  ERROR:  While executing gem ... (Errno::ENOENT)
      No such file or directory @ rb_sysopen - /opt/hostedtoolcache/Ruby/3.0.1/x64/lib/ruby/gems/3.0.0/specifications/bundler-2.2.20.gemspec
  Took   0.75 seconds
Error: The process '/opt/hostedtoolcache/Ruby/3.0.1/x64/bin/gem' failed with exit code 1

Run ruby/setup-ruby@v1
Error: Command failed: rm -rf "/opt/hostedtoolcache/Ruby/3.0.1/x64"
rm: cannot remove '/opt/hostedtoolcache/Ruby/3.0.1/x64/lib/ruby/gems/3.0.0/gems/bundler-2.2.20/lib/bundler/man': Directory not empty

Other setup actions like node cache the version in the sub-folder _work/_tool of the respective runner. What I got from the source code so far is that you can't change the path because of the pre-built ruby version. Since you are much more familiar with Ruby and this repository, I would be happy if you have a solution for this problem.

[eregon]

Does the GHA runner generally support being run in parallel on the same filesystem?
GH-hosted runners don't do that at least.

For ruby releases, the toolcache is used if possible:

setup-ruby/common.js

Lines 116 to 118 in 517a3b1

	export function shouldUseToolCache(engine, version) {
	return engine === 'ruby' && !isHeadVersion(version)
	}

setup-ruby/ruby-builder.js

Lines 20 to 26 in 517a3b1

	if (common.shouldUseToolCache(engine, version)) {
	inToolCache = tc.find('Ruby', version)
	if (inToolCache) {
	rubyPrefix = inToolCache
	} else {
	rubyPrefix = common.getToolCacheRubyPrefix(platform, version)
	}

So actually one thing that might work would be to run it once (not in parallel), and then the next execution should find it in the tool cache.
For tc.find('Ruby', version) to return a path, we need to create some marker file to say it's complete: #98 (comment)
That's already done in

setup-ruby/ruby-builder.js

Line 65 in 517a3b1

common.createToolCacheCompleteFile(rubyPrefix)

So I think it should work fine if you execute the first job not in parallel (e.g., maybe you can make it part of the initial runner setup), or you manually install the Ruby version you use in the toolcache + create the x64.complete file.

We indeed cannot choose where to install the Ruby, so there is no choice there. And the current setup is already useful for self-hosted runners as it will only download the Ruby once and not multiple times.

File locking or other workarounds seem inherently fragile and complicated, so I'd rather this to be solved at runner installation time than in this action.

[eregon]

Closing, file locking seems too tricky and changing the path of the installed Ruby is not possible.
Workaround is run it not in parallel for the versions you use once per machine before running anything in parallel, or implement your own locking on top.

[grossadamm]

ruby/setup-ruby@v1 is fabulous and I use it successfully in many workflows. However, I too wanted a self-hosted parallel execution and was willing to tradeoff the awesome support here in a very niche case.

Caveat, this is completely unsupported. If you don't understand what is happening here and cannot self-maintain, don't use it. Don't bother the maintainers of this action with this either.

If someone wants to take this and run with it, be my guest. I'm aware there are optimizations left on the table but I am comfortable with this workaround in my own use case.

@eregon if this causes problems for you, I'm happy to remove this comment. I have no desire to undermine your amazing work and I agree with your comments and opinions stated above.

      - name: Set up Homebrew
        uses: Homebrew/actions/setup-homebrew@master
      - name: Setup local ruby
        run: |
          DIRECTORY="${{ runner.tool_cache }}/ruby/3.1.3"
          if [ -d "$DIRECTORY" ]; then
            echo "$DIRECTORY does exist, skipping ruby install"
            echo "$DIRECTORY/bin" >> $GITHUB_PATH
            exit 0
          fi
          brew install ruby-install
          ruby-install --install-dir "$DIRECTORY" ruby 3.1.3
          echo "$DIRECTORY/bin" >> $GITHUB_PATH
          which gem

[eregon]

Right, actually we should support people building their own Rubies in the tool cache, since anyway that's necessary for architectures & runner images not available on GitHub-hosted runners. I think it already works but needs better documentation.

So yeah that's one way to solve this issue, make sure it's installed before using it, potentially can also just run setup-ruby as a single job to set the ruby version before any parallel job.

[eregon]

With #473 this action now gives instructions for how to make it work for self-hosted runners not matching a GitHub-hosted runner image.

But there is still the issue of parallel extraction for self-hosted runners which use an image matching a GitHub-hosted runner image.
The error in the description is likely caused by the await io.rmRF(rubyPrefix).
We can probably replace that by a check that the directory does not exist or is empty.

[eregon]

We can probably replace that by a check that the directory does not exist or is empty.

That doesn't fully solve the problem, but at least it would avoid removing files which can fail other workflows which otherwise might work fine.

To really solve it I'd guess we'd need something like https://www.npmjs.com/package/proper-lockfile

[Leee-xx]

I've been using v1 with parallel workflows on self-hosted runners and never had an issue until today.

[eregon]

@Leee-xx I don't think recent changes change anything about this, so it's probably just you were lucky before with the ordering.

[Leee-xx]

We've been using this action in our parallel workflows (which actually live on completely different hosts) multiple times a week in multiple repositories since at least before November, so I doubt that we've been lucky that often.

…

On Fri, Mar 3, 2023, 12:53 PM Benoit Daloze ***@***.***> wrote: @Leee-xx <https://github.com/Leee-xx> I don't think recent changes change anything about this, so it's probably just you were lucky before with the ordering. — Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD7M4PLYE37JVVYHRWG5IDW2JK5PANCNFSM46XK37TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[eregon]

Giving it more thoughts, I think this is enough: #477
File locking doesn't seem necessary if we can assume extraction is idempotent and that people don't put random things into their toolcache/*Ruby that would cause problems.

FWIW here was a try at file locking, I suspect it could cause tricky issues in practice: master...eregon:setup-ruby:file-locking

[eregon]

Should be fixed in https://github.com/ruby/setup-ruby/releases/tag/v1.143.0

[grossadamm]

🎉