Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rubocop --parallel failing with "negative array size" #349

Closed
mark-young-atg opened this issue Aug 8, 2024 · 15 comments
Closed

rubocop --parallel failing with "negative array size" #349

mark-young-atg opened this issue Aug 8, 2024 · 15 comments

Comments

@mark-young-atg
Copy link

Our github action to run rubocop started failing when parallel was upgraded from 1.25.1 to 1.26.0

This is the error seen

Run bundle exec rubocop --parallel
negative array size
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `initialize'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `new'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `create_workers'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:520:in `work_in_processes'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:291:in `map'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:235:in `each'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:103:in `warm_cache'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:76:in `run'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
...snip...

I presume this is because of the following change that uses Concurrent.available_processor_count.floor to determine the number of available processors if the gem concurrent-ruby is installed, which in my case it is.

I can see that I can use the environment variable PARALLEL_PROCESSOR_COUNT to override this behaviour for my configuration but I think there is an issue here to be understood and mitigated around the behaviour in github actions. Whilst I appreciate the incorrect value may come from concurrent-ruby, I think perhaps that parallel should handle a negative value in a different way. Perhaps by failing back to using Etc.nprocessors as the previous version did.

I have not created a PR for this because I felt it needed discussion and knowledge from those that understand parallel vastly more than I do.
@mark-young-atg
Copy link
Author

As a further comment on this, I checked our Amazon AWS instance and the use of Concurrent.available_processor_count.floor looks as though it will cause problems by returning a negative number. This is what I see on our ECS instance:

irb(main):001> Concurrent.available_processor_count.floor
=> -1
irb(main):002> Etc.nprocessors
=> 2

@grosser
Copy link
Owner

grosser commented Aug 8, 2024 via email

@ovsant
Copy link

ovsant commented Aug 8, 2024

Same issue

bundle exec rubocop --color
negative array size
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `initialize'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `new'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `create_workers'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:520:in `work_in_processes'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:291:in `map'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:235:in `each'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:103:in `warm_cache'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:76:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command.rb:11:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/environment.rb:18:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:122:in `run_command'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:129:in `execute_runners'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:51:in `block in run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:81:in `profile_if_needed'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:43:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `block in <top (required)>'
/usr/local/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `<top (required)>'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `load'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `<main>'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `eval'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `<main>'

@mark-young-atg
Copy link
Author

Thank you very much @grosser. I think that was a sensible move.

@mark-young-atg
Copy link
Author

mark-young-atg commented Aug 8, 2024 •
edited
Loading

@grosser although I think it might be sensible to fix forward by releasing v1.26.1 which reverts that commit. The reason being I'm still finding that bundle update and bundle install try to give me v1.26.0, possibly due to the gem still being installed locally on my laptop. Obviously I know to remove it with gem uninstall parallel:1.26.0 but this might catch out others and CI/CD systems.

@grosser
Copy link
Owner

grosser commented Aug 8, 2024 via email

grosser added a commit that referenced this issue Aug 8, 2024
@grosser
Revert "Merge pull request #348 from y-yagi/use_available_processor_c…
c316cc5 
…ount"

This reverts commit 21fdad7, reversing
changes made to 9644d13.

see #349
@grosser
Copy link
Owner

grosser commented Aug 8, 2024

1.26.1 is out

@grosser grosser closed this as completed Aug 8, 2024
@grosser grosser mentioned this issue Aug 8, 2024
Merged
y-yagi added a commit to y-yagi/concurrent-ruby that referenced this issue Aug 9, 2024
@y-yagi
Fix the return value of Concurrent.available_processor_count when `…
ddf62f6 
…cpu.cfs_quota_us` is -1

I tried to use `Concurrent.available_processor_count` in `parallel`
gem, but we got some feedback `Concurrent.available_processor_count`
returned a negative value.
grosser/parallel#348 (comment)
grosser/parallel#349 (comment)

According to the https://docs.kernel.org/scheduler/sched-bwc.html#management,
The default value of `cpu.cfs_quota_us` is -1. In that case,
cgroup does not adhere to any CPU time restrictions.

This PR adds the case of `cpu.cfs_quota_us` is -1 to `#cpu_quota`
to return processor count from `Concurrent.available_processor_count`
in that case.
@y-yagi y-yagi mentioned this issue Aug 9, 2024
Merged
y-yagi added a commit to y-yagi/concurrent-ruby that referenced this issue Aug 10, 2024
@y-yagi
Fix the return value of Concurrent.available_processor_count when `…
7306b26 
…cpu.cfs_quota_us` is -1

I tried to use `Concurrent.available_processor_count` in `parallel`
gem, but we got some feedback `Concurrent.available_processor_count`
returned a negative value.
grosser/parallel#348 (comment)
grosser/parallel#349 (comment)

According to the https://docs.kernel.org/scheduler/sched-bwc.html#management,
The default value of `cpu.cfs_quota_us` is -1. In that case,
cgroup does not adhere to any CPU time restrictions.

This PR adds the case of `cpu.cfs_quota_us` is -1 to `#cpu_quota`
to return processor count from `Concurrent.available_processor_count`
in that case.
y-yagi added a commit to y-yagi/concurrent-ruby that referenced this issue Aug 10, 2024
@y-yagi
Fix the return value of Concurrent.available_processor_count when `…
7fea31d 
…cpu.cfs_quota_us` is -1

I tried to use `Concurrent.available_processor_count` in `parallel`
gem, but we got some feedback `Concurrent.available_processor_count`
returned a negative value.
grosser/parallel#348 (comment)
grosser/parallel#349 (comment)

According to the https://docs.kernel.org/scheduler/sched-bwc.html#management,
The default value of `cpu.cfs_quota_us` is -1. In that case,
cgroup does not adhere to any CPU time restrictions.

This PR adds the case of `cpu.cfs_quota_us` is -1 to `#cpu_quota`
to return processor count from `Concurrent.available_processor_count`
in that case.
eregon pushed a commit to ruby-concurrency/concurrent-ruby that referenced this issue Aug 10, 2024
@y-yagi @eregon
Fix the return value of Concurrent.available_processor_count when `…
98d0f16 
…cpu.cfs_quota_us` is -1

I tried to use `Concurrent.available_processor_count` in `parallel`
gem, but we got some feedback `Concurrent.available_processor_count`
returned a negative value.
grosser/parallel#348 (comment)
grosser/parallel#349 (comment)

According to the https://docs.kernel.org/scheduler/sched-bwc.html#management,
The default value of `cpu.cfs_quota_us` is -1. In that case,
cgroup does not adhere to any CPU time restrictions.

This PR adds the case of `cpu.cfs_quota_us` is -1 to `#cpu_quota`
to return processor count from `Concurrent.available_processor_count`
in that case.
@eregon
Copy link

eregon commented Aug 10, 2024

Concurrent.available_processor_count is now fixed upstream and released: ruby-concurrency/concurrent-ruby#1060
Sorry for the bug!

@grosser
Copy link
Owner

grosser commented Aug 11, 2024

re-released as v1.26.2 🤞

@honeyankit
Copy link

honeyankit commented Aug 15, 2024 •
edited
Loading

@grosser : I am still getting negative array size error.

negative array size
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `initialize'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `new'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `create_workers'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:520:in `work_in_processes'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:291:in `map'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:235:in `each'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:103:in `warm_cache'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:76:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command.rb:11:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/environment.rb:18:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:122:in `run_command'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:129:in `execute_runners'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:51:in `block in run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:81:in `profile_if_needed'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:43:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `block in <top (required)>'
/usr/share/rbenv/versions/3.3.4/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `load'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:492:in `exec'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:34:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:28:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:37:in `block in <top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `load'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `<main>'

Edit: I had to manually update concurrent-ruby to 1.3.4 to fix the issue based on the comment here

@grosser
Copy link
Owner

grosser commented Aug 15, 2024 via email

@grosser
Copy link
Owner

grosser commented Aug 15, 2024 •
edited
Loading

... and try updating to latest bundle update concurrent-ruby and see if that fixes it 🤞

@VitaliySerov
Copy link

Shouldn't concurrent-ruby v1.3.4 set as minumal version of dependcy of parallel in that case?
or rubocop (not sure where the problem comes from)

y-yagi added a commit to y-yagi/parallel that referenced this issue Aug 16, 2024
@y-yagi
Ensure not to use old concurrent-ruby
43c9805 
It seems that some users encounter with a negative values issue of
`Concurrent.available_processor_count`.

grosser#349 (comment)

To avoid the issue, this PR ensure not to use old `concurrent-ruby`.
This PR also removes considering of `NoMethodError` because we can
assume to use `available_processor_count` always.
@y-yagi y-yagi mentioned this issue Aug 16, 2024
Merged
@eregon
Copy link

eregon commented Aug 16, 2024

A simple solution could be to check if Concurrent.available_processor_count returns <= 0 and if so use Etc.nprocessors.
Or to add concurrent-ruby as a gem dependency to ensure it's always >= 1.3.4.

@grosser
Copy link
Owner

grosser commented Aug 16, 2024

v1.26.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@grosser @eregon @honeyankit @ovsant @mark-young-atg @VitaliySerov