Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

out of memory creating hazelcast client #2823

Closed
monchalve opened this issue Apr 13, 2017 · 3 comments
Closed

out of memory creating hazelcast client #2823

monchalve opened this issue Apr 13, 2017 · 3 comments
Assignees
@ulfjack
Labels
P2 We'll consider working on this in future. (Assignee optional) type: bug
Milestone
0.6

Comments

@monchalve
Copy link

monchalve commented Apr 13, 2017 •
edited
Loading

Description of the problem / feature request / question:

The change in 0.4.4 to create a new action cache client on every invocation of the RemoteSpawnStrategy is causing out of memory errors if a Hazelcast client is used.

The patch that introduced this:
d299729

If possible, provide a minimal example to reproduce the problem:

Happens when using --hazelcast_node

Environment info

  • Operating System: Ubuntu 14.04

  • Bazel version (output of bazel info release):
    0.4.5

cc @ola-rozenfeld

Stack trace:

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at com.hazelcast.util.executor.StripedExecutor.(StripedExecutor.java:61)
at com.hazelcast.client.spi.impl.listener.ClientListenerServiceImpl.(ClientListenerServiceImpl.java:63)
at com.hazelcast.client.spi.impl.listener.ClientSmartListenerService.(ClientSmartListenerService.java:53)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.initListenerService(HazelcastClientInstanceImpl.java:291)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.(HazelcastClientInstanceImpl.java:175)
at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:84)
at com.google.devtools.build.lib.remote.ConcurrentMapFactory.createHazelcast(ConcurrentMapFactory.java:65)
at com.google.devtools.build.lib.remote.ConcurrentMapFactory.create(ConcurrentMapFactory.java:225)
at com.google.devtools.build.lib.remote.RemoteSpawnStrategy.exec(RemoteSpawnStrategy.java:198)
at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:344)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:763)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:706)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:100)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:604)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:566)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:376)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:440)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:196)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:381)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:497)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
unable to create new native thread
bazel ran out of memory and crashed.```
@ola-rozenfeld
Copy link
Contributor

ola-rozenfeld commented Apr 13, 2017 via email

@damienmg damienmg added this to the 0.6 milestone Apr 21, 2017
@damienmg damienmg added the P2 We'll consider working on this in future. (Assignee optional) label Apr 21, 2017
@damienmg
Copy link
Contributor

(cannot assign to @ola-rozenfeld because she hasn't accepted my invite to the org yet)

@ulfjack
Copy link
Contributor

ulfjack commented May 10, 2017

Technically, I fixed this in the replacement implementation, but that hasn't landed yet.

@ulfjack ulfjack self-assigned this Jun 21, 2017
hlopko pushed a commit that referenced this issue Jun 22, 2017
@ulfjack @hlopko
Only create a single per-build instance of the remote cache / executor
6268498 
Fixes #3189.
Fixes #2823.

PiperOrigin-RevId: 159699146
hlopko pushed a commit that referenced this issue Jun 22, 2017
@ulfjack @hlopko
Only create a single per-build instance of the remote cache / executor
4356c44 
Fixes #3189.
Fixes #2823.

PiperOrigin-RevId: 159699146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ulfjack ulfjack

Labels
P2 We'll consider working on this in future. (Assignee optional) type: bug
Projects
None yet
Milestone
0.6
Development

No branches or pull requests
5 participants
@monchalve @buchgr @damienmg @ulfjack @ola-rozenfeld