Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pantsd seg faults when trying to build large target #5162

Closed
youngderekm opened this issue Dec 5, 2017 · 19 comments
Closed

pantsd seg faults when trying to build large target #5162

youngderekm opened this issue Dec 5, 2017 · 19 comments
Labels
Milestone

Comments

@youngderekm
Copy link

youngderekm commented Dec 5, 2017

I'm trying to test using pantsd. I'm running 1.4.0.dev22. I enable with enable_pantsd: True. If I use ./pants compile on a reasonably small target it works as expected. If I compile our "world" target that depends on most other targets in our build, pantsd crashes. I see a segmentation fault in the pantsd.log file.

pantd.log file output below. Please let me know if I should gather any additional information:

I1205 11:16:12.112061 7653 pailgun_server.py:73] handling pailgun request: `./pants compile nucleus:world`
Fatal Python error: Segmentation fault

Thread 0x0000700014e18000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700014a15000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 315 in <genexpr>
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 340 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700014612000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Current thread 0x000070001420f000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1436 in proxy_bypass_macosx_sysconf
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1487 in proxy_bypass
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/utils.py", line 676 in should_bypass_proxies
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/utils.py", line 692 in get_environ_proxies
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 672 in merge_environment_settings
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 499 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700013e0c000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 303 in _is_owned
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 383 in notify
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 179 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700013a09000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700013606000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700013203000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700012e00000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000129fd000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000125fa000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000121f7000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011df4000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000119f1000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000115ee000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000111eb000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700010de8000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000109e5000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000f5ca000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/reporting/report.py", line 34 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000f1c7000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000dd1a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000d917000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000d514000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000d111000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000cd0e000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 154 in _read_until
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 175 in read_chunk
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 201 in iter_chunks
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 66 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000edc4000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e9c1000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 556 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 561 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 251 in map
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 69 in pings
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 237 in get_available_urls
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 273 in create_remote_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 282 in _do_create_artifact_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 155 in get_read_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 138 in read_cache_available
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 249 in artifact_cache_reads_enabled
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 342 in invalidated
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17 in __enter__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/simple_codegen_task.py", line 209 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 47 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 224 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 261 in _execute_engine
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 272 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 79 in _run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 37 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 172 in post_fork_child
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/process_manager.py", line 451 in daemonize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 128 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 58 in _run_pants
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 81 in handle
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 41 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 135 in process_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295 in _handle_request_noblock
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 280 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/pailgun_service.py", line 94 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e5be000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 339 in readBytes
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 695 in receive
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman_client.py", line 50 in stream_query
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman.py", line 177 in subscribed
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/fs_event_service.py", line 116 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e1bb000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 177 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 63 in _process_event_queue
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 101 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007fffd3e4d3c0 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 960 in join
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 259 in _run_services
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 283 in run_sync
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 329 in launch
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58 in load_and_execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69 in main
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/bin/pants", line 11 in <module>
@stuhood
Copy link
Member

stuhood commented Dec 5, 2017

The stack involving proxy_bypass_macosx_sysconf is the relevant one... it's an issue with the python requests library and forking. But it's interesting that you're seeing it in the daemon context: we haven't repro'd it there.

Thanks for the report!

@stuhood stuhood added this to the Daemon Beta milestone Dec 5, 2017
@stuhood
Copy link
Member

stuhood commented Dec 5, 2017

@youngderekm : As a workaround, you should be able to set export no_proxy="*".

@stuhood stuhood modified the milestones: Daemon Beta, 1.4.x Dec 5, 2017
@youngderekm
Copy link
Author

youngderekm commented Dec 5, 2017

I tried to restart the daemon with export no_proxy="*" first. After a few watchman timeouts and increasing watchman_startup_timeout to 60 I tried again and got what I think is a different crash, unless I just set no_proxy incorrectly (I set it in the shell I used to launch pants after killing the previous pantsd).

I1205 14:54:10.507005 18625 pailgun_server.py:73] handling pailgun request: `./pants compile nucleus:world`
Fatal Python error: Segmentation fault

Thread 0x0000700011f7f000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011b7c000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011779000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011376000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700010f73000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700010b70000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070001076d000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070001036a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000ff67000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000fb64000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000f761000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e346000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/reporting/report.py", line 34 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000df43000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000db40000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000d73d000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000d33a000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000cf37000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000cb34000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 154 in _read_until
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 175 in read_chunk
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 201 in iter_chunks
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 66 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000b719000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000b316000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000af13000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000ab10000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Current thread 0x000070000a70d000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/reporting/reporting_utils.py", line 30 in items_to_report_element
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 368 in invalidated
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17 in __enter__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/backend/jvm/tasks/jvm_platform_analysis.py", line 195 in validate_platform_dependencies
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/backend/jvm/tasks/jvm_platform_analysis.py", line 245 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 47 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 224 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 261 in _execute_engine
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 272 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 79 in _run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 37 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 172 in post_fork_child
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/process_manager.py", line 451 in daemonize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 128 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 58 in _run_pants
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 81 in handle
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 41 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 135 in process_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295 in _handle_request_noblock
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 280 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/pailgun_service.py", line 94 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000a30a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 177 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 63 in _process_event_queue
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 101 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700009f07000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 339 in readBytes
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 695 in receive
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman_client.py", line 50 in stream_query
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman.py", line 177 in subscribed
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/fs_event_service.py", line 116 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007fffd3e4d3c0 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 960 in join
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 259 in _run_services
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 283 in run_sync
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 329 in launch
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58 in load_and_execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69 in main
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/bin/pants", line 11 in <module>

@stuhood
Copy link
Member

stuhood commented Dec 5, 2017

(fyi: can put triple backticks around code to improve readability)

@stuhood
Copy link
Member

stuhood commented Dec 5, 2017

Interesting! Yes, that looks different. I don't repro with 1.4.0.dev22 on OSX 10.13 and python 2.7.12

Python 2.7.10 is pretty old... you don't happen to have a newer install around, do you?

@stuhood
Copy link
Member

stuhood commented Dec 5, 2017

Failing in items_to_report_element is very unexpected.

Is there anything unusual about the jvm platforms that are in use?

@youngderekm
Copy link
Author

I upgraded python (with brew upgrade python) to bring me to 2.7.14. Stack traces below.

About jvm platforms, I have this configuration and use java8 for everything:

default_platform: java8
platforms: {
    'java8': {'source': '8', 'target': '8', 'args': [] },
  }
I1205 15:57:01.058384 29034 pailgun_server.py:73] handling pailgun request: `./pants compile nucleus:world`


Fatal Python error: Segmentation fault

Thread 0x0000700017cd9000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000178d6000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000174d3000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000170d0000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855 in send
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893 in _send_output
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049 in endheaders
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093 in _send_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700016ccd000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855 in send
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893 in _send_output
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049 in endheaders
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093 in _send_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000168ca000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855 in send
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893 in _send_output
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049 in endheaders
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093 in _send_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Current thread 0x00007000164c7000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855 in send
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893 in _send_output
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049 in endheaders
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093 in _send_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000160c4000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700015cc1000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000158be000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000154bb000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000150b8000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700014cb5000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000148b2000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000144af000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000140ac000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700013ca9000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007000138a6000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000faa1000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/reporting/report.py", line 34 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000f69e000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 389 in _handle_results
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000f29b000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 336 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000ee98000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 327 in _handle_workers
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000ea95000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e692000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070000e28f000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 154 in _read_until
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 175 in read_chunk
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 201 in iter_chunks
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 66 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011879000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011476000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700011073000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700010c70000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070001086d000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 177 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 63 in _process_event_queue
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 101 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x000070001046a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 556 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 561 in get
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 251 in map
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 69 in pings
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 237 in get_available_urls
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 273 in create_remote_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 282 in _do_create_artifact_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 155 in get_read_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 138 in read_cache_available
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 249 in artifact_cache_reads_enabled
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 342 in invalidated
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17 in __enter__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py", line 472 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 47 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 224 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 261 in _execute_engine
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 272 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 79 in _run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 37 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 172 in post_fork_child
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/process_manager.py", line 451 in daemonize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 128 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 58 in _run_pants
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 81 in handle
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 41 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 135 in process_request
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295 in _handle_request_noblock
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 280 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/pailgun_service.py", line 94 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x0000700010067000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 339 in readBytes
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 695 in receive
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman_client.py", line 50 in stream_query
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman.py", line 177 in subscribed
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/fs_event_service.py", line 116 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763 in run
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 783 in __bootstrap

Thread 0x00007fffd3e4d3c0 (most recent call first):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 960 in join
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 259 in _run_services
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 283 in run_sync
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 329 in launch
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58 in load_and_execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69 in main
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/bin/pants", line 11 in <module>

@youngderekm
Copy link
Author

I notice 2.7.10_2 in the stacks, so I must not have done the right thing to upgrade. I'll try again.

@youngderekm
Copy link
Author

I was able to upgrade python. I'm still having a number of problems. I won't keep pasting verbose callstacks unless that's helpful, but a few things I ran into are below:

I often get a timeout watching for watch-project:

W1206 15:53:55.698817 75171 pants_daemon.py:50] SocketTimeout: timed out waiting for response, while executing (u'watch-project', '/Users/dyoung/r9/r9git')
I1206 15:53:55.910837 75171 pants_daemon.py:193] terminating pantsd service: <pants.pantsd.service.fs_event_service.FSEventService object at 0x107852f90>

I upped watchman_startup_timeout: 120 to 120, but still get this. I could up it further.

./pants kill-pantsd hangs after this timeout. I have to kill -9 pants, then I manually kill pantsd and watchman.

Sometimes I get a permission error running ./pants goals

➜  ~R9SRC git:(master) ./pants goals                               
2017-12-06T15:51:53,031: [cli] unable to talk to your watchman on /Users/dyoung/r9/r9git/.pids/watchman/watchman.sock! (Permission denied)

I get this seg fault sometimes (pasting current thread only):

Current thread 0x00007000043ab000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/native.py", line 591 in lib
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/native.py", line 662 in new_tasks
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/scheduler.py", line 84 in __init__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/scheduler.py", line 343 in __init__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/engine_initializer.py", line 166 in setup_legacy_graph
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 120 in _setup_legacy_graph_helper
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 87 in create
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/core_tasks/pantsd_kill.py", line 19 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 47 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 224 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 261 in _execute_engine
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 272 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 79 in _run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 37 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 172 in post_fork_child
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/process_manager.py", line 451 in daemonize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 128 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 58 in _run_pants
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 81 in handle
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 41 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 135 in process_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 290 in _handle_request_noblock
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 275 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/pailgun_service.py", line 94 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

I get these problems running ./pants goals and ./pants kill-pantsd.

@youngderekm
Copy link
Author

seg fault/stacks from another compile attempt (this time with watchman_startup_timeout at 300):

I1206 16:31:24.433592 78780 pailgun_server.py:73] handling pailgun request: `./pants compile nucleus:world`
Fatal Python error: Segmentation fault

Thread 0x000070000ff3a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 392 in _handle_results
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000fb37000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 338 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000f734000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 329 in _handle_workers
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000f331000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844 in send
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882 in _send_output
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038 in endheaders
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082 in _send_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Current thread 0x000070000ef2e000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844 in send
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882 in _send_output
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038 in endheaders
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082 in _send_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000eb2b000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844 in send
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882 in _send_output
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038 in endheaders
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082 in _send_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000e728000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844 in send
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882 in _send_output
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038 in endheaders
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082 in _send_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000e325000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 392 in _handle_results
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000df22000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 338 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000db1f000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 329 in _handle_workers
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000d71c000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000d319000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000cf16000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000cb13000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000c710000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000c30d000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000bf0a000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000bb07000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000a6ec000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/reporting/report.py", line 34 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000a2e9000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 392 in _handle_results
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700009ee6000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 338 in _handle_tasks
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700009ae3000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 329 in _handle_workers
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007000096e0000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007000092dd000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 138 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700008eda000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 154 in _read_until
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 175 in read_chunk
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_protocol.py", line 201 in iter_chunks
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/java/nailgun_io.py", line 66 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700002969000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700002566000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 168 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65 in _worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700002163000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 340 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 561 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 566 in get
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 253 in map
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 69 in pings
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 237 in get_available_urls
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 273 in create_remote_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 282 in _do_create_artifact_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 155 in get_read_cache
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/cache_setup.py", line 138 in read_cache_available
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 249 in artifact_cache_reads_enabled
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/task/task.py", line 342 in invalidated
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17 in __enter__
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py", line 472 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 47 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/round_engine.py", line 224 in attempt
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26 in execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 261 in _execute_engine
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 272 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 79 in _run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 37 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 172 in post_fork_child
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/process_manager.py", line 451 in daemonize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/daemon_pants_runner.py", line 128 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 58 in _run_pants
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 81 in handle
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 41 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pailgun_server.py", line 135 in process_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 290 in _handle_request_noblock
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 275 in handle_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/pailgun_service.py", line 94 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700001d60000 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 177 in get
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 63 in _process_event_queue
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/scheduler_service.py", line 101 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x000070000195d000 (most recent call first):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 339 in readBytes
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pywatchman/__init__.py", line 695 in receive
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman_client.py", line 50 in stream_query
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/watchman.py", line 177 in subscribed
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/service/fs_event_service.py", line 116 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007fffd3e4d3c0 (most recent call first):
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 359 in wait
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 951 in join
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 259 in _run_services
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 283 in run_sync
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/pantsd/pants_daemon.py", line 329 in launch
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58 in load_and_execute
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65 in run
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69 in main
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/bin/pants", line 11 in <module>

@stuhood
Copy link
Member

stuhood commented Dec 8, 2017

Is this machine memory constrained by any chance? Wonder if you're getting OOM killed.

@kwlzn
Copy link
Member

kwlzn commented Dec 21, 2017

seems odd that the segfault is happening consistently across python versions. Stu's idea of looking at resource utilization with Activity Monitor et al during the repro seems like a good first step.

are you also able to repro this against current master?

what does python2.7 -c 'import multiprocessing; print(multiprocessing.cpu_count())' say on your machine?

the Current Thread and 3 other threads point to this same stack in the pinger in at least 2 of the stacktraces:

  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/util/connection.py", line 60 in create_connection
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 141 in _new_conn
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connection.py", line 166 in connect
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844 in send
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882 in _send_output
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038 in endheaders
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082 in _send_request
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357 in _make_request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601 in urlopen
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/adapters.py", line 440 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 618 in send
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/sessions.py", line 508 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 58 in request
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/requests/api.py", line 98 in head
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 35 in _try_ping
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 45 in _get_ping_time
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/util/memo.py", line 101 in memoize
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.4.0.dev22/lib/python2.7/site-packages/pants/cache/pinger.py", line 65 in ping
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 65 in mapstar
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 113 in worker
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

which would make me want to look for abnormalities in ./pants options |grep cache |grep "read_from ="..

but it's not clear to my how that would relate to the daemon. you never see these crashes with the daemon disabled?

@youngderekm
Copy link
Author

Hi, I tested again today. Sorry for the delay.

I am now using 1.4.0.dev24. I've restarted the machine since I last test and I'm running with fewer applications running (quit Eclipse, etc. so I have more RAM free).

It's working better this time. I've done a few compiles without crashes. It's possible like you suggested it was a resource exhaustion issue.

I see that watchman in particular is taking a large amount of RAM, 2.90 GB at the moment. Looking at watchman logs, it looks like watchman is watching directories like .pants.d and .cache. Could that be a reason that it's using so much RAM? Is it possible to only watch source files?

To answer your question about CPU cores, python2.7 -c 'import multiprocessing; print(multiprocessing.cpu_count())' returns 8.

I'll keep testing with pantsd enabled so see if I get any more crashes.

@kwlzn
Copy link
Member

kwlzn commented Dec 21, 2017

ah, great. but it still crashes somewhat frequently?

watchman taking up 2.90GB sounds super abnormal to me. on our internal monorepo (very large), it clocks in at 240MB for me. I've also just bumped the watchman version in master to a build that should be more efficient fwiw.

I also wouldn't expect .pants.d to be considered by watchman (it isn't for me) - it should be ignoring all .* dirs by default.

@stuhood
Copy link
Member

stuhood commented Jan 12, 2018

@youngderekm : Hey! We've been able to begin an internal beta of the daemon as of approximately the 1.4.0.dev27 release, and we haven't had any reports of this kind of issue. Have you been able to try it out recently?

@youngderekm
Copy link
Author

Hi - I'm testing again with pants 1.4.0rc0 and I'm having better success. I've done some builds with it and haven't seen a crash or other weird behavior yet (just used for half a day again though).

watchman is using 759MB now, which is much better than it was, but still higher than I would like (developers are constrained on RAM already). Is there any way to dump out what watchman is watching? or get any other internal stats from it?

@stuhood
Copy link
Member

stuhood commented Jan 22, 2018

Thanks for the report! I haven't looked at the memory usage of watchman, but @kwlzn might be able to answer.

We do have a backgrounded task running via #4999 that might eventually remove our dependency on watchman... but certainly not in the 1.4.x timeframe.

@kwlzn
Copy link
Member

kwlzn commented Jan 23, 2018

I'm not aware of a way to dump stats from watchman, unfortunately - but 800MB still seems high to me.

I can tell you that:

  1. pantsd will only ever setup 1 subscription for "all files" under the build root.
  2. the way we use watchman under the hood guarantees a 1:1 instance of pantsd:watchman, so I wouldn't expect a pants-managed watchman instance to ever encounter more than 1 active subscription for the build root.
  3. watchman does have debug logging, but it's verbose to the point of being unusable and thus hobbled by default here:
    return {'warn': '0', 'info': '1', 'debug': '1'}.get(level, '1')
    - but you could mod this line then pass -ldebug to get more output into the .pants.d/watchman/watchman.log which may or may not help.
  4. we don't currently do a great job of plumbing ignores to watchman via pants, but if you have a surplus of dirs that should be ignored in your build root you can drop something like this into your repo:
$ cat .watchmanconfig 
{
  "ignore_dirs": ["dist", ".pants.d", ".buildcache", ".pex", ".idea"]
}```

@stuhood
Copy link
Member

stuhood commented Jan 24, 2018

Ok, it sounds like this report is not a 1.4.x blocker. Going to bump it to 1.5.x.

@stuhood stuhood modified the milestones: 1.4.x, 1.5.x Jan 24, 2018
@stuhood stuhood closed this as completed Jun 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants