Gunicorn 19.7.0 raises Socket Error! #1487
Comments
|
can you try with #1483 ? |
|
Hello @benoitc, Thanks for your prompt reply. I have checked out Gunicorn at the specified branch and commit in #1483 and installed it into the virtual environment. Unfortunately, the same error keeps reappearing. |
|
Hi, gevent developer here. We recently had a somewhat similar issue come up with Python 2 sockets and SOCK_CLOEXEC that makes me wonder if there might be something going on at the gevent level. I primarily run on OS X and I haven't been able to reproduce this yet in a simplified example (not using gunicorn)...but I haven't tried that hard yet :) A few questions that might help narrow things down: Correct me if I'm wrong, but you're using the default gunicorn listener configuration, so TCP, and not unix sockets? You look to be on OS X 10.12 sierra? Did you build gevent from source (looks that way) or did you install the binary egg from PyPI? |
|
I also can't reproduce using gunicorn 19.7.0 and the simplest possible app (on OS X 10.12, Python 3.6 from macports, using the gevent wheel from PyPI: EDIT: I also cannot reproduce using the python.org installed version of 3.6.0. |
|
Hello @jamadden, Thanks for trying to help. The error doesn't appear right after starting the app. It actually appears 5-10 times (probably for 5-10 requests) when I test the app with Regarding your questions, I am using Gunicorn with the standard configuration, I am binding it to I don't know what is causing the problem to be honest but I can reproduce it every single time I test the app with
The above 3 commands always reproduced the problem on Python 3.6.0. Also, I mentioned in my first comment that the issue disappears when downgrading to Gunicorn==19.6.0, but it still shows up as long as my Python's version is 3.6.0. The issue only disappeared so far with the combination: Python==3.5.3 + Gunciron==19.6.0. I hope the above information helped. |
|
The issue looks to be even worse! I have just ran a longer test (5 minutes) with Here is how I am currently running Gunicorn: Here is what happens when I grep the log for errors: All these 9 cases have the same traceback, here is a sample: At this point, I am clueless to what is causing the problem. Nothing has changed in the list of dependencies. EDIT: copy-paste the current command. |
|
I had already tried doing concurrency tests with Can you produce a problem using the simple app? If so, that tells us its system dependent somehow. If not, that tells us its related to the code in the app you're running in some way and we'd need to simplify that to pin down the issue (in particular, I would be concerned about C extensions or CFFI code). |
|
I wonder if you're seeing this behaviour, which basically comes down to timing/race-type conditions. The difference in our test machines, and possibly in the amount of data being sent, might explain the reproduction issues, as well as explaining why different versions of Python and gunicorn behave differently. |
|
Hello @jamadden, I didn't test the simple app yet on my Mac machine, but I have tested it on my Linux (Lenovo) machine with Ubuntu 14.04. The error didn't show up at all. Here are the test results: Operating System: Application: #!/usr/bin/env python3
# app.py
def app(environ, start_response):
start_response("200 OK", [('Content-Type', 'text/plain')])
return [b"This is some text to stream"]Gunicorn Command: gunicorn -k gevent -w 17 ignored_dir.app:app > test_app/log_file.log 2>&1Python 3.5.3 + Gunicorn 19.6.0: The log file contained no errors and no critical messages. Python 3.5.3 + Gunicorn 19.7.0: The log file contained no errors and no critical messages. Python 3.6.0 + Gunicorn 19.7.0: The log file contained no errors and no critical messages. Python 3.6.0 + Gunicorn 19.7.1: The log file contained no errors and no critical messages. After all the tests above have resulted in good news, I went ahead and tested my own Flask application on the same machine with Python 3.6.0 and Gunicorn 19.7.1 and the error didn't appear, not even once! I tested the app twice, once with a standard Gunicorn command, and then with my own Gunicorn configuration, here are the commands: Gunicorn Commands:
Wrk command: Same command was used twice, sample result: $ wrk -c 1000 -t 180 -d 300 http://127.0.0.1:8000/api/v1.0/users
Running 5m test @ http://127.0.0.1:8000/api/v1.0/users
180 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 67.27ms 55.75ms 1.99s 98.86%
Req/Sec 15.88 6.69 70.00 92.42%
65618 requests in 5.00m, 280.06MB read
Socket errors: connect 59, read 6747, write 0, timeout 910
Non-2xx or 3xx responses: 10
Requests/sec: 218.65
Transfer/sec: 0.93MBSo apparently the issue, at least so far, only happens on macOS Sierra 10.12.1. I will re-run the same tests this evening on my Mac machine and copy-paste the tests results here as well. |
That makes sense to me, based on the research I've done. It appears to be mac-specific, quite probably even mac version specific and possibly machine specific too. |
|
Hello @jamadden, I have tested the simple app on my Mac machine once with standard configuration and a second time with my app's Gunicorn configuration and the error appeared. I have used the following I also tested my app with Python 3.6.0 + Gunicorn 19.7.1 and the error appeared. The weird thing though is that I tested the index page of my app which only returns an HTML static page, no database or 3rd services being accessed whatsoever, and to be 100% precise, the index page checks a cookie header and redirects to another static page for user login, but still no additional database or network access. The same error ( I tried testing one of my API endpoints on Python 3.6.0 + Gunicorn 19.7.1 and It looks like a Mac issue to me. What do you guys think? CC @benoitc. P.S. here is my Gunicorn config file in case you guys would like to know about all the details: import multiprocessing as mp
bind = "127.0.0.1:5000"
worker_class = "gevent"
workers = (mp.cpu_count() * 2) + 1
daemon = False
preload_app = False
pidfile = PID_FILE_PATH
proc_name = "sa7beh_web_server"
backlog = 2048
worker_connections = 1024
max_requests = 2048
max_requests_jitter = 100
timeout = 30
keepalive = 2
logger_class = "gunicorn.glogging.Logger"
loglevel = "info"
accesslog = "-"
errorlog = "-"
log_file = LOG_FILE_PATH
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" "response_time=%(L)s seconds"' |
|
I'm also getting
|
|
Alright, seems like everyone experiences this as a mac error. If anyone gives me a repro repo I can try it on 10.13. I'm not sure there's any action to take for Gunicorn. |
|
Seems to be an unhandled error returned by the OSX kernel. Taken from here:
It seems this error is returned by OSX in a call to Presumably this needs to be fixed in a low-level python library? |
|
|
|
Any suggestions for what to do here? |
|
macOS still got this error sometime, which is really weired. macOS version 10.13.3 |
|
Also seeing this pop up for me on macOS version 10.12.6 |
|
Since I couldn't find it on the python bugtracker, I filed it here @tilgovi It's been a long time since I looked at it, but since it seems to be intermittent and probably some kind of race condition, catch the exception as low as possible in gunicorn and try again, possibly adding a small delay? You could even make the code mac-specific and comment it as a workaround until python/Apple fix it? Alternatively, since the application will probably retry anyway, do nothing and mark as "known issue"? |
|
I know it's been a while, but I'm going to hop on the bandwagon. I run Siege to test my servers, using aiohttp, and sometimes it's behind guincorn. It appears to only happen (in my case) when I ctrl+C Siege. So I'd personally agree with the race condition idea? Or maybe it's to do with closed sockets? But if anybody needs to test ideas, I'm more than happy to get in contact with whomever to help repro bugs. But, with my software, I can consistently produce this. I'm running OSX 10.12.6, mid-2015 15" MBP. Sorry if I'm late to the party or compounding something already known, or if I'm even being useless. Let me know if there's anything I can do to help. |
|
The moment I stopped socketIO, the error is gone :-) I believe the problem is related with socketIO. |
|
[ERROR] Socket error processing request |
|
FYI - I came here while searching about this same error that I am seeing with with version 3.0.6 of the Django Development server on MacOS, so this error does not seem specific to gunicorn. Here is the stack trace I'm seeing: Just realized that I should have noted my MacOS Version: 10.14.6 (18G4032) |
|
I can +1 brian-farrell's post - older django and python versions, but pretty identical stacktrace. OSX version 10.15.2 (19C57). |
|
I am still seeing this error. Has there been any resolution yet? |
|
I am seeing this error on OSX version 10.14. In my use case the error relates to GCS emulator. I can confirm that in Linux the error does not occur in the same scenario. Is there any update or fix on that? |
|
I'm not sure there's anything we can do here. We have this discussion for anyone who searches for the issue, but it seems to be an issue with macOS. I'm going to close this, but if anyone has suggestions for some action we should take, please open an issue or a PR to address this. |
* Rename internal "args" variable in parseargs * Add SSL parameters and combination validation * Add certfile & keyfile validation (check they exist) * Invert "--requiretls" to "--no-requiretls" * Assert default values * Implement SSL Context injection for STARTTLS and SMTPS * Implement test for "TLS Required" * Update CLI docs * Properly generate man page * Version Up + NEWS * Run Bandit as part of testing * Silence Bandit complaints * Update README.rst * Build manpage as well in GA (detect errors) * Tidying up * Reduce housekeep.py superclean clutter
* MacOS sometimes raises OSError instead of ConnectionError Refs: - racitup/static-ranges#1 - benoitc/gunicorn#1487 - http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/
## Change shared ctypes to MP.Queue PyPy has problems with shared ctypes Value, especially on Windows. Not sure why. This resulted in flakey pypy3-* tests. Converting to MP.Queue might be slightly slower, but it's much more stable/deterministic. And since this change impacts only 3 tests out of several hundreds, it's worth it for the expansion of Windows test scope. ## Activate pypy3 testing in tox.ini ## Specify non-"win32" platforms for "static" This enables us to add "static" into tox/envlist, for easier invocation. ## Change "_dynamic" to "_dump" ## Make ENV dump ".py" So we can leverage syntax-highlighting when checking ## Also delete temporary "parallel" coverage files ## Move coverage/diffcov files Temporary/intermediate files go to _dump Reports go to htmlcov ## Exclude _dump from pytype ## Prefix ENV dump file with "ENV = \" To reduce IDE protests ## Extend watched process delay by 50% Exact same with AUTOSTOP_DELAY makes the test flakey. ## Up version to 1.4.0a4 and NEWS.rst ## Replace ConnectionError with OSError (for MacOS) Refs: - racitup/static-ranges#1 - benoitc/gunicorn#1487 - http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ ## Update README.rst - Now pypy3 is no longer a stepchild. - Notes on which testenvs work/don't work on Cygwin.
Hello guys,
I have been testing the performance of an application built with Flask, SQLAlchemy and gevent on Gunicorn==19.7.0 with
wrkand it keeps raising Socket Errors, the issue disappears by downgrading to Gunicorn==19.6.0, here is the full stack trace:Full list of pip dependencies:
Commands:
gunicorn -k gevent -w 18 sa7beh.app:appwrk -c 3600 -t 2000 -d 60 <API URL>Let me know if you need more information.
The text was updated successfully, but these errors were encountered: