Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nunit run flask application in non stable way #288

Closed
KursLabIgor opened this issue Aug 7, 2019 · 4 comments
Closed

nunit run flask application in non stable way #288

KursLabIgor opened this issue Aug 7, 2019 · 4 comments

Comments

@KursLabIgor
Copy link

KursLabIgor commented Aug 7, 2019 •
edited
Loading

I use docker. AWS FARGATE launch type.
I cant change any kernel settings.

FROM python:3.7-alpine

RUN mkdir /cb_api
WORKDIR /cb_api


COPY . /cb_api/

ENV PYTHONPATH "${PYTHONPATH}:/cb_api"


RUN \
 apk add --no-cache linux-headers bash curl unit-openrc unit unit-python3 && \
 apk add --no-cache postgresql-libs && \
 apk add --no-cache --virtual .build-deps gcc musl-dev postgresql-dev && \
 python3 -m pip install -r requirements.txt --no-cache-dir && \
 apk --purge del .build-deps
RUN ln -sf /dev/stdout /var/log/unit.log
EXPOSE 80

I have python 3.7 flask application.
my unit config:

{
    "listeners": {
        "*:80": {
            "pass": "applications/cb_api"
        }
    },
    "applications": {
        "cb_api": {
            "type": "python 3",
            "processes": 10,
            "working_directory": "/cb_api/api_gateway",
            "module": "wsgi"
        }
    }
}

In logs i see such strings:

Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 Traceback (most recent call last):
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/wsgi.py", line 1, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from api_gateway.app import create_app
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/app.py", line 4, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from api_gateway import auth, api
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/auth/__init__.py", line 1, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from . import views, controller, admin
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/auth/views.py", line 11, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from api_gateway.models import User
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/models/__init__.py", line 1, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from .user import User
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/models/user.py", line 1, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from api_gateway.extensions import db, pwd_context
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/cb_api/api_gateway/extensions.py", line 7, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from celery import Celery
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/local.py", line 509, in __getattr__
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     module = __import__(self._object_origins[name], None, None, [name])
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/app/__init__.py", line 5, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from celery import _state
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/_state.py", line 17, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from celery.utils.threads import LocalStack
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/utils/__init__.py", line 16, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from .log import LOG_LEVELS     # noqa
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/utils/log.py", line 20, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from .term import colored
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "/usr/local/lib/python3.7/site-packages/celery/utils/term.py", line 13, in <module>
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50     from celery.platforms import isatty
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap_external>", line 724, in exec_module
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap_external>", line 859, in get_code
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50   File "<frozen importlib._bootstrap_external>", line 917, in get_data
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 OSError: [Errno 14] Bad address
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 notice 2019/08/07 13:27:14 [notice] 9#9 process 556 exited with code 1
Aug 7 16:27:15 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:15 [alert] 9#9 process 542 exited on signal 9
Aug 7 16:27:15 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:15 [info] 548#548 "cb_api" application started
Aug 7 16:27:18 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:18 [info] 558#558 "cb_api" application started
Aug 7 16:27:19 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:19 [alert] 9#9 process 544 exited on signal 9
Aug 7 16:27:19 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:19 [info] 550#550 "cb_api" application started
Aug 7 16:27:22 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:22 [alert] 9#9 process 550 exited on signal 9
Aug 7 16:27:22 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:22 [info] 560#560 "cb_api" application started
Aug 7 16:27:23 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:23 [alert] 9#9 process 546 exited on signal 9
Aug 7 16:27:23 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:23 [info] 552#552 "cb_api" application started
Aug 7 16:27:26 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:26 [alert] 9#9 process 554 exited on signal 9
Aug 7 16:27:26 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:26 [info] 562#562 "cb_api" application started
Aug 7 16:27:26 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:26 [alert] 9#9 process 550 exited on signal 9
Aug 7 16:27:26 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:26 [info] 554#554 "cb_api" application started
Aug 7 16:27:30 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:30 [alert] 9#9 process 560 exited on signal 9
Aug 7 16:27:30 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:30 [info] 564#564 "cb_api" application started
Aug 7 16:27:30 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:30 [alert] 9#9 process 548 exited on signal 9
Aug 7 16:27:30 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:30 [info] 556#556 "cb_api" application started
Aug 7 16:27:34 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:34 [alert] 9#9 process 552 exited on signal 9
Aug 7 16:27:34 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:34 [info] 558#558 "cb_api" application started
Aug 7 16:27:34 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:34 [alert] 9#9 process 558 exited on signal 9
Aug 7 16:27:34 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:34 [info] 566#566 "cb_api" application started
Aug 7 16:27:38 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:38 [alert] 9#9 process 556 exited on signal 9
Aug 7 16:27:38 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:38 [info] 560#560 "cb_api" application started
Aug 7 16:27:38 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:38 [alert] 9#9 process 562 exited on signal 9
Aug 7 16:27:38 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:38 [info] 568#568 "cb_api" application started
Aug 7 16:27:42 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:42 [alert] 9#9 process 558 exited on signal 9
Aug 7 16:27:42 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:42 [info] 562#562 "cb_api" application started
Aug 7 16:27:42 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:42 [alert] 9#9 process 564 exited on signal 9
Aug 7 16:27:42 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:42 [info] 570#570 "cb_api" application started
Aug 7 16:27:46 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:46 [alert] 9#9 process 534 exited on signal 9
Aug 7 16:27:46 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:46 [info] 564#564 "cb_api" application started
Aug 7 16:27:46 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:46 [alert] 9#9 process 568 exited on signal 9
Aug 7 16:27:46 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:46 [info] 572#572 "cb_api" application started
Aug 7 16:27:50 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:50 [alert] 9#9 process 560 exited on signal 9
Aug 7 16:27:50 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 2019/08/07 13:27:50 [info] 566#566 "cb_api" application started
Aug 7 16:27:51 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:51 [alert] 9#9 process 566 exited on signal 9
Aug 7 16:27:51 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 info 2019/08/07 13:27:51 [info] 574#574 "cb_api" application started
Aug 7 16:27:54 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:54 [alert] 9#9 process 562 exited on signal 9
Aug 7 16:27:54 Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 info 201
Aug 7 16:26:03 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:26:03 [alert] 13#35 accept4(11) failed (23: Too many open files in system), new connections are not accepted within 1s
Aug 7 16:26:26 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:26:26 [alert] 13#34 sendmsg(16, -1, 2) failed (32: Broken pipe)
Aug 7 16:27:06 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:27:06 [alert] 13#34 sendmsg(78, -1, 2) failed (9: Bad file descriptor)
Aug 7 16:27:14 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:27:11 [alert] 556#556 Python failed to import module "wsgi"
Aug 7 16:29:21 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:29:19 [alert] 610#610 Python failed to import module "wsgi"
Aug 7 16:30:15 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:30:12 [alert] 638#638 Python failed to import module "wsgi"
Aug 7 16:30:35 -Dev-Api-Service-Log-Group 0994744017f846e1a7a8503eb4beff24 alert 2019/08/07 13:30:33 [alert] 648#648 Python failed to import module "wsgi"
Aug 7 16:30:54 -Dev-Api-Service-Log-Group 973d0036f80a4e17a98e4877da520c50 alert 2019/08/07 13:30:54 [alert] 13#34 *1085 shm_open(/unit.13.e24b6b18) failed (28: No space left on device)

Not clear what is going on.

Seems like unit just stop and close application.
@VBart
Copy link
Contributor

VBart commented Aug 7, 2019

It seems that all bad things are started from:
2019/08/07 13:26:03 [alert] 13#35 accept4(11) failed (23: Too many open files in system), new connections are not accepted within 1s

It looks like that you've run out of available resources inside the container, notably file descriptors.

Do you also run nginx in the same container or just Unit?

@KursLabIgor
Copy link
Author

It seems that all bad things are started from:
2019/08/07 13:26:03 [alert] 13#35 accept4(11) failed (23: Too many open files in system), new connections are not accepted within 1s

It looks like that you've run out of available resources inside the container, notably file descriptors.

Do you also run nginx in the same container or just Unit?
Sorry, I attached old docker file.
Actually i dont install nginx on latest version of docker file.
Maybe i have to many processes ?

@VBart
Copy link
Contributor

VBart commented Aug 8, 2019

Yes, reducing number of processes can help.

@KursLabIgor
Copy link
Author

yep, reduce number for workers. solved problem.

@akaihola akaihola mentioned this issue Jul 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@VBart @KursLabIgor