Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak #53

Open
tonyliu7870 opened this issue Aug 19, 2021 · 0 comments
Open

Memory leak #53

tonyliu7870 opened this issue Aug 19, 2021 · 0 comments

Comments

@tonyliu7870
Copy link

tonyliu7870 commented Aug 19, 2021 •
edited
Loading

Memory leak detected when running the nginx for a long time. Reproducible by introducing a rapid accounting_interval e.g. 1s
with ~0.9KB per report period per nginx worker

reading file "heaptrack.nginx.4144.0917.gz" - please wait, this might take some time...
reading diff file "heaptrack.nginx.4144.0742.gz" - please wait, this might take some time...
Debuggee command was: Debuggee command was: ./sbin/nginx -c /myworkspace/nginx/sbin/nginx.conf./sbin/nginx -c /myworkspace/nginx/sbin/nginx.conf

finished reading file, now analyzing data:

MOST CALLS TO ALLOCATION FUNCTIONS
144733 calls to allocation functions with 2.63MB peak consumption from
ngx_memalign
  at src/os/unix/ngx_alloc.c:57
  in /myworkspace/nginx/sbin/nginx
48191 calls with 512B peak consumption from:
    ngx_create_pool
      at src/core/ngx_palloc.c:23
      in /myworkspace/nginx/sbin/nginx
    ngx_event_accept
      at src/event/ngx_event_accept.c:156
      in /myworkspace/nginx/sbin/nginx
    .
    .
    .
    .
    .


PEAK MEMORY CONSUMERS

WARNING - the data below is not an accurate calcuation of the total peak consumption and can easily be wrong.
 For an accurate overview, disable backtrace merging.
2.63MB peak memory consumed over 144733 calls from
ngx_memalign
  at src/os/unix/ngx_alloc.c:57
  in /myworkspace/nginx/sbin/nginx
999.42KB consumed over 61 calls from:
    ngx_palloc_block
      at src/core/ngx_palloc.c:186
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:127
      in /myworkspace/nginx/sbin/nginx
    .
    .
    .
    .
    .

MEMORY LEAKS
2.62MB leaked over 144733 calls from
ngx_memalign
  at src/os/unix/ngx_alloc.c:57
  in /myworkspace/nginx/sbin/nginx
999.42KB leaked over 61 calls from:
    ngx_palloc_block
      at src/core/ngx_palloc.c:186
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:127
      in /myworkspace/nginx/sbin/nginx
    ngx_pcalloc
      at src/core/ngx_palloc.c:302
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_metrics_init
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_period_metrics.c:23
      in /myworkspace/nginx/sbin/nginx
    ngx_http_accounting_request_handler
      at ../traffic-accounting-nginx-module-43c9b9d/src/http/ngx_http_accounting_module.c:249
      in /myworkspace/nginx/sbin/nginx
    ngx_http_log_request
      at src/http/ngx_http_request.c:3674
      in /myworkspace/nginx/sbin/nginx
    ngx_http_free_request
      at src/http/ngx_http_request.c:3620
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_connection
      at src/http/ngx_http_request.c:2720
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_request
      at src/http/ngx_http_request.c:2612
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_rewrite_phase
      at src/http/ngx_http_core_module.c:925
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_run_phases
      at src/http/ngx_http_core_module.c:858
      in /myworkspace/nginx/sbin/nginx
    ngx_http_handler
      at src/http/ngx_http_core_module.c:841
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request
      at src/http/ngx_http_request.c:2055
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_headers
      at src/http/ngx_http_request.c:1480
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_line
      at src/http/ngx_http_request.c:1151
      in /myworkspace/nginx/sbin/nginx
    ngx_http_wait_request_handler
      at src/http/ngx_http_request.c:500
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
983.04KB leaked over 60 calls from:
    ngx_palloc_block
      at src/core/ngx_palloc.c:186
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:127
      in /myworkspace/nginx/sbin/nginx
    ngx_pcalloc
      at src/core/ngx_palloc.c:302
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_metrics_init
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_period_metrics.c:16
      in /myworkspace/nginx/sbin/nginx
    ngx_http_accounting_request_handler
      at ../traffic-accounting-nginx-module-43c9b9d/src/http/ngx_http_accounting_module.c:249
      in /myworkspace/nginx/sbin/nginx
    ngx_http_log_request
      at src/http/ngx_http_request.c:3674
      in /myworkspace/nginx/sbin/nginx
    ngx_http_free_request
      at src/http/ngx_http_request.c:3620
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_connection
      at src/http/ngx_http_request.c:2720
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_request
      at src/http/ngx_http_request.c:2612
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_rewrite_phase
      at src/http/ngx_http_core_module.c:925
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_run_phases
      at src/http/ngx_http_core_module.c:858
      in /myworkspace/nginx/sbin/nginx
    ngx_http_handler
      at src/http/ngx_http_core_module.c:841
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request
      at src/http/ngx_http_request.c:2055
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_headers
      at src/http/ngx_http_request.c:1480
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_line
      at src/http/ngx_http_request.c:1151
      in /myworkspace/nginx/sbin/nginx
    ngx_http_wait_request_handler
      at src/http/ngx_http_request.c:500
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
360.45KB leaked over 22 calls from:
    ngx_palloc_block
      at src/core/ngx_palloc.c:186
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:127
      in /myworkspace/nginx/sbin/nginx
    ngx_pcalloc
      at src/core/ngx_palloc.c:302
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_period_insert
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_period_metrics.c:46
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_period_fetch_metrics
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_period_metrics.c:135
      in /myworkspace/nginx/sbin/nginx
    ngx_http_accounting_request_handler
      at ../traffic-accounting-nginx-module-43c9b9d/src/http/ngx_http_accounting_module.c:246
      in /myworkspace/nginx/sbin/nginx
    ngx_http_log_request
      at src/http/ngx_http_request.c:3674
      in /myworkspace/nginx/sbin/nginx
    ngx_http_free_request
      at src/http/ngx_http_request.c:3620
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_connection
      at src/http/ngx_http_request.c:2720
      in /myworkspace/nginx/sbin/nginx
    ngx_http_finalize_request
      at src/http/ngx_http_request.c:2612
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_rewrite_phase
      at src/http/ngx_http_core_module.c:925
      in /myworkspace/nginx/sbin/nginx
    ngx_http_core_run_phases
      at src/http/ngx_http_core_module.c:858
      in /myworkspace/nginx/sbin/nginx
    ngx_http_handler
      at src/http/ngx_http_core_module.c:841
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request
      at src/http/ngx_http_request.c:2055
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_headers
      at src/http/ngx_http_request.c:1480
      in /myworkspace/nginx/sbin/nginx
    ngx_http_process_request_line
      at src/http/ngx_http_request.c:1151
      in /myworkspace/nginx/sbin/nginx
    ngx_http_wait_request_handler
      at src/http/ngx_http_request.c:500
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
278.53KB leaked over 17 calls from:
    ngx_palloc_block
      at src/core/ngx_palloc.c:186
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:127
      in /myworkspace/nginx/sbin/nginx
    ngx_pcalloc
      at src/core/ngx_palloc.c:302
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_period_create
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_module.c:16
      in /myworkspace/nginx/sbin/nginx
    ngx_traffic_accounting_period_rotate
      at ../traffic-accounting-nginx-module-43c9b9d/src/ngx_traffic_accounting_module.c:37
      in /myworkspace/nginx/sbin/nginx
    worker_process_alarm_handler
      at ../traffic-accounting-nginx-module-43c9b9d/src/http/ngx_http_accounting_module.c:209
      in /myworkspace/nginx/sbin/nginx
    ngx_event_expire_timers
      at src/event/ngx_event_timer.c:94
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:256
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
512B leaked over 48191 calls from:
    ngx_create_pool
      at src/core/ngx_palloc.c:23
      in /myworkspace/nginx/sbin/nginx
    ngx_event_accept
      at src/event/ngx_event_accept.c:156
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
  and 0B from 2 other places

-2.16KB leaked over 0 calls from
ngx_strerror_init
  at src/os/unix/ngx_errno.c:60
  in /myworkspace/nginx/sbin/nginx
-2.16KB leaked over 0 calls from:
    main
      at src/core/nginx.c:206
      in /myworkspace/nginx/sbin/nginx

1.02KB leaked over 96382 calls from
ngx_alloc
  at src/os/unix/ngx_alloc.c:22
  in /myworkspace/nginx/sbin/nginx
1.02KB leaked over 48191 calls from:
    ngx_palloc_large
      at src/core/ngx_palloc.c:220
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:131
      in /myworkspace/nginx/sbin/nginx
    ngx_http_keepalive_handler
      at src/http/ngx_http_request.c:3261
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx
0B leaked over 48191 calls from:
    ngx_palloc_large
      at src/core/ngx_palloc.c:220
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:131
      in /myworkspace/nginx/sbin/nginx
    ngx_create_temp_buf
      at src/core/ngx_buf.c:22
      in /myworkspace/nginx/sbin/nginx
    ngx_http_wait_request_handler
      at src/http/ngx_http_request.c:408
      in /myworkspace/nginx/sbin/nginx
    ngx_epoll_process_events
      at src/event/modules/ngx_epoll_module.c:902
      in /myworkspace/nginx/sbin/nginx
    ngx_process_events_and_timers
      at src/event/ngx_event.c:242
      in /myworkspace/nginx/sbin/nginx
    ngx_single_process_cycle
      at src/os/unix/ngx_process_cycle.c:310
      in /myworkspace/nginx/sbin/nginx
    main
      at src/core/nginx.c:379
      in /myworkspace/nginx/sbin/nginx


MOST TEMPORARY ALLOCATIONS
48190 temporary allocations of 96382 allocations in total (50.00%) from
ngx_alloc
  at src/os/unix/ngx_alloc.c:22
  in /myworkspace/nginx/sbin/nginx
48190 temporary allocations of 48191 allocations in total (100.00%) from:
    ngx_palloc_large
      at src/core/ngx_palloc.c:220
      in /myworkspace/nginx/sbin/nginx
    ngx_palloc
      at src/core/ngx_palloc.c:131
      in /myworkspace/nginx/sbin/nginx
    .
    .
    .
    .
    .


total runtime: 5998.42s.
bytes allocated in total (ignoring deallocations): 520.77MB (86.82KB/s)
calls to allocation functions: 241115 (40/s)
temporary memory allocations: 48190 (8/s)
peak heap memory consumption: 2.63MB
peak RSS (including heaptrack overhead): 0B
total memory leaked: 2.62MB

It is caused by the accounting data allocating memory from nginx pool memory of conf. However, the small memory pool will only be released after the whole memory pool releases. While the conf memory pool is never release.
It caused the conf memory pool keep expanding and introduce a memory leak
tonyliu7870 pushed a commit to tonyliu7870/traffic-accounting-nginx-module that referenced this issue Aug 19, 2021
use ngx_calloc to replace ngx_pcalloc to avoid memory leak
900fcee 
Lax#53
@tonyliu7870 tonyliu7870 mentioned this issue Aug 19, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tonyliu7870