Memory leak

[mloo-sp]

We have been noticing a memory leak with watchtower for long running processes.
Logs are being delivered to Cloudwatch but memory usage keeps going up.
Python version 2.7.10 and watchtower 0.3.3

Snippet of code to reproduce

import watchtower, logging, time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(str(time.time()))
wt = watchtower.CloudWatchLogHandler(use_queues=True, send_interval=60, max_batch_count=10)
logger.addHandler(wt)
logger.propagate = False

for _ in xrange(50000):
    logger.info('Junk data | {}'.format("0" * 10000))
    time.sleep(0.05)

[kageurufu]

I'm seeing this as well, and had to remove the watchtower handler to keep our servers alive

[jcerjak]

I checked the snippet above with Python 2.7.6 and latest watchtower master by using the memory_profiler.

When using queues (use_queues=True):

Without using queues (use_queues=False):

Upon investigation, it seems that the issue is related to botocore, and might be further amplified with the way threads are used in watchtower: boto/botocore#805

This seems to be confirmed by doing manual garbage collection after pushing logs to CloudWatch:

import gc

...
response = self.cwl_client.put_log_events(**kwargs)
gc.collect()

When using queues (use_queues=True) and gc:

[abend-arg]

According to this: boto/botocore#1246 (comment)

To avoid memory leaks using botocore/boto3, the idea is to create one session per thread. Currently this package creates a single session that is being used by every thread. This is causing this issue apparently. It is the recommendation from AWS as well.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-and-multiprocessing

[StevenMapes]

Is this still an issue? I was about to use this package with for some clients but if it has memory leaks still like those shown above as their services tend to run a long time without restarting and that sort of memory burn rate will drain resources too quickly

[abend-arg]

I have not experienced problems related to this, but my service is small in comparison to others. According to this opened issue the problem with botocore still persists: boto/boto3#1670

[kislyuk]

We use watchtower in production and do not observe any memory leaks. Also, watchtower does not share boto3 clients across threads unless explicitly passed a configuration that forces it to do so.

I am going to close this issue for now. When I run the reproduction script above, it stabilizes at a steady state memory consumption of 60MB. Network outages and associated retries may temporarily increase this footprint. If somebody has a specific concern, please post a complete reproduction and an explanation of why it is different from what you expect.