Circular references when creating Botocore objects

[kkopachev]

I discovered recently, that even though thumbor with built-in loaders does not seem to leak (at least heavily), there are still a lot of collectible objects hanging in memory when using tc_aws for storage/loader.

Although, collectible objects does not mean there is a leak, from my understanding there is no guarantee gc will be ever run (or at least run before memkiller chimes in).

Digging dipper, with method described in thumbor/thumbor#753 (comment) I realized that even creating vanilla botocore client and then deleting it would leave some circularly-referenced objects hanging in memory. This behavior also described in boto/botocore#805

botocore team seems to think there is no issue, since all these objects collectible.

Loking at tc_aws package, I realized that code creates new botocore client every time thumbor calls loader. Which then calls Bucket, which calls tornado_botocore

As far as I can see, we could instantiate Bucket objects just once per operation (get/put/delete) and reuse.

My testing shown 1760 objects collected after each creation of Bucket and making request to get a file from S3. However, if we reuse same Bucket object over multiple requests, number of collected objects falls to 34.