Handling transaction log directory in AWS Lambda

[alisheykhi]

When using the Python library in a Lambda function, the high volume of transaction logs causes a significant exponential decrease in writing speed to the table (S3). To address this issue, I used the create_checkpoint function after each batch, which improved the speed. However, a new problem has arisen due to the proliferation of checkpoint files, leading to exponential growth in the directory's size. Furthermore, using compact and vacuum with each batch does not appear to be a practical solution. Additionally, while the vacuum function helps reduce the number of files in the data directory, it does not clean up the log directory. so I implemented a custom function for deleting expired checkpoints and logs!

My question is, what is the best practice for this purpose?

[rtyler]

When you say high volume, how high of a volume of writes coming in and are they little writes? I have trigger a number of Lambdas to perform various tasks, but I do route everything through SQS by default, which leads to my Lambdas being triggered with batches of writes to execute.

I'm guessing you're also seeing a slow down because Lambda invocations are competing for dynamodb locks?

[alisheykhi]

I use Kinesis streams for controlling my batches, and the write volume for each batch is approximately 1.5 MB (for 150 events). Additionally, 40 instances of my Lambdas are running concurrently, and I don't observe any significant idle time for acquiring locks.

In fact, when my Lambda starts writing to an empty table, it works perfectly and is very fast. However, over time, the writing process becomes slower, which is due to the accumulation of extensive logs for transactions.

So, if I compact the table and then vacuum it, and after a checkpoint, delete all transaction logs, the writing speed in Lambdas increases again.

[rtyler]

@alisheykhi are checkpoints being created by every Lambda after they write, or only by one, or something different? I would also be curious to know how large those checkpoint files are during normal operations? I'm wondering if the checkpoint files are potentially becoming very large or if they're itty bitty and therefore not serving much of an optimization since so many new commits are coming into the table

[alisheykhi]

Initially, I used to run checkpoint operations after every write operation(in each lambda), resulting in a proliferation of checkpoint files. However, subsequent write operations became more efficient. Later, I introduced a new lambda function that generates a checkpoint every few minutes and subsequently handles cleanup (custom cleanup function!!).

Regarding the checkpoint file sizes, they are largely dependent on the number of write operations. In my case, I can estimate that, on average, each file is around 150MB in size. It's evident that after optimizing the table, the checkpoint file sizes should decrease, but let's assume I can only optimize the table once per day. Consequently, the number of checkpoint files grows exponentially over time!