Implement atomic append operation

[ijsong]

Current Situation

Currently, Varlog's Append API writes payloads to disk by dividing them into batchlets and processing each independently. This can result in partial success/failure scenarios where some batchlets are successfully written while others fail. This leads to two main issues:

1. Users neither expect nor desire partial success/failure when appending their payloads.
2. It's challenging for Varlog to manage and communicate these partial success/failure states.

Proposed Solution

Implement an atomic append operation for the entire payload. Key changes include:

1. Remove the concept of batchlets: Write all batches in the user's payload to disk at once.
2. Utilize the existing atomic batch write functionality in Varlog's Storage layer.
3. Introduce batch length limit settings:
   • Global setting (applied to all Varlog topics)
   • Topic-specific setting (if necessary)
4. Remove the Error field from the AppendResult message type.

Expected Benefits

1. Simplified user experience: Users can rely on a clear success/failure status for their entire payload.
2. Simplified system architecture: Removing the batchlet concept reduces system complexity.
3. Simplified error handling and recovery: Atomic operations make error scenarios more straightforward to handle and recover from.
4. Potential performance improvement: Eliminating the step of dividing into batchlets may reduce overall processing time.

Challenges and Next Steps

1. Handling large payloads

   • Challenge: Potential increase in memory usage
   • Action: Analyze memory usage and research optimization strategies

2. Batch length limit settings

   • Challenge: Changes in user experience and determining optimal values
   • Action: Research and decide on optimal values for batch length limits
   • Action: Implement global and topic-specific settings

3. Maintaining backward compatibility

   • Challenge: Compatibility issues with existing systems
   • Action: Develop migration strategy and plan for phased implementation

4. Performance impact assessment

   • Challenge: Impact of atomic writes on huge batches
   • Action: Implement prototype and conduct performance tests under various conditions

5. API and client library updates

   • Action: Modify API response structure (remove Error field)
   • Action: Update client libraries and plan new version release

6. Documentation and communication

   • Action: Update API documentation
   • Action: Create and distribute user guide for the changes

Discussion Points

1. What should be the appropriate default value for the batch length limit?
2. Are there specific use cases that require topic-specific settings?
3. How can we minimize the impact of this change on systems currently using Varlog?
4. Are there additional methods to optimize the performance of atomic batch writes?

Testing Plan

• Develop unit tests for the new atomic append operation
• Conduct integration tests to ensure compatibility with existing Varlog components
• Perform stress tests with various payload sizes to assess performance and stability