Commits on Aug 10, 2024

1. [fix](routine load) reset Kafka progress cache when routine load job … …

   …topic change (apache#38474)
   
   When change routine load job topic from test_topic_before to
   test_topic_after by
   ```
   ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after");
   ```
   (test_topic_before has 5 rows and test_topic_after has 1 rows)
   
   Exception happened, which cannot consume any data:
   ```
   2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
   fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
   ```
   
   It is necessary to reset Kafka progress cache when routine load job
   topic change.

   

   sollhui committed Aug 10, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 4dbff58
   • Browse repository at this point

   Copy the full SHA

   4dbff58 View commit details

   Browse the repository at this point in the history