the progress bar is not accurate

[djouallah]

Describe the bug

it is not a deal breaker, but very annoying, the progress bar is not accurate, when reading 2000 files, I expect daft to show the exact total count of tasks

To Reproduce

No response

Expected behavior

No response

Component(s)

Python Runner

Additional context

No response

[colin-ho]

Hi @djouallah, the number of tasks does not always equal number of files. Daft will merge files together or split them into separate tasks depending on a few factors, such as file size.

Just also as an FYI, we are in the process of replacing the Python Runner soon, so the progress bar will likely change, we will take this feedback into consideration for it. Stay tuned for that!

[jaychia]

My dream (something like this from BigQuery):

[djouallah]

sorry, that's not what I meant, I expect the number of tasks to be fixed, daft, keep increasing the number of task, there is not clarity what is the actual progress