Why does the progress bar not show the total steps when using iterable dataset?

[hiyyg]

Bug description

The parameters to Trainer:

max_epochs=-1, max_steps=200000

The progress bar does not show the total iters and the estimated remaining time is also nonsense.

Epoch 1: |                                                             | 43/? [00:26<00:00 ****

What version are you seeing the problem on?

master

How to reproduce the bug

The version is 2.3.3

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- PyTorch Lightning Version (e.g., 1.5.0):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):

More info

No response

[awaelchli]

@hiyyg When using iterable datasets, the length can't be determined and therefore the progress bar can't take a total. This is very intentional and not a bug.

Also note that when you set:

max_steps=200000

that doesn't mean the progress bar can show "200000" as the total, because the training in Lightning is epoch based. So 20000 just refers to the total steps taken in the entire training across epochs.

Let me know if you have any further questions.

[hiyyg]

Thanks. I ended up using my own iteration-based training loop.

[TopCoder2K]

When using iterable datasets, the length can't be determined and therefore the progress bar can't take a total. This is very intentional and not a bug.

But this is really inconvenient since I have to calculate the remaining time by myself... I anyway have to know the total number of steps in one epoch as I run validation more often than once per epoch. @awaelchli, what do you think about adding a special parameter into Trainer.__init__() for IterableDatasets? Something like steps_count_per_epoch or steps_in_one_epoch?

[Boltzmachine]

I think it would be a useful function to show the actual total steps. Otherwise one has to calculate the remaining time themself.