[RFC] Future for `accelerator` and `devices` default values

[kaushikb11]

Discussion

Currently, the default for both the accelerator and devices flags is None. There has been discussions about updating the defaults. #10192 #10410 (comment)

We have three options to consider and vote for! Comments appreciated.

1. 😄 None sounds good for both the flags. Let's keep it the same way.
2. 🚀 Have accelerator="auto" and devices=1 as defaults. (Jax like)
3. 👀 Have accelerator="auto" and devices="auto" as defaults.

Also, to note with the last two changes, it will take priority over gpus, tpu_cores, etc.

cc @Borda @justusschock @kaushikb11 @awaelchli @rohitgr7 @akihironitta

[kaushikb11]

Why I don't agree with the last two options?

We shouldn't have defaults for devices. Option 3 is a definite no for me, as there could be multiple cases like where user wouldn't want to use all the available gpus on the node. And having this behavior as a default would be bad UX and a very strong enforcement on the user.

Option 2: Auto assigning only one device for the users doesn't sound good to me. We should just crash it if user does Trainer(accelerator="auto") and tell them to define the devices flag. For that particular reason, we support devices="auto".

And the last two options would be a breaking change.

Option 1 ftw!

[tchaton]

🚀 Have accelerator="auto" and devices=1 as defaults. (Jax like)

I believe it should be more granular:

CPU -> 1

GPUS:

• If 1 only GPU is available -> default to 1
• If multiple GPUS are available -> request devices the users

IPUS -> request

TPUS -> request

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[tchaton]

@kaushikb11

I would like to propose a different behavior.

We don't change the default, but if the user was to do Trainer(devices="auto"), this would set accelerator="auto" by the same occasion.

@awaelchli @carmocca @ananthsub Thoughts ?

[ananthsub]

the reason why I voted for accelerator="auto", devices=1 is that this should in very high likelihood be safe. we can fall back to a single-device strategy while preserving most of the behavior today.

If devices="auto", and if the accelerator supports multiple devices, this means we also need to automatically select the strategy, if one wasn't already available.

That means, depending on the device availability, setting Trainer(devices="auto") could lead to very different behaviors:

• is GPU is available? do DDP
• is TPU is available? do tpu spawn?
• is IPU available? do IPU
• is HPU available? do HPU

Some concerns with this approach:

• Where do we log/output what configuration is finally used? this is critical for debuggability and reproducibility of experiments
• automagically figuring this out will add even more initialization paths the trainer supports. this all lands up in the accelerator connector, which is already incredibly hard to read & debug. what flags take priority over other flags? how can we validate that the passed in flags actually make sense and don't fail much later, or worse, don't fail but silently produce incorrect results?

@four4fish has been going through this part of the codebase in much more detail and I'd love to hear what she thinks

[carmocca]

I would suggest that each person writes their table of preferences, these are mine:

      |       User passed         |          We set           |

Case	accelerator	devices	accelerator	devices
1	None	None	auto	1
2	xpu	None	xpu	1
3	auto	None	auto	1
4	None	N	auto	N
5	xpu	N	xpu	N
6	auto	N	auto	N
8	None	auto	auto	auto
8	xpu	auto	xpu	auto
9	auto	auto	auto	auto

This adheres to:

🚀 Have accelerator="auto" and devices=1 as defaults. (Jax like)

We don't change the default, but if the user was to do Trainer(devices="auto"), this would set accelerator="auto" by the same occasion.

Then there's the question of how many devices are used with "auto" depending on the accelerator used ("x"pu). But I think the answer to that is "as many as possible" but for CPU.

Then there are the questions of which strategy and accelerator to pick when there are multiple options, but we can easily define the priority list.