Work with DeepSpeed for large scale training

[kuizhiqing]

DeepSpeed is an excellent framework for training LLMs on a large scale, while the mpi-operator is the ideal tool to facilitate this within the Kubernetes ecosystem.

I'm planning to submit a series of PRs to make this project more ready-to-use for very large scale training with the DeepSpeed/mpi-style training framework.

The upcoming features may include the following modifications:

Support for IP-style hostfile
This is for performance efficiency and to prevent the environment variable length from exceeding its limit when using svc for those who wish to wrap it into an environment variable.

Support for fault tolerance and elasticity
This is a quasi-fault tolerance since NCCL communication must always be recreated when an error occurs. However, it's still worth implementing because recreating pods can be costly on a very large scale.

Configuration decoupling
There are some requirements that are currently left to the docker image maker to handle, such as ssh_config and sshd_config. Perhaps the operator can manage all of these.

There are also some minor changes under consideration. Please feel free to share your thoughts on this topic.

[alculquicondor]

cc @aojea @vsoch

[alculquicondor]

Support for IP-style hostfile

How do you plan to obtain the IPs?
The current model uses a ConfigMap for the hostfile, which would have pretty decent size limits. Also, it's all transparent to the user.

[kuizhiqing]

Support for IP-style hostfile

How do you plan to obtain the IPs? The current model uses a ConfigMap for the hostfile, which would have pretty decent size limits. Also, it's all transparent to the user.

The idea is to update the hostfile with IP after all containers(not pod) are ready.

Files have no length limit, environ variable does, AFAIK, one may want to convert the nodes list into env or args to make execution, while using svc may exceed the length limit in very large scale.

Two main reason to do that, with very large scale job, svc is not performance efficient and it may not stable enough since it depend on coreDNS or alternatives.

BTW, CM for hostfile is a good idea since it support runtime update which makes it possible to support quasi-elastic.

[vsoch]

That was a design I chose for the Flux Operator before switching to a headless service, would be interesting to see how it works for you! I can point you to old code if you want an example of how I did that - it was basically designed exactly as you stated.

I am not familiar with DeepSpeed, but is ssh bootstrap the best / only way available?

[aojea]

The current model uses a ConfigMap for the hostfile

does this means it creates a configmap with the etc hosts content and mounts it with a volume in /etc/hosts?

Files have no length limit, environ variable does, AFAIK, one may want to convert the nodes list into env or args to make execution, while using svc may exceed the length limit in very large scale.

What environment variables limitation and length limit are you referring to here?

With a Headless service each pod has a well known record https://github.com/kubernetes/dns/blob/master/docs/specification.md#241---aaaaa-records

[alculquicondor]

Ok, just to get rid of unnecessary feature requests, let's forget about passing a hostfile via an environment variable.

Two main reason to do that, with very large scale job, svc is not performance efficient and it may not stable enough since it depend on coreDNS or alternatives.

But instead, you will have to wait for all worker pods to be ready before being able to create the launcher Pod. This means that you will have to wait for container images to download. That time will probably be more significant than DNS name resolution.

Also, if anything happens to a worker Pod, then you will have to start over and recreate the hostfile.

BTW, CM for hostfile is a good idea since it support runtime update which makes it possible to support quasi-elastic.

How does elasticity work in deepspeed? Is it similar to horovod?

does this means it creates a configmap with the etc hosts content and mounts it with a volume in /etc/hosts?

No, a hostfile is a requirement from MPI (https://www.open-mpi.org/faq/?category=running#mpirun-hostfile). The mpi-operator populates using the domain name for each Pod. If we were to wait for the pods to be created, we could put the Pod IPs instead, which is what the OP is proposing.
Which sounds like we would be re-implementing DNS.

[vsoch]

Also, if anything happens to a worker Pod, then you will have to start over and recreate the hostfile.

This was the main point that made me pause on the design. However, I do think the length of the hostnames are an issue, especially when you get into indexed job. With regular job, it's usually successful with a reasonable (and somewhat short) service and job name. However, when you go into JobSet (if you do) I found that I had to make everything tiny - the service name often 2 letters, and the different jobs also indicated by letters. The design of how the JobSet members map to the names didn't allow for the same lengths as regular job.

When we were debugging networking issues, at least for small numbers of pods I did compare writing the hostfile to using the headless service, and the first didn't help much - the means / quartiles were very much overlapping. kubernetes/kubernetes#117819 (comment)

[kuizhiqing]

@vsoch Great, thx, I'm already completed the implementations and make those features into production, what I'm wandering is to merge some features back here.

[kuizhiqing]

Also, if anything happens to a worker Pod, then you will have to start over and recreate the hostfile.

This was the main point that made me pause on the design. However, I do think the length of the hostnames are an issue, especially when you get into indexed job. With regular job, it's usually successful with a reasonable (and somewhat short) service and job name. However, when you go into JobSet (if you do) I found that I had to make everything tiny - the service name often 2 letters, and the different jobs also indicated by letters. The design of how the JobSet members map to the names didn't allow for the same lengths as regular job.

When we were debugging networking issues, at least for small numbers of pods I did compare writing the hostfile to using the headless service, and the first didn't help much - the means / quartiles were very much overlapping. kubernetes/kubernetes#117819 (comment)

Well, since NCCL is not support fault tolerance, perfect recovery is impossible in GPU training scenario, what the we can do it to replace one pod a time without recreating thousands/hundred of pods. The failure node trigger the internal restart and IP update in the hostfile is the current design for me.

[alculquicondor]

If the host names are too large, maybe we can make this optional #453 (via command line flag, for example). In most scenarios, we don't actually need the service name.

[kuizhiqing]

When I say env length limit, I refer https://informix.hcldoc.com/14.10/help/index.jsp?topic=%2Fcom.ibm.sqlr.doc%2Fids_sqr_405.htm.

It happen when user use mpirun --host $hosts or pdsh -w $hosts or something similar.

[kuizhiqing]

Well I will hold the PR process and welcome more discussion on the feature requirement.

More ideas are really appreciate, @alculquicondor @vsoch @aojea @willb .

Thx

[alculquicondor]

It happen when user use mpirun --host $hosts or pdsh -w $hosts or something similar.

We shouldn't support that. It feels unnecessary.

[alculquicondor]

Regarding using IPs instead of DNS, I would like to hear from @aojea, as TL for Kubernets SIG network.

My gut feeling is that improving DNS building and lookup should be done by k8s or providers, and we shouldn't re-implement it in mpi-operator.