Skip to content

Commit

Permalink
Merge pull request #332 from JesseStutler/website_update
Browse files Browse the repository at this point in the history
Update home page and introduction
  • Loading branch information
volcano-sh-bot authored Jan 21, 2025
2 parents 947d0ac + 3c7d4b0 commit 21fc4df
Show file tree
Hide file tree
Showing 21 changed files with 301 additions and 143 deletions.
2 changes: 1 addition & 1 deletion config/_default/languages.toml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
[[zh.menu.main_right]]
name = "<img src=\"/img/icon_slack.svg\" alt=\"slack\" style=\" font-size: 1rem; line-height: 1.25; width:20px; height:20px; margin-top: 9px;\">"
post = ""
url = "https://volcano-sh.slack.com"
url = "https://cloud-native.slack.com/archives/C011GJDQS0N"
weight = 30

# Documentation version latest
Expand Down
2 changes: 1 addition & 1 deletion config/_default/menus.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@

name = "<img src=\"/img/icon_slack.svg\" alt=\"slack\" style=\" font-size: 1rem; line-height: 1.25; width:20px; height:20px; margin-top: 9px;\">"
post = ""
url = "https://volcano-sh.slack.com"
url = "https://cloud-native.slack.com/archives/C011GJDQS0N"
weight = 30

# Documentation version latest
Expand Down
20 changes: 15 additions & 5 deletions content/en/author/volcano/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,20 @@ interests = []
link = "https://twitter.com/volcano_sh"

+++
Volcano is system for running high-performance workloads on Kubernetes. It features powerful batch scheduling capability that Kubernetes cannot provide but is commonly required by many classes of high-performance workloads, including:
Volcano is CNCF's first cloud native batch computing project, focusing on high performance computing scenarios such as AI, big data, and genomics analysis. Its core capabilities include:

- Machine learning/Deep learning
- Bioinformatics/Genomics
- Other big data applications
• Unified Scheduling: Supports integrated job scheduling for both Kubernetes native workloads and mainstream computing frameworks (such as TensorFlow, Spark, PyTorch, Ray, Flink, etc.).

These types of applications typically run on generalized domain frameworks like TensorFlow, Spark, PyTorch, and MPI. Volcano is integrated with these frameworks to allow you to run your applications without adaptation efforts while enjoying remarkable batch scheduling.
• Queue Management: Provides multi-level queue management capabilities, enabling fine-grained resource quota control and task priority scheduling.

• Heterogeneous Device Support: Efficiently schedules heterogeneous devices like GPU and NPU, fully unleashing hardware computing potential.

• Network Topology Aware Scheduling: Greatly enhancing model training efficiency in AI distributed training scenarios.

• Multi-cluster Scheduling: Supports cross cluster job scheduling, improving resource pool management capabilities and achieving large scale load balancing.

• Online and Offline Workloads Colocation: Enables online and offline workloads colocation, improving cluster resource utilization through intelligent scheduling strategies.

• Load Aware Descheduling: Optimizing cluster load distribution and enhancing system stability.

As the industry's first cloud native batch computing engine, Volcano has been widely applied in high-performance computing scenarios such as artificial intelligence, big data, and genome sequencing, providing powerful support for enterprises to build elastic, efficient, and intelligent computing platforms.
89 changes: 62 additions & 27 deletions content/en/docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ type = "docs" # Do not modify.
+++

## What is Volcano
Volcano is a cloud native system for high-performance workloads, which has been accepted by [Cloud Native Computing Foundation (CNCF)](https://www.cncf.io/) as its first and only official container batch scheduling project. Volcano supports popular computing frameworks such as [Spark](https://spark.apache.org/), [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), [Flink](https://flink.apache.org/), [Argo](https://argoproj.github.io/), [MindSpore](https://www.mindspore.cn/en), and [PaddlePaddle](https://www.paddlepaddle.org.cn/). Volcano also supports scheduling of computing resources on different architecture, such as x86, Arm, and Kunpeng.

Volcano is a cloud native system for high-performance workloads, which has been accepted by [Cloud Native Computing Foundation
(CNCF)](https://www.cncf.io/) as its first and only official container batch scheduling project. Volcano supports popular computing
frameworks such as [Spark](https://spark.apache.org/), [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/),
[Flink](https://flink.apache.org/), [Argo](https://argoproj.github.io/), [MindSpore](https://www.mindspore.cn/en),
[PaddlePaddle](https://www.paddlepaddle.org.cn/) and [Ray](https://www.ray.io/). Volcano also provides various scheduling capabilities including heterogeneous device scheduling, network topology-aware scheduling, multi-cluster scheduling, online-offline workloads colocation and so on.

## Why Volcano
Job scheduling and management become increasingly complex and critical for high-performance batch computing. Common requirements are as follows:

Expand All @@ -27,25 +31,30 @@ Job scheduling and management become increasingly complex and critical for high-

Volcano is designed to cater to these requirements. In addition, Volcano inherits the design of Kubernetes APIs, allowing you to easily run applications that require high-performance computing on Kubernetes.
## Features
### Rich scheduling policies
Volcano supports a variety of scheduling policies:

* Gang scheduling
* Fair-share scheduling
* Queue scheduling
* Preemption scheduling
* Topology-based scheduling
* Reclaim
* Backfill
* Resource reservation

You can also configure plug-ins and actions to use custom scheduling policies.
### Enhanced job management
You can use enhanced job features of Volcano for high-performance computing:

* Multi-pod jobs
* Improved error handling
* Indexed jobs
### [Unified Scheduling](/en/docs/unified_scheduling/)
* Support native Kubernetes workload scheduling
* Provide complete support for frameworks like PyTorch, TensorFlow, Spark, Flink, Ray through VolcanoJob
* Unified scheduling for both online microservices and offline batch jobs to improve cluster resource utilization

### Rich Scheduling Policies
* **Gang Scheduling**: Ensure all tasks of a job start simultaneously, suitable for distributed training and big data scenarios
* **Binpack Scheduling**: Optimize resource utilization through compact task allocation
* **Heterogeneous Device Scheduling**: Efficiently share GPU resources, support both CUDA and MIG modes for GPU scheduling, and NPU scheduling
* **Proportion/Capacity Scheduling**: Resource sharing/preemption/reclaim based on queue quotas
* **NodeGroup Scheduling**: Support node group affinity scheduling, implementing binding between queues and node groups
* **DRF Scheduling**: Support fair scheduling of multi-dimensional resources
* **SLA Scheduling**: Scheduling guarantee based on service quality
* **Task-topology Scheduling**: Support task topology-aware scheduling, optimizing performance for communication-intensive applications
* **NUMA Aware Scheduling**: Supports scheduling for NUMA architecture, optimizing resource allocation for tasks on multi-core processors, enhancing memory access efficiency and computational performance.
* ...

Volcano supports custom plugins and actions to implement more scheduling algorithms.

### [Queue Resource Management](/en/docs/queue_resource_management/)
* Support multi-dimensional resource quota control (CPU, Memory, GPU, etc.)
* Provide multi-level queue structure and resource inheritance
* Support resource borrowing, reclaiming and preemption between queues
* Implement multi-tenant resource isolation and priority control

### Multi-architecture computing
Volcano can schedule computing resources from multiple architectures:
Expand All @@ -56,24 +65,50 @@ Volcano can schedule computing resources from multiple architectures:
* Ascend
* GPU

### Faster scheduling
Compared with existing queue schedulers, Volcano shortens the average scheduling delay through a series of optimizations.
### Network Topology-aware Scheduling
* Supports network topology-aware scheduling, fully considering the network bandwidth characteristics between nodes. In AI scenarios, this network topology-aware scheduling effectively optimizes data transmission for communication-intensive distributed training tasks, significantly reducing communication overhead and improving model training speed and overall efficiency.

### Online and Offline Workloads Colocation
* Supports online and offline workloads colocation, enhancing resource utilization while ensuring online worloads QoS through unified scheduling, dynamic resource overcommitment, CPU burst, and resource isolation.

### Multi-cluster Scheduling
* Support cross-cluster job scheduling for larger-scale resource pool management and load balancing

> For more details about multi-cluster scheduling, see: [volcano-global](https://github.com/volcano-sh/volcano-global)
### Descheduling
* Support dynamic descheduling to optimize cluster load distribution and improve system stability

> For more details about descheduling, see: [descheduler](https://github.com/volcano-sh/descheduler)
### Monitoring and Observability
* Complete logging system
* Rich monitoring metrics
* Provides a dashboard, facilitating graphical interface operations for users.

> For more details about dashboard, see: [dashboard](https://github.com/volcano-sh/dashboard)
>
> For more details about volcano metrics, see: [metrics](https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md)
## Ecosystem
Volcano allows you to use mainstream computing frameworks:
Volcano has become the de facto standard in batch computing scenarios and is widely used in the following high-performance computing frameworks:

* [Spark](https://spark.apache.org/)
* [TensorFlow](https://www.tensorflow.org/)
* [PyTorch](https://pytorch.org/)
* [Flink](https://flink.apache.org/)
* [Argo](https://argoproj.github.io/)
* [MindSpore](https://www.mindspore.cn/en)
* [Ray](https://www.ray.io/)
* [MindSpore](https://www.mindspore.cn/)
* [PaddlePaddle](https://www.paddlepaddle.org.cn/)
* [Open MPI](https://www.open-mpi.org/)
* [OpenMPI](https://www.open-mpi.org/)
* [Horovod](https://horovod.readthedocs.io/)
* [MXNet](https://mxnet.apache.org/)
* [Kubeflow](https://www.kubeflow.org/)
* [KubeGene](https://github.com/volcano-sh/kubegene)
* [Cromwell](https://cromwell.readthedocs.io/)

Volcano has been commercially used as the infrastructure scheduling engine by companies and organizations.
Additionally, Volcano has been widely adopted by various enterprises and organizations in the fields of AI and big data. With its powerful resource management capabilities, efficient job management mechanisms, and rich scheduling strategies (such as Gang scheduling, heterogeneous device scheduling, and topology-aware scheduling), it effectively meets the complex demands of distributed training and data analysis tasks. At the same time, Volcano enhances scheduling performance while ensuring the flexibility and reliability of task scheduling, providing strong support for enterprises to build an efficient resource utilization system.

## Future Outlook
Volcano will continue to expand its functional boundaries through community collaboration and technical innovation, becoming a leader in high-performance computing and cloud-native batch scheduling.
2 changes: 1 addition & 1 deletion content/en/docs/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The community is committed to developing a system that helps running high-perfor

You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged.

If you have any question about the development process, visit the [Slack Channel](https://volcano-sh.slack.com) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).

#### Find Something to Work On
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/v1-10-0/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ The community is committed to developing a system that helps running high-perfor

You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged.

If you have any question about the development process, visit the [Slack Channel](https://volcano-sh.slack.com) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).

#### Find Something to Work On
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/v1-7-0/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The community is committed to developing a system that helps running high-perfor

You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged.

If you have any question about the development process, visit the [Slack Channel](https://volcano-sh.slack.com) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).

#### Find Something to Work On
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/v1-8-2/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The community is committed to developing a system that helps running high-perfor

You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged.

If you have any question about the development process, visit the [Slack Channel](https://volcano-sh.slack.com) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).

#### Find Something to Work On
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/v1-9-0/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ The community is committed to developing a system that helps running high-perfor

You can contribute in different areas, including filing issues, developing features, fixing critical bugs, and getting your work reviewed and merged.

If you have any question about the development process, visit the [Slack Channel](https://volcano-sh.slack.com) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
If you have any question about the development process, visit the [Slack Channel](https://cloud-native.slack.com/archives/C011GJDQS0N) ([sign up](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk))
or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).

#### Find Something to Work On
Expand Down
Loading

0 comments on commit 21fc4df

Please sign in to comment.