[Roadmap] Q4 2024

[peterschmidt85]

This issue outlines the major items planned for Q4 2024.

For major bugs, see major

• Gateway & Services
  • [Feature]: Allow running services and models without a gateway #1595
  • Support services in-place updates via dstack apply #1926
  • [Feature] Provide a public API to scale services #1762
  • [Feature]: Continuous health checks on services with reactions like pod-replacements  #1349
  • [Feature] Support vision models with gateway #1777
• Observability
  • [Feature] Collect hardware utilization metrics, incl. CPU and GPU #1761
  • Service & LLM metrics
  • [Feature] An API to export metrics, incl. requests and hardware utilization #1763
• Distributed training
  • [Feature]: AWS EFA (Elastic Fabric Adapter) support #1781
  • [Feature]: Enable all available network cards on AWS instances #1804
  • Batch cluster provisioning #1489
  • Support multi-host TPUs #1337
  • Lambda 1CC
• Volumes
  • Support volumes for Lambda #1863
  • Support volumes for Azure #1481
  • [Feature] Allow to attach multiple volumes based on backend and region #1949
  • Support multi-attach volumes #1467
• On-prem
  • SSH fleet volumes #1796
  • NFS on-prem volumes
  • [Feature] Support Singularity to allow root-less access to HPC #1862
  • [Bug] On-prem fleets require sudo access #1381
• UI
  • Add UI for volumes #1683
  • [UI] Add a Code button to the model playground page (similar to OpenAI/Anthropic) #1639
  • [Feature]: Show all jobs in runs UI #1723
• Misc
  • Support services in-place updates via dstack apply #1926
  • For fleets, support ranges (0.., 1..10, 1.., etc) in addition to a fixed number of nodes #1448
  • Drop all custom docker image requirements #1535
  • Implement secrets #708
  • [Feature] Allow to configure configure tags per backend and or per configuration #1838
  • [Feature] Support private subnets on Azure #1672
  • [Feature]: Support the Capacity Blocks for ML feature for AWS #1155
  • [Feature]: Support H100 with Azure #1239
  • Support for H200 in AWS/GCP/Azure and other providers
• Experimentation
  • [Feature]: Initial support for routers (using AWS Bedrock) #1631
  • Treat gateways as runs to cover these issues:
    • [Feature]: Support resources with the gateway configuration type #1664
    • [Feature]: on-prem gateway #1402
    • gateways support in various backends
    • dstack server must collect logs from gateways and instances #1673
  • [Feature] Run multiple jobs concurrently on one instance with a fraction of resources GPU/RAM/disk #1780
  • Intel Gaudi support
• Examples
  • [Examples] Add TGI example #1751
  • [Examples] Add vLLM example #1623
  • [Examples] Add NVIDIA Nim example #1748
  • [Examples] An example of fune-tuning with torchtune #1787
  • [Examples] Add GitHub Actions example #1749
  • [Examples] Add Airflow example #1752
  • [Examples] Add TensorRT-LLM example (end-to-end) #1753
  • [Examples] Add FLUX example #1750
  • Whisper
  • [Examples] Add Airflow example #1752
  • Ray
  • Spark