0.18.11

AMD

With the latest update, you can now specify an AMD GPU under resources. Below is an example.

type: service
name: amd-service-tgi

image: ghcr.io/huggingface/text-generation-inference:sha-a379d55-rocm
env:
  - HUGGING_FACE_HUB_TOKEN
  - MODEL_ID=meta-llama/Meta-Llama-3.1-70B-Instruct
  - TRUST_REMOTE_CODE=true
  - ROCM_USE_FLASH_ATTN_V2_TRITON=true
commands:
  - text-generation-launcher --port 8000
port: 8000

resources:
  gpu: MI300X
  disk: 150GB

spot_policy: auto

model:
  type: chat
  name: meta-llama/Meta-Llama-3.1-70B-Instruct
  format: openai

Note

AMD accelerators are currently supported only with the runpod backend. Support for on-prem fleets and more backends
is coming soon.

GPU vendors

The gpu property now accepts the vendor attribute, with supported values: nvidia, tpu, and amd.

Alternatively, you can also prefix the GPU name with the vendor name followed by a colon, for example: tpu:v2-8 or amd:192GB, etc. This change ensures consistency in GPU requirements configuration across vendors.

Encryption

dstack now supports encryption of sensitive data, such as backend credentials, user tokens, etc. Learn more on the reference page.

Storing logs in AWS CloudWatch

By default, the dstack server stores run logs in ~/.dstack/server/projects/<project name>/logs. To store logs in AWS CloudWatch, set the DSTACK_SERVER_CLOUDWATCH_LOG_GROUP environment variable.

Project manager role

With this update, it's now possible to assign any user as a project manager. This role grants permission to manage project users but does not allow management of backends or resources.

Default permissions

By default, all users can create and manage their own projects. If you want only global admins to create projects, add the following to ~/.dstack/server/config.yml:

default_permissions:
  allow_non_admins_create_projects: false

Other

• [Feature] Allow to store logs in AWS CloudWatch by @un-def in #1597 and #1597
• [Feature] Introduce default permissions #1559 by @olgenn in #1567
• [Feature] Support the vendor property under resources.gpu @un-def in #1558
• [Feature] Implement configurable default permissions by @r4victor in #1591
• [Bugfix] Provision AWS instances in all eligible availability zones by @r4victor in #1585
• [Bugfix] Support users without projects @olgenn in #1578
• [UI] Support manager project role @olgenn in #1566
• [Docs] Mention AMD GPUs, describe gpu.vendor property by @un-def in #1570
• [Bugfix] Fix global admin restricted by manager role by @r4victor in #1592
• [Bugfix] Fixed defect with incorrect setting project role in the UI by @olgenn in #1593
• [Bugfix] Abort provisioning fleet when parsing ssh key fails(#1442) by @swsvc in #1589
• [UI] Ensure users can create projects #191 by @olgenn in #1554
• [UI] Use a toggle button switching themes #190 by @olgenn in #1556
• [UI] Fix the Logs component appearance for the dark theme by @olgenn in #1579
• [UI] Minor restyle of the side navigation by @olgenn in #1580
• [Bugfix] Avoid TGI error logit_bias: invalid type by @jvstme in #1557
• [Docs] Document projects #1547 by @peterschmidt85 in #1548
• [Docs] Document AMD support on RunPod by @peterschmidt85 in #1598
• [Internal] Approximate on-prem GPU memory size by @jvstme in #1588
• [Docs] Fix some of the broken links by @jvstme in #1602
• [Docs] Fix broken links in README.md by @jvstme in #1604
• [Docs] Document configuring logs storage in AWS CloudWatch @un-def in #1606
• [Docs] Publish the blog post and examples about AMD on RunPod by @peterschmidt85 in #1598
• [Internal] Force root in Kubernetes runs by @jvstme in #1555
• [Internal] Improve gateway auth issues troubleshooting by @jvstme in #1569
• [Feature] Implement "encryption at rest" by @r4victor in #1561
• [Feature] Implement project manager role by @r4victor in #1572
• [Feature] Implement user activation/deactivation by @r4victor in #1575
• [Internal] Reintroduce tpu- prefix; add tpu vendor alias by @un-def in #1587

New contributors

• @swsvc made their first contribution in #1589

Full changelog: 0.18.10...0.18.11