Runs

Priorities

Run configurations now support a priority field. This is a new property that accepts a number between 0 and 100. The higher the number, the higher the priority of the run. This influences the order in which runs are provisioned and executed in the queue.

type: task
name: train
# Can be 0–100; higher means higher priority
priority: 50
python: "3.10"
# Commands of the task
commands:
  - pip install -r fine-tuning/qlora/requirements.txt
  - python fine-tuning/qlora/train.py
# Retry policy to queue tasks
retry:
  on_events: [no-capacity]
  duration: 1d

Previously, submitted jobs were processed in a FIFO manner, with older jobs handled first. Now, jobs are first sorted by descending priority. Note that if a high-priority run cannot be scheduled, it does not block lower-priority runs from being scheduled (i.e., best-effort FIFO).

The priority field is updatable, so it can be modified for already submitted runs and will take effect.

CLI

dstack project command

The new dstack project command replaces the existing dstack config command.

1. dstack project (same as dstack project list)

$ dstack project
 PROJECT         URL                    USER            DEFAULT
 peterschmidt85  https://sky.dstack.ai  peterschmidt85
 main            https://127.0.0.1:3000  admin              ✓

2. dstack project set-default

$ dstack project set-default peterschmidt85
OK

3. dstack project add (similar to old dstack config, but --project is changed to --name)

$ dstack project add --name peterschmidt85 --url https://sky.dstack.ai --token 76d8dd51-0470-74a7-24ed9ec18-fb7d341
OK

dstack ps -n/--last

The dstack ps command now supports a new -n/--last parameter to show last N runs:

✗ dstack ps -n 3
 NAME             BACKEND             RESOURCES                                    PRICE    STATUS      SUBMITTED    
 good-panther-2   gcp (europe-west4)  cpu=2 mem=8GB disk=100GB                     $0.0738  terminated  49 mins ago  
 new-chipmunk-1   azure (westeurope)  cpu=2 mem=8GB disk=100GB (spot)              $0.0158  terminated  23 hours ago 
 fuzzy-panther-1  runpod (EU-RO-1)    cpu=6 mem=31GB disk=100GB RTX2000Ada:16GB:1  $0.28    terminated  yesterday

Azure

Fsv2 series

The Azure backend now supports compute-optimized Fsv2 series:

✗ dstack apply -b azure
 Project              main                           
 User                 admin                          
 Configuration        .dstack.yml                    
 Type                 dev-environment                
 Resources            cpu=4.. mem=8GB.. disk=100GB.. 
 Spot policy          auto                           
 Max price            -                              
 Retry policy         -                              
 Creation policy      reuse-or-create                
 Idle duration        5m                             
 Max duration         -                              
 Inactivity duration  -                              
 Reservation          -                              
 #  BACKEND             RESOURCES                         INSTANCE TYPE      PRICE     
 1  azure (westeurope)  cpu=4 mem=8GB disk=100GB (spot)   Standard_F4s_v2    $0.0278   
 2  azure (westeurope)  cpu=4 mem=16GB disk=100GB (spot)  Standard_D4s_v3    $0.0312   
 3  azure (westeurope)  cpu=4 mem=32GB disk=100GB (spot)  Standard_E4-2s_v4  $0.0416   
    ...                                                                                
 Shown 3 of 98 offers, $40.962max

Major bugfixes

• [Bug]: Instances with blocks feature cannot be used for multi-node runs #2650

Deprecations

• The dstack config CLI command is deprecated in favor of dstack project add.

What's changed

• [Bug] Allow multi-node tasks on idle instances with blocks by @un-def in #2651
• [UX] Make local code upload size limit configurable by @colinjc in #2673
• [Feature] Implement run priorities by @r4victor in #2635
• [Bug] Fix IllegalStateChangeError in delete_metrics task by @un-def in #2639
• [Examples] Renamed some example groups for better extensibility by @peterschmidt85 in #2641
• [Azure] Support Azure Fsv2-series by @r4victor in #2647
• [UX]: Add dstack project CLI to configure, list and switching between projects by @peterschmidt85 in #2653
• [UI] Dark/light theme toggler state is reset after page reload #289 by @olgenn in #2675
• [UX] Support dstack ps -n NUM by @peterschmidt85 in #2654
• [Docs] Added Clusters guide by @peterschmidt85 in #2646
• [UX] Replace conda with uv in dstackai/base images by @un-def in #2649
• [Docs]: Mention SSH fleet networking requirements by @jvstme in #2643
• [Bug] Put lower bounds on oci deps by @r4victor in #2658
• [UX]: Replace conda with uv in dstack's default Docker image #2625 by @peterschmidt85 in #2652
• [UX]: Replace conda with uv in dstack's default Docker image #2625 by @peterschmidt85 in #2659
• [Internal] Support building staging Docker images by @r4victor in #2664
• [Bug] Forbid scaling.target <= 0 by @jvstme in #2672

Full changelog: 0.19.9...0.19.10