Quickstart

Your onboarding includes 2 steps.

Before deploying your inference graphs you need to install the Dynamo Inference Platform and the Dynamo Cloud. Dynamo Cloud acts as an orchestration layer between the end user and Kubernetes, handling the complexity of deploying your graphs for you. You could install from Published Artifacts or Source
Once you install the Dynamo Cloud, proceed to the Examples to deploy an inference graph.

1. Installing Dynamo Cloud from Published Artifacts

Use this approach when installing from pre-built helm charts and docker images published to NGC.

Prerequisites

export NAMESPACE=dynamo-cloud
export RELEASE_VERSION=0.3.2

Install envsubst, kubectl, helm

Authenticate with NGC

Go to https://ngc.nvidia.com/org to get your NGC_CLI_API_KEY.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --username='$oauthtoken' --password=<YOUR_NGC_CLI_API_KEY>

Fetch Helm Charts

# Fetch the CRDs helm chart
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz

# Fetch the platform helm chart
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz

Install Dynamo Cloud

Step 1: Install Custom Resource Definitions (CRDs)

helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz \
  --namespace default \
  --wait \
  --atomic

Step 2: Install Dynamo Platform

kubectl create namespace ${NAMESPACE}

helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace ${NAMESPACE}

2. Installing Dynamo Cloud from Source

Use this approach when developing or customizing Dynamo as a contributor, or using local helm charts from the source repository.

Prerequisites

Ensure you have the source code checked out and are in the dynamo directory:

Set Environment Variables

Our examples use the nvcr.io but you can setup your own values if you use another docker registry.

export NAMESPACE=dynamo-cloud # or whatever you prefer.
export DOCKER_SERVER=nvcr.io/nvidia/ai-dynamo/  # your-docker-registry.com
export DOCKER_USERNAME='$oauthtoken'  # your-username if not using nvcr.io
export DOCKER_PASSWORD=YOUR_NGC_CLI_API_KEY  # your-password if not using nvcr.io

Pick the Dynamo Inference Image

Export the tag of the Dynamo Runtime Image. If you are using a pre-defined release:

export IMAGE_TAG=RELEASE_VERSION # i.e. 0.3.2 - the release you are using

Or build your own image first and tag it with IMAGE_TAG

export IMAGE_TAG=<your-pick>
./container/build.sh
docker tag dynamo:latest-vllm <your-registry>/dynamo-base:$IMAGE_TAG
docker login <your-registry>
docker push <your-registry>/dynamo-base:latest-vllm

Install Dynamo Cloud

You need to build and push the Dynamo Cloud Operator Image by running

cd deploy/cloud/operator
earthly --push +docker --DOCKER_SERVER=$DOCKER_SERVER --IMAGE_TAG=$IMAGE_TAG

The Nvidia Cloud Operator image will be pulled from the $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG.

You could run the deploy.sh or use the manual commands under Step 1 and Step 2.

Installing with a script (alternative to the Step 1 and Step 2)

Create the namespace and the docker registry secret.

kubectl create namespace ${NAMESPACE}
kubectl create secret docker-registry docker-imagepullsecret \
  --docker-server=${DOCKER_SERVER} \
  --docker-username=${DOCKER_USERNAME} \
  --docker-password=${DOCKER_PASSWORD} \
  --namespace=${NAMESPACE}

You need to add the bitnami helm repository by running:

helm repo add bitnami https://charts.bitnami.com/bitnami

./deploy.sh --crds

if you want guidance during the process, run the deployment script with the --interactive flag:

./deploy.sh --crds --interactive

Installing CRDs manually (alternative to the script deploy.sh)

*Step 1: Install Custom Resource Definitions (CRDs)

helm install dynamo-crds ./crds/ \
  --namespace default \
  --wait \
  --atomic

*Step 2: Build Dependencies and Install Platform

helm dep build ./platform/

kubectl create namespace ${NAMESPACE}

# Create docker registry secret
kubectl create secret docker-registry docker-imagepullsecret \
  --docker-server=${DOCKER_SERVER} \
  --docker-username=${DOCKER_USERNAME} \
  --docker-password=${DOCKER_PASSWORD} \
  --namespace=${NAMESPACE}

# Install platform
helm install dynamo-platform ./platform/ \
  --namespace ${NAMESPACE} \
  --set "dynamo-operator.controllerManager.manager.image.repository=${DOCKER_SERVER}/dynamo-operator" \
  --set "dynamo-operator.controllerManager.manager.image.tag=${IMAGE_TAG}" \
  --set "dynamo-operator.imagePullSecrets[0].name=docker-imagepullsecret"

Uninstall CRDs for a clean start

We provide a script to uninstall CRDs should you need a clean start.

./uninstall.sh

Explore Examples

If deploying to Kubernetes, create a Kubernetes secret containing your sensitive values if needed:

export HF_TOKEN=your_hf_token
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN=${HF_TOKEN} \
  -n ${NAMESPACE}

Follow the Examples For more details on how to create your own deployments follow Create Deployment Guide

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!