genai-microservices-connector(GMC)

This repo defines the GenAI Microservice Connector(GMC) for OPEA projects. GMC can be used to compose and adjust GenAI pipelines dynamically on kubernetes. It can leverage the microservices provided by GenAIComps and external services to compose GenAI pipelines. External services might be running in a public cloud or on-prem by providing an URL and access details such as an API key and ensuring there is network connectivity. It also allows users to adjust the pipeline on the fly like switching to a different Large language Model(LLM), adding new functions into the chain(like adding guardrails),etc. GMC supports different types of steps in the pipeline, like sequential, parallel and conditional.

Refer to usage_guide for sample use cases. Refer to chatqna_use_cases for more ChatQnA use cases.

Description

The GenAI Microservice Connector(GMC) contains the CustomResourceDefinition(CRD) and its controller to bring up the services needed for a GenAI application. Istio Service Mesh can also be leveraged to facilicate communication between microservices in the GenAI application.

Architecture

Personas

Getting Started

CRD defines are at config/crd/bases/ API is api/v1alpha3/ Controller is at internal/controller

Prerequisites

Access to a Kubernetes v1.11.3+ cluster.

Introduction

There are two components in this repo:

manager: controller manager to handle GMC CRD
router: route the traffic among the microservices defined in GMC

GMC build

Binaries building

make build

Docker images building

make docker.build

Binaries deleting

make clean

GMC Deployment on K8s cluster

GMC Images NOTE: This image ought to be published in OPEA docker hub, including gmcmanager and gmcrouter. Make sure you have the proper permission to the registry and use the latest images.

There are 2 methods for deploying GMC on K8s cluster:

Deploy via native kubectl
Deploy via helm chart

Deploy using native kubectl

There are 3 steps for deploying GMC on K8s cluster as below:

Install GMC CRD
Prepare GenAI Components and GMC Router manifests
Deploy GMC Manager

Deploy GMC NOTE:

Before installting the manifests, replace your own huggingface tokens
MOUNT_DIR is the hostPath to save the models on you cluster hosts, so make sure it is ready on every node of the k8s nodes and the microservices have enough permission to access it.
The SYSTEM_NAMESPACE should keep the same with the namespace defined in gmc-manager.yaml and gmc-manager-rbac.yaml
The configmap name gmcyaml is defined in gmcmanager deployment Spec. Modify accordingly if you want use a different name for the configmap

# Install GMC CRD
kubectl apply -f config/crd/bases/gmc.opea.io_gmconnectors.yaml
# Prepare GenAI Components and GMC Router manifests
cp $(pwd)/config/gmcrouter/gmc-router.yaml -p $(pwd)/config/manifests/
export YOUR_HF_TOKEN=<your hugging facetoken>
export MOUNT_DIR=<your model path>
find . -name '*.yaml' -type f -exec sed -i "s#insert-your-huggingface-token-here#$YOURTOKEN#g" {} \;
find . -name '*.yaml' -type f -exec sed -i "s#path: /mnt/opea-models#path: $MOUNT_DIR#g" {} \;
# Deploy GMC Manager
export SYSTEM_NAMESPACE=system
kubectl create namespace $SYSTEM_NAMESPACE
kubectl create configmap gmcyaml -n $SYSTEM_NAMESPACE --from-file $(pwd)/config/manifests
kubectl apply -f $(pwd)/config/rbac/gmc-manager-rbac.yaml
kubectl apply -f $(pwd)/config/manager/gmc-manager.yaml

Check the installation result

kubectl get pods -n system
NAME                              READY   STATUS    RESTARTS   AGE
gmc-controller-78f9c748cb-ltcdv   1/1     Running   0          3m

To Uninstall via kubectl

# Delete the instances (CRs) from the cluster
kubectl delete -k config/samples/
# Delete the APIs(CRDs) from the cluster
make uninstall
# UnDeploy the controller from the cluster
make undeploy

Deploy via helm chart

Refer to helm chart README for deploying GMC using helm chart.

Next Step

Refer to usage_guide for sample use cases. Refer to advanced_use_cases for more use cases based on ChatQnA example.

Troubleshooting guide

Refer to this troubleshooting_guide for identifying GMC Custom Resource issues.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!