SentenceTransformers is a Python framework for state-of-the-art sentence, text and image embeddings.

This is a BentoML example project, demonstrating how to build a sentence embedding inference API server, using a SentenceTransformers model all-MiniLM-L6-v2. See here for a full list of BentoML example projects.

git clone https://github.com/bentoml/BentoSentenceTransformers.git
cd BentoSentenceTransformers
# Recommend Python 3.11
pip install -r requirements.txt

We have defined a BentoML Service in service.py. Run bentoml serve in your project directory to start the Service.

$ bentoml serve .
2024-01-18T06:40:53+0800 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "service:SentenceEmbedding" can be accessed at https://localhost:3000/metrics.
2024-01-18T06:40:54+0800 [INFO] [cli] Starting production HTTP BentoServer from "service:SentenceEmbedding" listening on https://localhost:3000 (Press CTRL+C to quit)
Model loaded device: cpu

The server is now active at https://localhost:3000. You can interact with it using the Swagger UI or in other different ways.

CURL

curl -X 'POST' \
  'https://localhost:3000/encode' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "sentences": [
    "hello world"
  ]
}'

Python client

import bentoml
with bentoml.SyncHTTPClient("https://localhost:3000") as client:
    result = client.encode(
        sentences=[
                "hello world"
        ],
    )

For detailed explanations of the Service code, see Sentence Transformer.

After the Service is ready, you can deploy the application to BentoCloud for better management and scalability. Sign up if you haven't got a BentoCloud account.

Make sure you have logged in to BentoCloud, then run the following command to deploy it.

bentoml deploy .

Once the application is up and running on BentoCloud, you can access it via the exposed URL.

Note: For custom deployment in your own infrastructure, use BentoML to generate an OCI-compliant image.