Sophisticated docker builds for parent project oobabooga/text-generation-webui.

Easy setup. Compatible. Tweakable. Scaleable.

TODO support different GPTQ-for-Llama's TODO fixp for compose mounts / dev env

-cc-1.2.3 - NVidia compute capability, see NVidia CUDA GPUs

amd64

main

• docker and docker compose are available on your system

These containers require nvidia-container-toolkit installed and reboot

Run text-generation-webui with a pretrained LLama-30B model on e.g. RTX 3090, RTX 4090

• get docker-compose.yml (clone repo, copy or else)
• Download OpenAssistant-SFT-7-Llama-30B-GPTQ into ../models
• Run docker compose up
• Model will load into gpu
• open/refresh https://localhost:7860

Environment variables to set for the specific service

Prepend, e.g. COMPUTE_VERSION=6.1 for Tesla P40

Prepend, e.g. LOCAL_MODEL_DIR=/my/path/to/models

docker compose pull

docker compose rm

When there is a new version and there is need of builds or you require the latest main build, feel free to open an issue

Open an issue on the Issue Tracker

We cannot support issues regarding the base software. Please refer to the main project page mentioned in the second line of this card.