Zum Inhalt springen
Trainingslager IT-Trainingslager

Anhang A: Spickzettel

Stand der Kapitel: kind 0.33, RKE2 v1.36.5, GPU Operator v26.7.1, vLLM v0.31.0, LiteLLM 1.104.2, Open WebUI Chart 16.6.0, Velero 1.18. Passwörter und Keys sind Lab-Werte.

Namen, Hosts und Ports

Was Name / Host Port / Ort
kind-Cluster ki-lab, Kontext kind-ki-lab Ingress 127.0.0.1:80/443
RKE2-Cluster Kontext rke2-lab, ~/ki-lab/rke2.yaml API-VIP 10.10.20.10:6443
RKE2-Join rke2.lab.internal Supervisor :9345
Server rke2-cp-1..3 (VMID 211–213) 10.10.20.11–13
Agents rke2-wk-1..2 (221–222), rke2-gpu-1 (231) .21–.22, .31
LB-Pool / Traefik Cilium lab-pool, Traefik .100 10.10.20.100–120
Proxmox pve1, Token tofu@pve!provider https://10.10.20.5:8006
S3 (Tag 15) Garage auf s3-lab, s3.lab.internal 10.10.20.40:3900
Template ubuntu-2404-tpl VMID 9000
Dienst Namespace Intern Extern
llama.cpp (kind) ki-models llamacpp:8080 –
vLLM Chat / Embed / Code ki-models vllm-chat, vllm-embed, vllm-code :8000 –
LiteLLM ki-gateway litellm.ki-gateway:4000 llm.ki.localtest.me / llm.lab.internal
Postgres LiteLLM ki-gateway kind postgres:5432, RKE2 litellm-db-rw –
Redis ki-gateway redis:6379 –
Open WebUI ki-chat open-webui:80 chat.ki.localtest.me / chat.lab.internal
Qdrant ki-chat qdrant:6333 –
Keycloak keycloak Realm ki-lab sso.lab.internal
Tabby ki-code tabby:8080 code.lab.internal
Grafana monitoring kube-prometheus-stack-grafana grafana.lab.internal
Prometheus monitoring kube-prometheus-stack-prometheus:9090 –
Alias (LiteLLM) Backend RKE2 served-model-name
chat-large vllm-chat, Fallback chat-large-eu (Mistral) qwen3-8b
chat-small vllm-chat, max_tokens: 512 qwen3-8b
code vllm-code (kein Fallback) qwen2.5-coder-7b
embed vllm-embed (1024 Dim.) bge-m3
Lab-Wert Wofür
sk-lab-master-1234 LiteLLM-Master-Key (kind und RKE2)
lab-grafana-1234 Grafana admin
[email protected] / lab-admin-1234 Open WebUI in kind
anna, ben, gast / lab-<name>-1234 Keycloak-Benutzer (Tag 13)
lab-qdrant-key-1234 Qdrant-API-Key

kind (Teil A)

# Cluster bauen / abreißen (idempotent, Images vorladen)
~/ki-lab/code/kind-lab/up.sh
PRELOAD=0 ~/ki-lab/code/kind-lab/up.sh
~/ki-lab/code/kind-lab/down.sh
# Image plattformgenau in die Nodes laden (Docker 29, containerd-Store)
docker save --platform linux/arm64 $IMG -o /tmp/img.tar
kind load image-archive /tmp/img.tar --name ki-lab
# Images im Node prüfen
docker exec ki-lab-worker crictl images | grep llama
# Modell-Cache im Node finden (local-path)
docker exec ki-lab-worker ls /var/local-path-provisioner
# Schnelltest über Ingress
curl -s http://llm.ki.localtest.me/health/readiness | jq -c '{status, db}'

Proxmox

# Rolle, Benutzer, Token für OpenTofu
pveum role add TofuProv -privs "VM.Allocate,VM.Clone,..."
pveum user token add tofu@pve provider --privsep 0
pveum user token list tofu@pve
# VMs ansehen, starten, stoppen (hart / sauber)
qm list
qm config 231 | grep -E '^(hostpci0|machine|bios|balloon)'
qm start 212; qm shutdown 212; qm stop 212
# Template aus Cloud-Image
qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/noble.img
qm template 9000
# GPU: IOMMU, vfio, Mapping
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
lspci -nnk -s 01:00.0 | grep 'driver in use'
pvesh get /cluster/mapping/pci --output-format json | jq -r '.[].id'
# API-Token testen
curl -sk -H "Authorization: PVEAPIToken=$PROXMOX_VE_API_TOKEN" \
  "${PROXMOX_VE_ENDPOINT}api2/json/version"

OpenTofu

cd ~/ki-lab/code/infra-proxmox && source ~/ki-lab/pve.env
tofu init && tofu fmt -check && tofu validate
tofu plan -out=tag07.plan && tofu apply tag07.plan
# Eine VM gezielt ersetzen (VM + Snippet)
tofu plan -replace='proxmox_virtual_environment_vm.node["rke2-wk-2"]' \
  -replace='proxmox_virtual_environment_file.user_data["rke2-wk-2"]'
# Inventory für Ansible
tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
tofu state list | grep _vm

Ansible

cd ~/ki-lab/code/infra-rke2 && source ~/ki-lab/rke2-token.env
ansible-inventory --graph
ansible all -m ansible.builtin.ping -o
ansible-playbook --syntax-check site.yml && ansible-lint
ansible-playbook site.yml                       # ganzer Cluster
ansible-playbook site.yml --limit rke2-wk-2     # ein Node
ansible-playbook gpu.yml                        # Treiber auf gpu_nodes
ansible gpu_nodes -b -m ansible.builtin.reboot

RKE2 und kubectl

export KUBECONFIG=~/ki-lab/rke2.yaml; kubectl config current-context
kubectl get nodes -o wide -L nvidia.com/gpu.product
# Wer hält die API-VIP?
kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}'
# Auf einem Server: Dienst, Logs, mitgelieferte Binaries
sudo systemctl status rke2-server; sudo journalctl -u rke2-server -f
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get no
sudo CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml \
  /var/lib/rancher/rke2/bin/crictl ps --name etcd
# etcd-Snapshots: anlegen, auflisten, alte löschen, als CR ansehen
sudo rke2 etcd-snapshot save --name pre-upgrade
sudo rke2 etcd-snapshot list
sudo rke2 etcd-snapshot prune --snapshot-retention 5
kubectl get etcdsnapshotfiles
# Quorum zurücksetzen bzw. Restore (alle Server vorher stoppen)
sudo rke2 server --cluster-reset
sudo rke2 server --cluster-reset --cluster-reset-restore-path="$SNAP"
# Upgrade per system-upgrade-controller
kubectl -n system-upgrade get plans -o wide
kubectl -n system-upgrade patch plan server-plan --type merge \
  -p '{"spec":{"version":"v1.36.6+rke2r1"}}'
# Node warten
kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data
kubectl uncordon rke2-wk-2
Datei / Pfad Inhalt
/etc/rancher/rke2/config.yaml Token, server:, tls-san, cni: cilium, etcd-S3
/var/lib/rancher/rke2/server/manifests/ Auto-Deploy: kube-vip, HelmChartConfigs
/var/lib/rancher/rke2/server/db/snapshots/ lokale etcd-Snapshots
/var/lib/rancher/rke2/agent/etc/containerd/config.toml containerd inkl. Runtime nvidia

Cilium

kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --brief
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status | grep -E 'KubeProxyReplacement|Cluster health'
kubectl get ciliumloadbalancerippools; kubectl get ciliuml2announcementpolicy
# Wer beantwortet ARP für eine LB-IP?
kubectl -n kube-system get lease | grep cilium-l2announce
# Feste LB-IP für einen Service
kubectl annotate svc "$SVC" lbipam.cilium.io/ips=10.10.20.101
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list

Helm-Releases

Release Chart Namespace Version
metrics-server metrics-server/metrics-server kube-system 3.14.0
cert-manager jetstack/cert-manager cert-manager v1.21.2
longhorn longhorn/longhorn longhorn-system 1.13.0
kube-prometheus-stack prometheus-community/kube-prometheus-stack monitoring 92.2.0
gpu-operator nvidia/gpu-operator gpu-operator v26.7.1
keda kedacore/keda keda 2.21.0
cnpg cnpg/cloudnative-pg cnpg-system 0.29.1
litellm oci://ghcr.io/berriai/litellm-helm ki-gateway 1.104.2
external-secrets external-secrets/external-secrets external-secrets 2.12.0
qdrant qdrant/qdrant ki-chat 1.19.2
open-webui open-webui/open-webui ki-chat 16.6.0
velero vmware-tanzu/velero velero 12.2.1
rke2-traefik, rke2-cilium RKE2-intern, per HelmChartConfig kube-system mit RKE2
helm list -A
helm get values gpu-operator -n gpu-operator
helm upgrade longhorn longhorn/longhorn --version 1.13.0 -n longhorn-system \
  --reuse-values --set metrics.serviceMonitor.enabled=true
helm template x "$CHART" -f values.yaml | kubeconform -strict -summary
helm history litellm -n ki-gateway; helm rollback litellm 1 -n ki-gateway

GPU

# Im Gast bzw. im Pod
nvidia-smi
nvidia-smi --query-gpu=name,driver_version,memory.used,memory.total --format=csv
nvidia-smi --query-gpu=name,mig.mode.current --format=csv
lsmod | grep ^nvidia; uname -r
# Kubernetes-Sicht
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
kubectl get node rke2-gpu-1 -L nvidia.com/gpu.product,nvidia.com/gpu.replicas
kubectl get runtimeclass
kubectl -n gpu-operator get pods
kubectl -n gpu-operator logs -l app=nvidia-operator-validator \
  -c nvidia-operator-validator --tail=3
# Device Plugin neu registrieren (nach Treiber-Reparatur)
kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset
# Time-Slicing-Profil pro Node
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config=l4-2 --overwrite
Baustein Aufgabe Prüfen mit
Treiber (Rolle nvidia_host) Kernel-Modul im Gast nvidia-smi per SSH
Container Toolkit Runtime nvidia in RKE2-containerd kubectl get runtimeclass
Device Plugin meldet nvidia.com/gpu Allocatable am Node
GFD Labels nvidia.com/gpu.* kubectl get node -L
DCGM Exporter Metriken :9400 promq DCGM_FI_DEV_GPU_UTIL

vLLM-Flags

Flag Default v0.31 Lab-Wert / Zweck
erstes Argument (Modell) – Qwen/Qwen3-8B-FP8; Entrypoint ist vllm serve
--served-model-name wie Modell qwen3-8b (Name für LiteLLM)
--max-model-len aus config.json (40960) 16384 bzw. 8192 – spart KV-Cache
--gpu-memory-utilization 0.92 chat 0.50, embed 0.12, code 0.33 (Summe < 1)
--max-num-seqs 256 (< 70 GiB GPU) 32 bzw. 16
--max-num-batched-tokens 2048 (< 70 GiB GPU) höher = Durchsatz, niedriger = TTFT
--kv-cache-dtype auto fp8 verdoppelt die Tokens im Cache
--enable-prefix-caching an System-Prompt und RAG-Kontext wiederverwenden
--runner pooling auto Embedding-Modelle (--task gibt es nicht mehr)
--enable-auto-tool-choice aus mit --tool-call-parser=hermes (Qwen)
--reasoning-parser – qwen3: <think> nach reasoning_content
--tensor-parallel-size 1 Modell über mehrere GPUs
--api-key / VLLM_API_KEY – schützt nur /v1/*, nicht /metrics
Umgebung Wirkung
HF_HOME=/models Hub-Cache auf dem PVC
HF_TOKEN (Secret hf-token) gated Modelle; ungültig = 401 auch bei offenen
HF_HUB_OFFLINE=1 nur aus dem Cache starten, kein Internet
VLLM_CPU_KVCACHE_SPACE KV-Cache in GiB beim CPU-Image
kubectl -n ki-models logs deploy/vllm-chat | grep -E 'KV cache size|concurrency'
kubectl -n ki-models logs deploy/vllm-chat --previous | grep -A3 ValueError
kubectl -n ki-models exec deploy/vllm-chat -- vllm bench serve --model qwen3-8b \
  --tokenizer Qwen/Qwen3-8B-FP8 --dataset-name random --random-input-len 1024 \
  --random-output-len 256 --num-prompts 200 --max-concurrency 48
python3 ~/ki-lab/code/vllm/kv-sizing.py Qwen/Qwen3-8B-AWQ 6.1
kubectl -n ki-models rollout undo deploy/vllm-chat

LiteLLM-API

API=https://llm.lab.internal      # kind: http://llm.ki.localtest.me
H=(-H "Authorization: Bearer sk-lab-master-1234" -H 'Content-Type: application/json')
curl -s $API/health/readiness | jq -c '{status, db}'
curl -s $API/v1/models "${H[@]}" | jq -r '.data[].id'
# Team mit Modellen, Budget (30 Tage) und Rate-Limit
curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-x",
  "models": ["chat-small"], "max_budget": 20, "budget_duration": "30d",
  "rpm_limit": 60}' | jq -r .team_id
curl -s $API/team/list "${H[@]}" | jq -r '.[] | "\(.team_alias) \(.spend)"'
curl -s "$API/team/info?team_id=$T" "${H[@]}" | jq -c '.team_info'
curl -s $API/team/update "${H[@]}" -d "{\"team_id\": \"$T\", \"max_budget\": 50}"
# Virtual Key: Modelle, Budget, Limits, Laufzeit, Routen, kein Fallback
curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$T\",
  \"key_alias\": \"x\", \"models\": [\"chat-small\"], \"max_budget\": 5,
  \"rpm_limit\": 10, \"duration\": \"30d\", \"allowed_routes\":
  [\"/v1/completions\"], \"metadata\": {\"disable_fallbacks\": true}}"
curl -s $API/key/info -H "Authorization: Bearer $KEY" | jq -c '.info'
curl -s $API/key/delete "${H[@]}" -d "{\"keys\": [\"$KEY\"]}"
# Spend und Fallback-Test
curl -s "$API/spend/logs?summarize=false" "${H[@]}" | jq -c '.[0]'
curl -s -D - $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
  "mock_testing_fallbacks": true, "messages": [{"role": "user",
  "content": "OK"}]}' -o /dev/null | grep -i x-litellm-model-group
Status / Typ Bedeutung
401 Key fehlt, ungültig oder Route nicht erlaubt
key_model_access_denied Modell nicht in der Key-/Team-Liste
429 rpm_limit/tpm_limit erreicht (Zähler in Redis)
422 budget_exceeded Budget von Key oder Team aufgebraucht (v1.104)
Header x-litellm-model-group welcher Alias wirklich geantwortet hat

Open WebUI, Qdrant, Continue

# Open WebUI
curl -s https://chat.lab.internal/health
kubectl -n ki-chat logs open-webui-0 | grep -iE 'oauth|qdrant|error' | head
kubectl -n ki-chat exec owui-db-1 -c postgres -- psql -U postgres -d openwebui \
  -c 'select email, role from "user";'
# Qdrant (Port-Forward 6333, Header api-key)
Q=(-H 'api-key: lab-qdrant-key-1234')
curl -s localhost:6333/collections "${Q[@]}" | jq -r '.result.collections[].name'
curl -s localhost:6333/collections/open-webui_knowledge "${Q[@]}" \
  | jq '.result | {points_count, size: .config.params.vectors.size}'
curl -s -X POST localhost:6333/collections/open-webui_knowledge/snapshots "${Q[@]}"
# Continue: Konfiguration und Key
sed "s|__HOME__|$HOME|g" ~/ki-lab/code/code-llm/continue-config.yaml \
  > ~/.continue/config.yaml
printf 'LITELLM_CODE_KEY=%s\n' "$CODE_KEY" > ~/.continue/.env
Open-WebUI-Variable Wirkung
OPENAI_API_BASE_URL http://litellm.ki-gateway:4000/v1
WEBUI_SECRET_KEY signiert Sessions; fest, sonst Logout je Replika
ENABLE_PERSISTENT_CONFIG=False Env gewinnt immer über Admin-UI
RAG_OPENAI_API_BASE_URL Embeddings über LiteLLM statt api.openai.com
RAG_TOP_K, CHUNK_SIZE 5 Chunks, 1000 Zeichen
Continue-Schlüssel Wert
useLegacyCompletionsEndpoint true – FIM an /v1/completions
promptTemplates.autocomplete Qwen-FIM-Tokens fim_prefix, fim_suffix, fim_middle
requestOptions.caBundlePath ~/ki-lab/lab-ca.crt
apiKey ${{ secrets.LITELLM_CODE_KEY }} aus ~/.continue/.env

Backup (Tag 15)

velero backup-location get
velero schedule get
velero backup create pre-upgrade --from-schedule velero-ki-taeglich --wait
velero backup create probe --include-namespaces backup-test --wait
velero backup describe probe --details | grep -A3 'Pod Volume'
velero restore create --from-backup probe --wait
velero restore create ca --from-backup vor-wettkampf --include-namespaces \
  cert-manager --include-resources secrets --existing-resource-policy update

PromQL

# GPU (DCGM)
avg by (Hostname) (DCGM_FI_DEV_GPU_UTIL)
sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
max by (Hostname) (DCGM_FI_DEV_XID_ERRORS) > 0
sum(kube_node_status_allocatable{resource="nvidia_com_gpu"})
# vLLM
sum by (model_name) (rate(vllm:generation_tokens_total[1m]))
histogram_quantile(0.95, sum by (le, model_name)
  (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum by (model_name) (vllm:num_requests_waiting)
max by (model_name) (vllm:kv_cache_usage_perc)
sum(rate(vllm:prefix_cache_hits_total[5m]))
  / sum(rate(vllm:prefix_cache_queries_total[5m]))
# LiteLLM
sum by (model_group) (rate(litellm_proxy_total_requests_metric_total[5m]))
sum by (team_alias) (increase(litellm_spend_metric_total[24h]))
max by (litellm_model_name) (litellm_deployment_state)    # 0 ok, 1 teil, 2 weg
sum by (fallback_model) (increase(litellm_deployment_successful_fallbacks_total[1h]))
# Plattform
max by (volume) (longhorn_volume_robustness)              # 1 ok, 2 degraded, 3 faulted
certmanager_certificate_expiration_timestamp_seconds - time()
time() - velero_backup_last_successful_timestamp{schedule="ki-taeglich"}

Formeln

Größe Formel Beispiel
Gewichte Parameter × Bytes je Gewicht (BF16 2, FP8 1, AWQ ≈ 0,5) 8,2 Mrd. × 2 B = 15,3 GiB
KV-Cache pro Token 2 × Layer × KV-Heads × head_dim × Bytes Qwen3-8B: 2×36×8×128×2 = 144 KiB
VRAM Gewichte + Tokens × KV/Token + ~1,5 GiB ≤ GPU × util 9,4 GB + KV + 1,5 ≤ 24 × 0,8
Volle Sequenzen KV-Budget / (max-model-len × KV/Token) 8,3 GB / (16384 × 144 KiB) ≈ 3
Decode-Obergrenze Speicherbandbreite / Gewichte (pro Stream) L4: 300 GB/s / 9,4 GB ≈ 32 Tokens/s
Little’s Law L = λ × W (parallel = Rate × Dauer) 2 req/s × 8 s = 16 → max-num-seqs ≥ 16

Little’s Law rückwärts: Bei --max-num-seqs=32 und 8 s pro Antwort schafft die Engine höchstens 32 / 8 = 4 Requests pro Sekunde. Kommen mehr, wächst vllm:num_requests_waiting – das Skalierungssignal aus Tag 11.