Anhang A: Spickzettel
Namen, Ports und Werte entsprechen dem Lab (Tag 0–15). Keys und
Passwörter sind Lab-Werte. Versionen wie im Buch getestet: go 1.26,
python 3.14, kind 0.33, kubectl 1.37, helm 4.3, tofu 1.12, Ollama
0.40.2, Prometheus 3.15.
Namen und Ports
| Komponente |
Ort |
Port |
Tag |
| llm-gateway |
llm-system, VM, Docker |
8080 (pprof 6060) |
4, 7 |
| Ollama |
llm-vm, llm-system/ollama |
11434 |
5, 7 |
| vLLM |
GPU-VM, vllm-ministral |
8000 |
5, 7 |
llama.cpp llama-server |
llm-vm |
8082 |
5 |
| Prometheus / Alertmanager |
monitoring |
9090 / 9093 |
8, 10 |
Grafana (admin/lab-admin) |
monitoring |
3000 (Service 80) |
8 |
| Loki / Tempo |
monitoring |
3100 / 3200 |
9 |
| OTel Collector (OTLP) |
otel-collector.monitoring |
4317 gRPC, 4318 HTTP |
9 |
| node-exporter / DCGM |
llm-vm / GPU-Node |
9100 / 9400 |
8 |
| Registry Signatur / Kunde |
Laptop |
5055 / 5050 (intern 5000) |
13, 14 |
| OpenBao (dev) |
Laptop |
8200 |
13 |
| Gateway-Env |
Default |
Bedeutung |
LISTEN_ADDR |
:8080 |
Listen-Adresse |
UPSTREAM_URL |
http://127.0.0.1:11434 |
Modellserver |
TENANT_KEYS |
– |
acme:lab-key-acme,globex:lab-key-globex |
RATE_LIMIT_RPS / _BURST |
5 / 10 |
Token-Bucket pro Mandant und Pod |
OTEL_EXPORTER_OTLP_ENDPOINT |
leer = aus |
Tracing (Tag 9) |
| HTTP vom Gateway |
Bedeutung |
Erste Prüfung |
| 401 |
Key unbekannt (tenant="unknown") |
Key, Secret llm-gateway-keys |
| 429 |
Rate-Limit, Retry-After: 1 |
Last pro Mandant, Replikas × RPS |
| 500 |
Fehler vom Modellserver |
Modellserver-Log, OOM |
| 502 |
Upstream nicht erreichbar |
DNS, NetworkPolicy, Endpoints |
| 503 |
/readyz ohne Upstream, oder Upstream voll |
Modellserver, Kapazität |
Linux: Prozesse, Signale, systemd
# Prozess durchleuchten
grep -E '^(Name|State|PPid|Threads|VmRSS)' /proc/$PID/status
ls -l /proc/$PID/fd | wc -l; grep 'open files' /proc/$PID/limits
tr '\0' ' ' < /proc/$PID/cmdline; echo
sudo prlimit --pid $PID --nofile=65536:65536
# Syscalls: gezielt, kurz, mit Dauer; Zusammenfassung
sudo strace -f -T -p $PID -e trace=accept4,connect,sendto,recvfrom,openat
strace -c -f curl -s localhost:8080/healthz
# Signale: Exit = 128 + Nummer (143 TERM, 137 KILL)
kill -STOP $PID; kill -CONT $PID; kill -TERM $PID; wait $PID; echo $?
# Namespaces
sudo lsns -t net; sudo nsenter -t $PID --net --uts ip -br addr
sudo unshare --pid --fork --mount-proc --net bash
# systemd
systemctl status ollama --no-pager
systemctl cat ollama # Unit + alle Drop-ins
systemctl show ollama -p MemoryMax -p NRestarts -p DropInPaths
sudo systemctl edit ollama # Drop-in anlegen
sudo systemctl set-property --runtime ollama MemoryMax=6G
sudo systemctl daemon-reload && sudo systemctl restart ollama
systemd-analyze security ollama | tail -1
journalctl -u ollama --since -10min -p warning --no-pager
journalctl -k | grep -i 'out of memory' # Kernel-OOM
sudo systemd-cgtop; systemd-cgls -u ollama.service
cgroups v2 und PSI
CG=/sys/fs/cgroup$(systemctl show -p ControlGroup --value ollama)
cat $CG/memory.max $CG/memory.high $CG/memory.current $CG/memory.peak
grep -E '^(high|max|oom_kill) ' $CG/memory.events # Drosselung, Limit, Kills
cat $CG/cpu.max # "50000 100000" = 0,5 CPU
grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat # CPU-Throttling
cat $CG/memory.pressure $CG/cpu.pressure # PSI pro cgroup
head -1 /proc/pressure/{cpu,memory,io} # PSI global
cat /proc/loadavg; vmstat 1 3; top -b -n1 -o %CPU | head -12
| systemd |
cgroup-Datei |
Wirkung |
MemoryHigh= |
memory.high |
drosseln (Reclaim), kein Kill |
MemoryMax= |
memory.max |
OOM-Kill in der cgroup |
CPUQuota=50% |
cpu.max |
Throttling, sichtbar als Latenz |
TasksMax= |
pids.max |
Fork-Grenze |
Netzwerk und TLS
ss -tlnp; ss -tnp state established '( dport = :11434 )'; ss -s
ip -br addr; ip route get 1.1.1.1; ip -s link
dig +short api.mistral.ai; dig +search +showsearch api.mistral.ai
getent hosts api.mistral.ai # wie die Anwendung (libc, /etc/hosts, search)
resolvectl status
# curl zerlegt Latenz (Werte kumulativ); -N = Streaming ohne Puffer
curl -s -o /dev/null -w 'dns=%{time_namelookup} conn=%{time_connect} \
tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' URL
curl -sN URL # SSE-Stream direkt sehen
curl --resolve llm-vm.lab:8443:127.0.0.1 https://llm-vm.lab:8443/
# Pakete
sudo tcpdump -i any -nn 'tcp port 11434' -w cap.pcap; tcpdump -r cap.pcap -nn -A
# Firewall und conntrack
sudo nft list ruleset; sudo conntrack -L -p tcp --dport 8080; sudo conntrack -C
cat /proc/sys/net/netfilter/nf_conntrack_{count,max}
# Latenz/Verlust/MTU
sudo tc qdisc add dev lo root netem delay 100ms loss 1%; sudo tc qdisc del dev lo root
ping -c1 -M do -s 1372 10.99.0.2 # 1372 + 28 = 1400 Byte
# TLS
echo | openssl s_client -brief -connect HOST:443 -servername HOST
echo | openssl s_client -connect HOST:443 2>/dev/null | openssl x509 -noout \
-subject -issuer -enddate -ext subjectAltName
openssl x509 -in crt -noout -checkend $((14*86400)) || echo "läuft bald ab"
openssl verify -CAfile ca.crt crt
| curl-Exit |
Bedeutung |
Typische Ursache |
| 6 |
Host nicht auflösbar |
DNS, ndots, Suchdomain |
| 7 |
Verbindung abgelehnt |
nichts lauscht, Service ohne Endpoints, reject |
| 18 |
Antwort unvollständig |
Stream beim Rollout gekappt |
| 28 |
Timeout |
drop, NetworkPolicy, Überlast |
| 52 / 56 |
leere Antwort / Reset |
Prozess beim Antworten gestorben |
| 60 |
Zertifikat nicht vertraut |
abgelaufen, falsche CA, Name nicht im SAN |
kubectl: Mandanten, Probes, Debugging
kubectl config current-context # kind-sre-lab
kubectl get ns -l sre-lab.io/role -L sre-lab.io/role,sre-lab.io/tenant
kubectl auth can-i create deployments -n tenant-acme --as=anna --as-group=team-acme
kubectl auth can-i --list -n tenant-acme --as=anna --as-group=team-acme
kubectl -n tenant-acme describe quota tenant-quota
kubectl -n tenant-acme get netpol,limitrange,rolebinding
kubectl label --dry-run=server --overwrite ns tenant-acme \
pod-security.kubernetes.io/enforce=restricted # PSA vorher prüfen
kubectl -n tenant-acme create token ci-deployer --duration=10m
# Zustand und Ursachen
kubectl -n llm-system get pods -o wide; kubectl -n llm-system get events \
--sort-by=.lastTimestamp | tail
kubectl -n llm-system describe pod POD | grep -E 'Reason|Exit Code|probe|Killing'
kubectl -n llm-system logs deploy/llm-gateway --tail=50; kubectl logs POD --previous
kubectl -n llm-system get pod POD \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl -n llm-system get endpointslice -l kubernetes.io/service-name=llm-gateway
kubectl top pod -n llm-system --containers
# Debuggen ohne Shell im Image (distroless)
kubectl -n llm-system debug POD -it --image=curlimages/curl:8.22.0 --target=gateway
kubectl debug node/sre-lab-worker -it --image=ubuntu:24.04
kubectl -n llm-system port-forward svc/llm-gateway 8080:8080
# Rollouts
kubectl -n llm-system rollout status deploy/llm-gateway # auch: history, undo, restart
kubectl -n llm-system set env deploy/llm-gateway RATE_LIMIT_RPS=5
kubectl -n llm-system set image deploy/llm-gateway '*=llm-gateway:otel'
kubectl -n llm-system get hpa,pdb
kind load docker-image llm-gateway:dev --name sre-lab
| Probe |
Prüft |
Fehler führt zu |
startupProbe |
Start fertig (Modell geladen) |
Neustart; blockiert andere Probes |
livenessProbe |
Prozess lebt (/healthz) |
Neustart – nie den Upstream prüfen |
readinessProbe |
kann bedienen (/readyz) |
aus Endpoints entfernt, kein Neustart |
Helm
helm lint --strict CHART; helm template REL CHART -f values.yaml | kubeconform -strict -
helm template REL CHART --show-only templates/gateway-deployment.yaml
helm upgrade --install REL CHART -n NS -f values.yaml \
--rollback-on-failure --timeout 10m
helm history REL -n NS; helm rollback REL -n NS --wait # ohne Revision: vorherige
helm get values REL -n NS; helm get manifest REL -n NS; helm status REL -n NS
helm package CHART -d bundle/charts
helm push bundle/charts/llm-platform-0.1.0.tgz oci://127.0.0.1:5050/charts --plain-http
diff <(helm template REL CHART -f v.yaml) <(helm template REL CHART -f v.yaml \
--set modelServer.image.tag=0.40.3) # was ändert ein Upgrade?
Modellserver und Mistral-API
# Ollama
ollama list; ollama ps; ollama show ministral-3:3b; ollama show --modelfile acme-assist
ollama pull ministral-3:3b; ollama create acme-assist -f Modelfile; ollama stop MODEL
ollama run ministral-3:3b --verbose "Sag hallo" # zeigt Tokens/s
curl -s localhost:11434/api/version; curl -s localhost:11434/v1/models
# Wichtige Ollama-Umgebung (Drop-in)
# OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, OLLAMA_NUM_PARALLEL,
# OLLAMA_CONTEXT_LENGTH, OLLAMA_MAX_LOADED_MODELS,
# OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0 (halber KV-Cache)
# vLLM (GPU)
vllm serve mistralai/Ministral-3-8B-Instruct-2512 --tokenizer-mode mistral \
--config-format mistral --load-format mistral --max-model-len 32768 \
--gpu-memory-utilization 0.90 --enable-auto-tool-choice --tool-call-parser mistral
# weitere: --tensor-parallel-size 2, --max-num-seqs 128, --enable-prefix-caching,
# --kv-cache-dtype fp8, --served-model-name NAME; offline: HF_HUB_OFFLINE=1
curl -s localhost:8000/health; curl -s localhost:8000/metrics | grep '^vllm:' | head
# llama.cpp
llama-server -hf mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M -c 4096 -np 2 \
--jinja --metrics --port 8082
# GPU
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
# OpenAI-kompatibel: Ollama, vLLM, Gateway und La Plateforme
curl -s https://api.mistral.ai/v1/chat/completions \
-H "Authorization: Bearer $MISTRAL_API_KEY" -H 'Content-Type: application/json' \
-d '{"model":"mistral-small-latest","max_tokens":50,
"messages":[{"role":"user","content":"Was ist ein SLO?"}]}'
# Über das Gateway (Lab-Key)
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
-d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"hi"}]}'
| Request-Feld |
Zweck |
stream: true |
SSE, data: … bis data: [DONE] |
max_tokens |
begrenzt Dauer und Kosten |
tools, tool_choice |
Function Calling (auto, any,
none) |
response_format |
json_object oder json_schema |
llmctl und Gateway-Werkzeuge
export LLMCTL_API_KEY=lab-key-acme LLMCTL_URL=http://127.0.0.1:8080 LLMCTL_MODEL=mistral
llmctl chat --max-tokens 20 "Was ist ein Fehlerbudget?"
llmctl load -n 200 -c 20 [--rps 4] [--json] # p50/p95/p99, Fehlerquote
llmctl slo --objective 0.995 --window 28d [--tenant acme] # Exit 2 = Budget weg
pytest -q; ruff check . && ruff format --check .; pip wheel --no-deps -w dist .
go vet ./... && go test -race -count=1 ./...; gofmt -l .
go build -tags pprof -o gw-pprof . # pprof nur auf 127.0.0.1:6060
go tool pprof -top 'http://127.0.0.1:6060/debug/pprof/profile?seconds=3'
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" .
PromQL
# RED pro Mandant
sum by (tenant) (rate(gateway_requests_total[5m]))
sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
/ sum by (tenant) (rate(gateway_requests_total[5m]))
histogram_quantile(0.95,
sum by (tenant, le) (rate(gateway_request_duration_seconds_bucket[5m])))
sum by (tenant, type) (rate(gateway_tokens_total[5m]))
sum by (tenant) (rate(gateway_ratelimited_total[5m]))
# Recording Rules (Tag 8, 10)
tenant:gateway_requests:rate5m
tenant:gateway_request_errors:ratio_rate5m
tenant:gateway_request_duration_seconds:p95_5m
tenant:slo_errors:ratio_rate1h # 5m 30m 1h 2h 6h 1d 3d
# Monitoring prüft sich selbst
up{job="llm-gateway"} == 0 # Target da, Scrape scheitert
absent(up{job="llm-gateway"}) # Target verschwunden
count by (__name__) ({__name__=~"gateway_.*"}) # Kardinalität
# LLM-Serving und Hardware
histogram_quantile(0.95,
sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(vllm:num_requests_waiting) # Warteschlange, Basis für KEDA
max(vllm:kv_cache_usage_perc) # nahe 1 = Preemption
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
increase(node_vmstat_oom_kill{host="llm-vm"}[10m]) > 0
rate(container_cpu_cfs_throttled_periods_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])
# Burn Rate (Page): 1h- und 5m-Fenster über 14,4 × Budget
tenant:slo_errors:ratio_rate1h > (14.4 * 0.005)
and tenant:slo_errors:ratio_rate5m > (14.4 * 0.005)
| Falle |
Richtig |
| Counter roh grafisch |
immer rate()/increase() |
histogram_quantile ohne le |
sum by (…, le) |
Mittelwert _sum/_count fürs SLO |
Perzentil aus Histogramm |
| Mandant ohne 5xx fehlt |
(… or … * 0) wie in den SLO-Regeln |
user, prompt_id als Label |
in Logs/Traces, nie als Label |
LogQL und TraceQL
{namespace="llm-system", app="llm-gateway"} | json | status >= 500
{app="llm-gateway"} |= "error" | json | line_format "{{.time}} {{.err}}"
sum by (tenant) (count_over_time({app="llm-gateway"} | json | msg="request" [30m]))
sum by (tenant) (rate({app="llm-gateway"} | json | status >= 500 [5m]))
quantile_over_time(0.95, {app="llm-gateway"} | json | msg="request"
| unwrap duration_ms [30m]) by (tenant)
# TraceQL (Tempo)
{resource.service.name="llm-gateway" && span.tenant="acme" && duration > 500ms}
{resource.service.name="llm-gateway" && status=error}
{span.gen_ai.usage.output_tokens > 1000}
curl -s -G localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={app="llm-gateway"}' --data-urlencode since=30m
curl -s -G localhost:3200/api/search --data-urlencode 'q={status=error}'
curl -s localhost:3200/api/v2/traces/$TRACE_ID | jq .
Regeln, Alertmanager, On-Call
P="docker run --rm -v $PWD:/r -w /r --entrypoint promtool prom/prometheus:v3.15.0"
$P check rules llm-gateway-alerts.rules.yaml
$P test rules llm-gateway.test.yaml
curl -s localhost:8080/metrics | $P check metrics
A="docker run --rm -v $PWD:/c --entrypoint amtool prom/alertmanager:v0.34.1"
$A check-config /c/alertmanager.yaml
$A config routes test --config.file=/c/alertmanager.yaml severity=page tenant=acme
# gegen den laufenden Alertmanager (im Pod: amtool --alertmanager.url=...)
amtool alert query; amtool alert query --inhibited
amtool alert add alertname=Test severity=ticket tenant=acme
amtool silence add alertname=X tenant=globex --duration=2h --comment="OPS-123"
amtool silence query; amtool silence expire ID
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
| "\(.labels.job) \(.health) \(.lastError)"'
OpenTofu
tofu init; tofu fmt -check -recursive; tofu validate
tofu plan -out=tfplan; tofu apply tfplan
tofu plan -detailed-exitcode # 0 = gleich, 1 = Fehler, 2 = Drift/Änderung
tofu plan -refresh-only # nur Abweichungen Cluster ↔ State zeigen
tofu state list
tofu state show 'module.tenant["acme"].kubernetes_resource_quota_v1.this'
tofu state mv OLD NEW; tofu state rm ADDR # Refactoring, Übergabe
tofu output -json; tofu console
tofu plan -var 'tenants={hooli={team_group="team-hooli",gpu_quota=8}}'
tofu force-unlock LOCK_ID # nur wenn sicher kein Lauf aktiv
import { # Bestand übernehmen statt neu anlegen
for_each = var.tenants
to = module.tenant[each.key].kubernetes_namespace_v1.this
id = "tenant-${each.key}"
}
Ansible
cd ~/sre-lab/tag12/ansible-llm-vm
ansible-inventory -i inventory/hosts.ini --graph
ansible llm_vms -i inventory/hosts.ini -m ansible.builtin.ping
ansible-playbook site.yml --syntax-check; ansible-lint
ansible-playbook -i inventory/hosts.ini site.yml --check --diff # Drift ansehen
ansible-playbook -i inventory/hosts.ini site.yml --limit llm-vm \
-e llm_vm_ollama_memory_max=8G
ansible llm_vms -i inventory/hosts.ini -b -m ansible.builtin.apt \
-a 'name=libssl3t64 state=latest update_cache=true' # Ad-hoc-Patch
ansible-doc ansible.builtin.systemd_service
ansible-vault encrypt_string 'geheim' --name 'api_key' # Secret in Vars
Idempotenz-Beweis: zweiter Lauf changed=0. Check-Mode
taugt für eingerichtete Hosts, nicht für den Erstlauf.
Supply Chain und CVE-Response
syft scan llm-gateway:dev -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json
syft scan oci-archive:bundle/images/ollama_0.40.2.tar -o json | jq '.artifacts | length'
trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 llm-gateway:dev
trivy fs -q --scanners vuln .; trivy config -q k8s/; trivy sbom sbom.spdx.json
trivy image --vex vex.json llm-gateway:dev # not_affected unterdrücken
grype -q sbom:sbom.spdx.json --fail-on high; grype db status
go run golang.org/x/vuln/cmd/govulncheck@latest ./... # Exit 3 = betroffen
go run golang.org/x/vuln/cmd/govulncheck@latest -mode=binary ./llm-gateway
pip-audit -r requirements.txt
# EPSS und KEV
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2025-30204' | jq '.data[0]'
# Signieren (Lab-Key) und keyless prüfen
cosign generate-key-pair
cosign sign --key cosign.key "$R@$DIGEST"
cosign verify --key cosign.pub --insecure-ignore-tlog "$R@$DIGEST" # nur Lab
cosign attest --key cosign.key --type spdxjson --predicate sbom.spdx.json "$R@$DIGEST"
cosign verify-attestation --key cosign.pub --type spdxjson "$R@$DIGEST"
cosign verify --certificate-identity-regexp 'https://github.com/ORG/.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com IMAGE@DIGEST
cosign sign-blob --key cosign.key --bundle SUMS.sigstore.json SHA256SUMS
cosign verify-blob --key cosign.pub --bundle SUMS.sigstore.json SHA256SUMS
cosign tree IMAGE@DIGEST; cosign save IMAGE@DIGEST --dir sigs/
kyverno apply pod-guardrails.yaml -r pod.yaml; kyverno test .
| Frage |
Werkzeug |
Entscheidet |
| Was steckt drin? |
syft (SBOM) |
Inventur |
| Welche Lücken? |
trivy, grype, govulncheck |
Kandidaten |
| Wird der Code erreicht? |
govulncheck, /proc/PID/maps |
Dringlichkeit |
| Wird es ausgenutzt? |
EPSS, CISA KEV |
Frist (Stunden oder Tage) |
Air-Gap
skopeo copy --override-os linux --override-arch amd64 \
docker://docker.io/ollama/ollama:0.40.2 oci-archive:bundle/images/ollama_0.40.2.tar
skopeo inspect --raw docker://docker.io/ollama/ollama:0.40.2 \
| jq -c '.manifests[].platform'
docker save llm-gateway:dev -o bundle/images/llm-gateway_dev.tar
skopeo copy --dest-tls-verify=false docker-archive:bundle/images/llm-gateway_dev.tar \
docker://127.0.0.1:5050/llm-gateway:dev
crane catalog --insecure 127.0.0.1:5050; crane ls --insecure 127.0.0.1:5050/llm-gateway
crane digest --insecure IMAGE; crane config --insecure IMAGE | jq -r .architecture
(cd bundle && find . -type f ! -name SHA256SUMS | sort | xargs sha256sum > SHA256SUMS)
(cd bundle && sha256sum -c --quiet SHA256SUMS) && echo "✔ Bundle vollständig"
hf download mistralai/Ministral-3-8B-Instruct-2512 --revision SHA --local-dir DIR
Formeln
Fehlerbudget = 1 − SLO
Ausfallzeit = (1 − SLO) × Fenster
99,5 % / 28 d = 201,6 min 99,9 % / 30 d = 43,2 min
99,95 % / 30 d = 21,6 min 99,99 % / 30 d = 4,3 min
Budget übrig = 1 − Fehlerquote / (1 − SLO) (llmctl slo)
Burn Rate = Fehlerquote / (1 − SLO)
Budget leer nach = SLO-Fenster / Burn Rate (14,4 × bei 30 d = 2,1 Tage)
Verbrauch = Burn Rate × Alarmfenster / SLO-Fenster (14,4 × 1 h / 30 d = 2 %)
Faktoren 30 d = 14,4 / 6 / 3 / 1 (2 %, 5 %, 10 %, 10 % Budget)
Faktoren 28 d = 13,44 / 5,6 / 2,8 / 0,93
Gewichte = Parameter × Bytes/Gewicht
FP16/BF16 2 · FP8 1 · Q8_0 ≈ 1,06 · Q4_K_M ≈ 0,6
KV-Cache/Token = 2 (K+V) × Layer × KV-Heads × Head-Dim × Bytes (FP16: 2)
Ministral 3 3B: 2 × 26 × 8 × 128 × 2 = 104 KiB
Mistral Small 24B: 2 × 40 × 8 × 128 × 2 = 160 KiB
KV-Cache gesamt = KV/Token × Kontext × parallele Sequenzen
Speicher ≈ (Gewichte + KV-Cache) × 1,1 … 1,2
Beispiel 24B BF16, 32k × 4: 44,7 + 20,0 = 64,7 GiB → H100 80 GB oder FP8
Little's Law L = λ × W (gleichzeitige Anfragen = Ankunftsrate × Verweildauer)
Verweildauer W ≈ TTFT + Output-Tokens × Zeit pro Token
Beispiel 20 req/s × (0,5 s + 300 × 25 ms) = 20 × 8 s = 160 Sequenzen
160 × 2 000 Token × 160 KiB ≈ 49 GiB KV-Cache (24B-Modell)
Kapazität Slots = Slots / Bedienzeit (4 / 0,5 s = 8 req/s, Tag 11)
GPUs ≈ benötigte Tokens/s ÷ Tokens/s pro GPU (Lasttest) ÷ Zielauslastung
Rate-Limit eff. = Replikas × RATE_LIMIT_RPS (pro Mandant, Tag 7)
HPA = ceil(Replikas × Ist-Wert / Ziel-Wert)
Tail bei Fan-out P(mind. eine langsame von n) = 1 − 0,99^n (n = 100: 63 %)
Startup-Budget = periodSeconds × failureThreshold ≥ 2 × Ladezeit
Grace Period > preStop + Shutdown-Timeout + längste Generierung