> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# E2E Quickstart

> Install Tensormesh Platform, point vLLM at the engine, run an inference, and verify the KV cache is reused.

## Step 1: Install Tensormesh Platform

Follow [Install with Helm](/v1.0.1/installation/helm.md)

> [!NOTE]
>
> If you already run your own vLLM Deployment, do not replace it with this demo manifest.
> Use [Modify an Existing Deployment](/v1.0.1/installation/existing-deployment.md) for the
> minimum patch set instead.

## Step 2: Deploy vLLM connected to the engine

The chart creates a ConfigMap named `<engine>-connection` holding the
`kv-transfer-config.json` that tells vLLM how to reach the engine over the MP connector.

### Create a namespace for vLLM

Run vLLM in its own namespace and keep the platform in `tensormesh-platform`. The connector
config below reaches the engine by its full in-cluster DNS name, so it works from any
namespace:

```bash theme={null}
kubectl create ns inference
```

### Read the connector config from the engine's ConfigMap

```bash theme={null}
kubectl get cm tensormesh-platform-default-engine-connection \
  -n tensormesh-platform \
  -o jsonpath='{.data.kv-transfer-config\.json}' | jq -c .
```

You should get one line like this:

```json theme={null}
{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-platform-default-engine.tensormesh-platform.svc.cluster.local","lmcache.mp.isolated_ipc":true,"lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}
```

### Create the Deployment

Paste that line as the `--kv-transfer-config` value in `vllm-demo.yaml`:

```yaml vllm-demo.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-demo
  namespace: inference
  labels: { app: vllm-demo }
spec:
  selector: { matchLabels: { app: vllm-demo } }
  template:
    metadata:
      labels: { app: vllm-demo }
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.6   # use the same version as the lmcache engine
          env:
            - { name: PYTHONHASHSEED, value: "0" }
          args:
            - Qwen/Qwen3-0.6B
            - --gpu-memory-utilization
            - "0.6"
            - --no-enable-prefix-caching  
            - --max-model-len
            - "32768"
            - --kv-transfer-config
            # the single line from the previous step
            - '{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-platform-default-engine.tensormesh-platform.svc.cluster.local","lmcache.mp.isolated_ipc":true,"lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}'
          ports: [{ name: http, containerPort: 8000 }]
          resources:
            limits:   { nvidia.com/gpu: "1", memory: 32Gi }  
            requests: { cpu: "2", memory: 6Gi }
          readinessProbe:                  
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 40
---
apiVersion: v1
kind: Service
metadata:
  name: inference
  namespace: inference
spec:
  selector: { app: vllm-demo }
  ports: [{ name: http, port: 8000, targetPort: http }]
```

Apply and wait for the Pod to become Ready (5–10 min on first boot for image pull + model download):

```bash theme={null}
kubectl apply -f vllm-demo.yaml
kubectl wait --for=condition=Ready pod -l app=vllm-demo \
  -n inference --timeout=15m
```

## Step 3: Fire two identical requests, see the cache work

The simplest possible "is LMCache working?" test is to send two identical long prompts.
The first populates the cache; the second should reuse the stored KV blocks.

Port-forward the service in one terminal:

```bash theme={null}
kubectl port-forward -n inference svc/inference 8000:8000 --address 127.0.0.1
```

> [!NOTE]
>
> If port `8000` is already in use locally, forward to a different local port (for example
> `... 8001:8000`) and use that port in the requests below.

In a second terminal, build the request once into a variable, then fire it twice. Keeping
the JSON in a `PAYLOAD` variable (rather than inlining `$(jq …)` inside a multi-line
`curl`) makes it safe to copy-paste and guarantees both calls send the **identical**
prompt:

```bash theme={null}
KV_PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 52)")
PAYLOAD=$(jq -n --arg p "$KV_PROMPT" \
  '{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}')

# Call 1: cold, should STORE KV into LMCache
time curl -sS http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'

# Call 2: identical, should RETRIEVE prefix KV from LMCache
time curl -sS http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'
```

Then check the engine logs for store and retrieve markers:

```bash theme={null}
# select by the engine CR's name; the component label (cache-engine) also matches
# CacheBlend engine pods when those are enabled
kubectl logs -n tensormesh-platform --tail=200 \
  -l app.kubernetes.io/instance=tensormesh-platform-default-engine \
  | grep -E "Stored [0-9]+ tokens|Prefetch request completed.*retained keys"
```

A working install logs `Stored N tokens` after call 1 and
`Prefetch request completed ... retained keys` during call 2.

> **What a successful run looks like**
>
> Sample output from a Qwen3-0.6B install, prompt repeated 52 times for 521 tokens (two
> 256-token chunks):
>
> ```text theme={null}
> ==========================================
>  CALL 1 (cold — expect STORE)
> ==========================================
> usage: {'prompt_tokens': 521, 'completion_tokens': 20, 'total_tokens': 541}
>
> ==========================================
>  CALL 2 (warm — expect RETRIEVE)
> ==========================================
> usage: {'prompt_tokens': 521, 'completion_tokens': 20, 'total_tokens': 541}
>
> ==========================================
>  ENGINE LOG MARKERS
> ==========================================
> LMCache INFO: Stored 512 tokens in 0.001 seconds
> LMCache INFO: Prefetch request completed (L1+L2): 2/2 retained keys (2 L1, 0 L2) in 1.0 ms
> ```
>
> The engine reports `2/2 retained keys` from L1, meaning the second request reused the KV
> blocks stored during the first. On a prompt this short both calls take about the same time;
> the speedup shows on long prompts, as in the benchmark below. vLLM's own metrics line
> confirms the hit from the inference side:
>
> ```text theme={null}
> External prefix cache hit rate: 49.1%
> ```
>
> (The counter is cumulative across both calls: call 1 hit 0 of 521 tokens, call 2 hit 512
> of 521, so 512/1042 = 49.1%. For a two-call test \~50% is the ceiling, not a partial hit.)

> [!TIP]
>
> Prompts must exceed the engine's `chunk_size` (256 tokens by default) for any KV blocks
> to be stored. Short prompts produce no markers. That's expected, not a failure.

## Step 4: Run the benchmark

For a more realistic measurement than two curls, use `vllm bench serve` to compare a cold
pass against a warm pass. Run **two passes with the same `--seed`** so the prompts repeat
and the second pass hits the cache.

Rather than installing the vLLM CLI locally, run it **inside the cluster as a `Job`**. The
vLLM image already ships the CLI, talks to the `inference` Service directly (no
port-forward), and a `Job` runs it once to completion and stops. The benchmark is a pure
HTTP client, so it needs **no GPU**.

Save as `benchmark-job.yaml`:

```yaml benchmark-job.yaml theme={null}
apiVersion: batch/v1
kind: Job
metadata:
  name: lmcache-benchmark
  namespace: inference
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels: { app: lmcache-benchmark }
    spec:
      restartPolicy: Never
      containers:
        - name: bench
          image: lmcache/vllm-openai:v0.5.6   # same tag as the Deployment, so the node's cached image is reused
          imagePullPolicy: IfNotPresent
          env:
            - { name: HF_HOME, value: /tmp/hf }   # writable under OpenShift's random UID
            - { name: HF_HUB_DISABLE_TELEMETRY, value: "1" }
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              run() {
                vllm bench serve --backend openai --base-url http://inference:8000 \
                  --model Qwen/Qwen3-0.6B --dataset-name random \
                  --num-prompts 20 --random-input-len 20480 \
                  --random-output-len 1 --seed 32
              }
              echo "### PASS 1 (cold) ###"; run
              sleep 5
              echo "### PASS 2 (warm) ###"; run
          volumeMounts:
            - { name: hf, mountPath: /tmp/hf }
          resources:
            requests: { cpu: "1", memory: 2Gi }
            limits:   { cpu: "2", memory: 4Gi }
      volumes:
        - { name: hf, emptyDir: {} }
```

Run it and stream the results:

```bash theme={null}
kubectl apply -f benchmark-job.yaml
kubectl wait --for=condition=complete job/lmcache-benchmark \
  -n inference --timeout=10m
kubectl logs -f job/lmcache-benchmark -n inference
```

The logs contain two `Serving Benchmark Result` blocks. A working cache shows **pass 2 with
much lower TTFT** (mean/p50/p99). The speedup scales with `--random-input-len`: with a fixed
`--seed`, both passes send the same 20 prompts, so each unique body is prefilled on the cold
pass and served from cache on the warm pass. Bigger `--random-input-len` → bigger delta.

Example result from a working run with `--num-prompts 20`, `--random-input-len 20480`, and
`--seed 32`:

```text theme={null}
### PASS 1 (cold) ###
Successful requests:                     20
Benchmark duration (s):                  11.00
Request throughput (req/s):              1.82
Mean TTFT (ms):                          6399.80
Median TTFT (ms):                        6456.31
P99 TTFT (ms):                           10896.55

### PASS 2 (warm) ###
Successful requests:                     20
Benchmark duration (s):                  5.64
Request throughput (req/s):              3.55
Mean TTFT (ms):                          3697.56
Median TTFT (ms):                        3753.21
P99 TTFT (ms):                           5584.58
```

In the healthy case above, pass 2 cuts TTFT substantially relative to pass 1 and nearly
doubles request throughput.

> [!NOTE]
>
> Keep `num_prompts × random-input-len` of KV under `engine.spec.l1.sizeGB` (else pass 1's
> entries evict before pass 2 reads them, causing warm misses), and `random-input-len + random-output-len ≤   --max-model-len` (32768 in the Deployment above).

Clean up the Job (it also self-deletes after 1h via `ttlSecondsAfterFinished`):

```bash theme={null}
kubectl delete job lmcache-benchmark -n inference
```

## Next steps

> **[Install with Helm](/v1.0.1/installation/helm.md)**
>
> Full chart reference, install modes, every tunable value.

> **[Configuration](/v1.0.1/reference/configuration.md)**
>
> Every `values.yaml` key, with example overlays.

> **[Observability](/v1.0.1/observability/metrics.md)**
>
> Metrics, dashboards, and performance tuning.

> **[Troubleshooting](/v1.0.1/installation/troubleshooting.md)**
>
> `Pending` pods, image pull, hung uninstall, ownership conflicts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.