> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting & FAQ

> Common setup problems and short answers to frequent questions.

## Troubleshooting

> **Pod stuck in ImagePullBackOff**
>
> The image can't be pulled: the registry pull secret is missing from the namespace, or its token
> can't read the package. Confirm the `imagePullSecrets` name in your values matches a secret that
> exists in the namespace, and that its token has `read:packages` (the operator's pull secret
> works). Recreate it with:
>
> ```bash theme={null}
> kubectl create secret docker-registry tensormesh-registry -n tensormesh-platform \
>   --docker-server=artifacts.tensormesh.ai --docker-username=<you> --docker-password=<token>
> ```

> **Can't open the dashboard / connection refused**
>
> The install doesn't expose the UI outside the cluster: the pod is fine, it's just not reachable
> yet. Start a port-forward and **keep the terminal open**:
>
> ```bash theme={null}
> kubectl -n tensormesh-platform port-forward svc/tmo-ui-tensormesh-operator-ui 8080:80
> ```
>
> then open [http://localhost:8080](http://localhost:8080). For shared team access, enable the
> ingress behind SSO instead. See [Access & Security](/v1.0.1/ui/access-and-security).

> **"No LMCacheEngines found"**
>
> `engineNamespace` doesn't match the namespace where your engines actually run.

> **Numbers sit at 0 (charts or GPU utilization)**
>
> The engines or GPUs are idle: utilization and hit rate only move under inference traffic.
> (GPU *memory* stays allocated, so it can be non-zero while utilization reads 0.) If there
> *is* traffic and charts are still flat, check that `prometheus.url` is right.

> **Trend charts empty, but Health and Quotas work**
>
> The engine metrics aren't in the Prometheus the UI reads. The most common cause: the
> engine's metrics endpoint was never enabled (it's off by default). Set
> `engine.spec.prometheus.enabled: true` and add the `ServiceMonitor` so Prometheus scrapes
> it, both shown in
> [Configuration](/v1.0.1/ui/configuration#trend-charts-connect-your-prometheus). Also
> confirm `prometheus.url` points at the right Prometheus
> (`kubectl get svc -A | grep -i prometheus`).

> **Per-node charts collapse into one "Engine-wide" line**
>
> Fleet-wide totals are still correct; the UI just can't tell which node each metric came
> from, usually because your metrics collector runs as one shared service instead of
> per-node. Ask the Tensormesh team for the collector setting that restores per-node
> attribution.

> **No per-GPU cards on the Fleet Map**
>
> GPU telemetry isn't connected: set `dcgmService` to your exporter's service (find it
> with `kubectl get svc -A | grep -i dcgm`; see
> [Configuration](/v1.0.1/ui/configuration)). Without it, node cards show one row per
> node; everything else is unaffected.

> **GPU trend charts empty (but per-GPU cards work)**
>
> The over-time GPU charts read history from Prometheus, so Prometheus has to be **scraping
> DCGM**; the live cards only need `dcgmService`. Check it: query `DCGM_FI_DEV_GPU_UTIL` in
> Prometheus and expect series back. If it's empty, the exporter's `ServiceMonitor` is often
> labelled for a different Prometheus (some managed clusters, e.g. CoreWeave, use
> `environment: grafana-monitoring` instead of `release: <your-release>`). Add a
> `ServiceMonitor` for the DCGM service with the label your Prometheus selects. See
> [Configuration](/v1.0.1/ui/configuration#per-gpu-detail-connect-gpu-telemetry-gpu-clusters).

> **A second cluster shows nothing (multiple clusters)**
>
> Cluster access is automatic only for the cluster the UI runs **in**. Each extra `clusters` entry
> needs its own `apiUrl` and an access token, supplied through a Kubernetes Secret wired in with
> the chart's `extraEnvFrom`, not in your values file. See
> [Configuration](/v1.0.1/ui/configuration#multiple-clusters).

***

## FAQ

> **Can I change quotas, restart pods, or clear the cache from the UI?**
>
> No. The UI observes only, by design. It names the failing component and the fix
> (Health → Issues & Fixes); you act via [`tmo-cli`](/v1.0.1/reference/cli), `kubectl`, or
> Helm.

> **What load does it put on my cluster?**
>
> A handful of lightweight read requests per open page every 30 seconds, paused when the tab
> is in the background. Nothing runs on your GPU nodes.

> **What happens when a data source is down?**
>
> It depends on what's unreachable. If the **control plane** can't be reached, the monitoring
> pages (Overview, Fleet Map, Metrics, Health) show an explicit error (*"The control plane
> didn't respond"*) with a **Retry** button, never zeros or mock data. If the cluster is
> simply empty or idle, you get real zeros and a "No LMCacheEngines found" banner. If only
> metric **history** is missing (Prometheus not connected), the trend charts are empty while
> everything else works.

***

## Related

* [Configuration](/v1.0.1/ui/configuration)
* [Installation](/v1.0.1/ui/installation)
* [Operator Installation Troubleshooting](/v1.0.1/installation/troubleshooting)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.