Catching LLM 'fake alive' by burning real tokens

What it does

Relay Pulse tracks the health of LLM relay services and API gateways by sending real language-model requests and confirming the responses actually contain generated content. It surfaces the results in a web dashboard with availability heatmaps spanning 24-hour, 7-day, and 30-day windows. The backend stores metrics locally in SQLite or PostgreSQL, and configuration changes apply on the fly via fsnotify without restarting the service.

The interesting bit

Traditional uptime monitors call an endpoint, see HTTP 200, and call it a day—but LLM relays can return empty bodies or error payloads inside a success status code, a “fake alive” scenario. Relay Pulse forces the service to perform actual inference, using max_tokens: 1 to keep costs low while proving the model is really “spitting words.” The README estimates this burns roughly 30,000 tokens per day per monitored endpoint.

Key highlights

• Real-token probing: validates actual LLM output rather than mere network connectivity
• Availability heatmaps: 24h/7d/30d visual matrices to spot quality trends at a glance
• Dual storage backends: SQLite for single-node setups, PostgreSQL for Kubernetes clusters
• Hot config reloads via fsnotify; add or edit monitors without restarting
• Cloud-native packaging: minimal Docker image designed for horizontal scaling

Caveats

• Automated probes are subject to network jitter, geographic variance, and cache lag; the README explicitly warns that displayed status is a technical signal, not a definitive quality judgment
• The public demo site disclaims responsibility for the reliability, legality, or financial safety of any third-party providers shown

Verdict

A solid fit if you run or rely on external LLM gateways and need to catch silent failures that traditional uptime tools miss. If you just need generic HTTP heartbeat monitoring, it is probably overkill.