Runtime GPU Telemetry · Closed-Loop Power Management
HBMGuard is a bare-metal C++ agent that detects GPU thermal throttling in real-time, cuts power draw by hundreds of watts, and automatically drains & restores Slurm nodes — without touching your application code.
HBM thermal throttling degrades compute silently. Your dashboards show "GPU busy" while SM Clock quietly drops 30–50%. The bill doesn't change. The throughput does.
Standard monitoring tools (node_exporter, nvidia-smi) expose power and temperature. They miss the SM occupancy vs. DRAM_ACTIVE dissonance that defines the Memory Wall — the actual throttle trigger.
Even with perfect local telemetry, a throttled node keeps receiving new jobs from the scheduler. The thermal event compounds. Eventually a human gets paged — at 2am — to run scontrol drain. HBMGuard closes this loop automatically, before the page goes out.
The agent samples SM occupancy, DRAM_ACTIVE bandwidth, and power draw every ~2 seconds using DCGM's Profiling API — bypassing NVML's limitations. A Memory Wall condition is flagged when DRAM_ACTIVE is saturated while SM efficiency collapses, creating the characteristic power-vs-work dissonance.
On detection, the C++ ECO Controller issues an NVML power cap command. The power cut forces the GPU to operate within its thermal envelope without interrupting the running workload. This is confirmed live on a GCP A100 Spot instance with Grafana validation.
If SM Clock degradation persists across 3 consecutive samples, the agent issues a Slurm drain with a machine-readable reason string. New jobs are blocked. The 3-sample buffer eliminates false positives from transient spikes — a deliberate tradeoff tuned from live workload data.
Recovery applies the same 3-sample logic in reverse. Three consecutive healthy SM Clock readings trigger an automatic state=resume. The node rejoins the cluster pool without human intervention. The full IDLE → DRAIN → IDLE state machine is validated and running.
Early Access · Pilot Program
We're working with a small number of AI inference providers and HPC operators to validate HBMGuard on their GPU fleets. If you're running A100 / H100 / H200 nodes on Slurm and you're tired of thermal surprises, let's talk.