One eight-GPU H100 server in a published power-measurement study drew a median 1.86 kW while idle. No training result was being produced, yet the node still had a substantial electrical load.
That figure belongs to one tested configuration, not a universal H100 idle specification. It makes the underlying distinction unusually clear: a GPU activity counter describes one kind of work, while the electricity meter sees an entire machine kept ready to do it.
The GPU can stop executing model kernels without the server—or even the accelerator board—becoming electrically off.
What the zero on the dashboard actually says
NVIDIA defines GPU utilisation in nvidia-smi as the fraction of the sampling interval during which at least one kernel was executing. That is an activity measurement. It is not a measure of the fraction of the server’s power budget being consumed.
Memory occupancy is a different observation again. A model can remain allocated in device memory between requests without executing a kernel. The amount allocated does not, by itself, reveal the watts used to retain it.
A single zero sample also says little about a longer interval with intermittent work. The sampling window, timestamp and scope of the reading matter. None of these counters is a substitute for measuring electrical input.
The powered machine around the accelerator
The CPU still runs an operating system and serving software. Host memory retains data. Network links remain available for the next request. Storage controllers, management electronics and power conversion remain part of the system. Fans respond to the thermal policy and the components that need cooling, rather than to one GPU-utilisation field.
The researchers behind Empirical Measurements of AI Training Power Demand on a GPU-Accelerated Node specifically caution that supporting CPUs, memory, interconnects and fans vary between systems. Their node contained eight 80 GB H100 GPUs, AMD EPYC processors and 1.5 TB of host memory. An idle value measured there cannot be assigned to a smaller inference box or another hardware generation.
There is a boundary to preserve even within that experiment. The authors measured node electrical input, while the rack’s rear-door heat exchanger was powered separately. Their idle figure is not a complete data-center electricity total.
A measured baseline that does not vanish
Read the meter that matches the question
A board-power sensor and a server-input meter have different responsibilities. NVIDIA’s documentation distinguishes GPU and module power telemetry, with support depending on the product. The field name and hardware determine which components are included.
At the server level, Dell’s iDRAC telemetry documentation illustrates how system power can be observed alongside other operating data. A metered rack outlet provides another electrical boundary. Either needs to be understood before comparing it with GPU readings.
Adding board values and calling the result “server power” omits whatever lies outside those sensors. Adding a facility-overhead estimate on top of a meter that already includes that overhead counts it twice. The measurement boundary has to travel with the number.
A useful investigation therefore aligns system-input power, accelerator telemetry and request activity over the same interval. It then separates a genuinely quiet serving period from CPU preprocessing, network waiting or bursts that an isolated snapshot missed.
A small baseline becomes energy when it lasts
Power is a rate. Energy also includes duration. A server drawing a steady baseline for an entire quiet night consumes energy even when its output counter barely changes.
This calculation assumes that the study’s idle median remains constant for all 16 hours. It is not a measured overnight result, and it excludes separately powered cooling and other facility loads. Its purpose is to show why the duration of an idle state matters alongside its wattage.
The arithmetic also explains why a system that finishes useful work sooner is not automatically more efficient over the whole day. The answer depends on what happens afterwards: another task runs, the service waits at its ready baseline, or the equipment enters a lower-power state.
“No requests” and “asleep” are separate software states
Serving software often keeps a model available to reduce the delay before the next answer. Reclaiming that state is an explicit operation, not a consequence of the request queue becoming empty.
vLLM’s Sleep Mode is a concrete example. Its documented level-one behaviour moves model weights to CPU memory and discards the KV cache. That releases GPU memory while keeping the server process available. It also requires sufficient host memory and a wake-up path.
Releasing GPU memory is not the same as cutting power to the board or the chassis. Any electricity saving has to be measured on the relevant hardware. The feature’s documented memory behaviour does not establish a universal wattage reduction.
Another option is consolidating compatible work onto fewer ready machines. Continuous batching explains one way of sharing a serving GPU, but admission and latency still matter: a large incoming prompt can delay other requests. Packing work together is useful only within the service’s response-time requirements.
The operational question is how much electricity it costs to remain ready, and how quickly that readiness must be restored. A zero GPU-utilisation reading answers neither. Measuring the quiet baseline—and choosing what the system does during long quiet intervals—turns “idle” from a dashboard label into an energy decision.
Sources
- Empirical Measurements of AI Training Power Demand on a GPU-Accelerated Node — measured configuration, node-power medians and study limits, December 2024.
- NVIDIA: nvidia-smi Documentation — utilisation definitions and power-telemetry scope.
- Dell Technologies: PowerEdge Server Telemetry — system-level power observation.
- vLLM: Sleep Mode — explicit memory release and resumption.







