Sundar Pichai, Demis Hassabis and Google - THE PEOPLE INSIDE THE MACHINE Book 3 - Part 02

SPEAKER_1 (0:01)

Part 2 The Machine Room. Failure is the normal case.

In a machine room in California, in October 2003, one of the green lights on a rack of beige machines is dark. The machine under it has been dead for six hours.

The air has weight. It smells of hot dust and hot plastic. The two smells that rise off consumer machines when they have been run long enough that the temperature inside the box, and the temperature outside it, stopped being different. The sound is not a hum. A hum would mean the fans were moving together. And these fans are not moving together. They are thousands of small fans, each off by a few revolutions from its neighbor. The result is a texture that rises and falls at no interval the ear can hold.

Racks of beige boxes stretch both directions, assembled from parts. Motherboards without cases, discs bought in bulk, power supplies replaced from a shelf when one dies. The dark LED is the fifth I count on this aisle. Others flicker, the half-alive blink of a disc deciding. The floor vibrates faintly, a mechanical tremor that comes up through the soles. The room is loud enough that the cart in the next aisle is invisible to the ear, until it turns the corner.

An operator walks past with a metal cart. Four dead discs lie flat on it, still warm, the way a pulled disc is warm when the spindle has just stopped. He does not stop at the dark LED. He has a different row, a clipboard with short lines of handwriting, a schedule that assumes more failures before the end of the shift.

Nothing about the room has changed since the machine died. The fans in the neighboring boards still push the same hot air. The discs that are alive still click and whisper. The rebuild of the index, running on the machines that still answer, has not slowed. The loss of one board, or 50, does not alter the arithmetic of that rebuild, because the arithmetic was written for it.

The operator turns the next corner. The cart's wheels are too close to the ground to make a sound above the fans, but the discs slide against one another with each push, a small glassy contact. Then the aisle is empty again, and no pager has gone off, because no pager was going to go off. A dead machine on this floor is not an event, it is a working condition, like the heat, the dust, and the sound. The operator walks on with his cart, four warm discs laid flat. The room continues around him, the way it has been continuing for hours.

The operator went where the clipboard sent him, into the next row, and then around the corner. The aisle settled back into itself. Hot dust air, small fans, the running arithmetic of the machines that still answered, raise the view a little, from the aisle to the ledger, and the floor itself becomes a price sheet, with two columns.

One column is what the industry in 2003 would call a serious machine. It has error-correcting memory, redundant power supplies, hot-swappable discs, a chassis built so that any single failure leaves the board running, and a service contract that promises a human being, arrives within four hours. The second column is the boards on this floor. The same processors, the same memory, the same discs, bought by the palette, at the price the palette commands. The gap between the two columns is not a refinement. Per processor, per gigabyte, per watt, the serious machine costs several times what the commodity one costs. That is before you have paid a dollar for the service contract. The contract itself is the industry's own admission that the engineering could not make the machine stop failing, only that somebody would be standing by when it did.

Put that way, the choice sounds like poverty. It is the opposite. And the argument for calling it the opposite, appeared in print that March, in IEEE Micro, signed by Luis Andre Barroso, Jeffrey Dean, and Urs Holtzle, Holtzle the first vice president of engineering, the man behind the infrastructure decisions. The paper did not present cheap machines as a stopgap for a company that could not yet afford better. It presented them as the architecture.

The argument can be stated in two sentences, though the paper needs pages to make it stick. A machine engineered not to fail, costs several times more per unit of arithmetic than the same parts bought without the engineering. So the real question is not what reliable hardware costs, it is what a failure costs. Once you own enough machines, a failure costs whatever it costs to have the software notice a machine is dead, stop sending work to it, and make another copy of the data somewhere else. That cost is paid once, in the writing. After that, every machine you buy can be a cheap one, and every dead board becomes a small event instead of a service call, because the software has already done the only thing that needed doing.