A power outage on Waldo road just outside of East Campus has affected cooling of the cluster server room.
A vehicle accident on Waldo road caused a power outage for most of East Campus this morning (Friday, October 18th), and while the cluster never lost power, the cooling systems did lose power for a short amount of time.
While no jobs were lost, the scheduler was paused for the duration until staff members felt that power had stabilized. The server room did get warmer than usual for a short amount of time, so pausing the scheduler to reduce the load of jobs running was felt prudent in order to reduce the heat load. Once power had been restored to the chilling units and emergency vehicles had cleared away from Waldo road, it was felt that the scheduler could be resumed for normal operations.
Work continues on Waldo road to restore the transformer that was damaged. No word yet from GRU on what effects this work may have on the data center.