DC2 had some pretty significant DNS slowdowns this
Sunday morning.  It all seems to have been triggered by
a power supply failure.

As far as we can tell, a failing power supply in a
large server caused a circuit breaker to pop on one of
the UPS B circuits, which took down enough of the DNS
caching cluster to cause the remaining DNS servers on
UPS A to have difficulty keeping up with requests.  We
have moved half of the DNS servers that were on UPS B
to UPS D and tonight when loads are lower we will move
half of the caching servers on UPS A to UPS C.  That
should evenly balance the DNS servers across 4 UPS's
instead of 2 and reduce the number of servers lost when
a given circuit fails.