We have finished our investigations, this is to let
everyone know what happened.

We have had two brief outages in the last couple of
days, each lasting for about 10 - 15 minutes or so. 
The first was Saturday, Jan 19 around 11:30 AM PST, the
second was Monday, Jan 21 around 12:30 AM PST.

      

The cause of the first outage was instability on our
new Level 3 circuit.  Qwest (who provides the local
loop) was having difficulty, which was causing the
circuit to go up and down.  As the circuit was going up
and down, the routing was taking time to update, thus
we were down to about 1/3 of the internet for about 15
minutes.

The cause of the second outage was self-inflicted.  I
was doing a simple move of a small Cisco switch from
one rack to another.  The switch only has two servers
on it but as it turned out, it is tied into the
internal routing infrastructure in such a way that when
it was powered down to be moved, the change triggered a
routing change on the main router that caused traffic
to stop flowing.  We have altered our internal routing
setup to keep this from happening in the future.

We work really hard to keep things as stable as
possible, and the second outage really stings since
that one in particular was my fault.  I apologize to
everyone for the trouble!

~Ethan B.