We had a pretty bad wind & rain storm here last night.
The power flickered on and off several times. One of
the flickers/surges seems to have caused issues with
the enclosure for many of the HOME3 drives, causing
enough drives to be dropped from the array that
performance has been cut to less than half normal.
We have been working through the night, monitoring the
rebuild and doing what we can to optimize things, but
the file server is having a very difficult time
rebuilding while under load. Our initial expectation
was for things to be back by 8:00 AM PST, but as the
morning dawn hit, the load on the fileserver rose
exponentially. Our current ETA (to 100% fileserver
performance) is 5:00 PM PST.
As a result of this incident, we'll be engineering a
solution to keep this from happening again. Our gut
feeling is that a transition over to SSD's for the
array is probably going to be our path of choice, but
we haven't ruled out just doubling the array's drive
count. We'll keep everyone posted as things develop.
|
