DNS was down this morning (Sat, Jul 12) between
midnight and 3:30 AM. The cause ended up being a
malformed domain name that had been plugged into the
system and wasn't caught by our regular expression filters.
Our apologies for how long it took to diagnose. The
BIND 9 dns server we use was refusing to start, but was
not writing any errors to the log files. Even when
running in the foreground mode with logging to stderr
nothing was printed to the screen. With nothing to go
on in terms of what was causing the failure, we ended
up perusing the config files, looking for possible
issues. Quite a daunting task, given that there are
around ten thousand domains in the system. We got
lucky and found the problem relatively quickly.
Our regular expression filter has been updated to keep
this from happening again. Many thanks to everyone
that called to let us know about the issue and again,
our apologies for the trouble.
|
