Outage Resolved05 Jan 2006
That outage at work just before Christmas turned out to be a software problem in legacy code on one of our minor sites. It cropped up a couple of more times last week, and ended up costing me 12 hours during a holiday week when I was supposedly on vacation, mostly because I at first misdiagnosed it as a hardware problem. Once I had it isolated, I simply shut the site down and went back on vacation. This week a programmer was able to spot the bug within minutes, and the bad site went back up.
It was a learning experience. When the router is not working, it's probably wrong to assume that the router is broken. Legacy code is a time bomb waiting to explode. And the biggest threat to your network is probably behind your firewall, not in front of it. I "knew" all that already, but now having lived it, I now really know it.