Server Matrix/The Planet experience network downtime Published: Mar 20, 2004

• Rating

  1/5

Router issues cause hundreds of servers to go offine for more than 3 hours.

Sever Matrix/The Planet is having large router issues causing hundreds of servers to go offline including The Planet customers starting around 11 PM March 19, 2004 which caused nearly 4 hours of outages for many customers, including one of our own servers.

Total downtime for servers behind CAR2-6.DLLSTX2 was:
0d 1h 50m 41s
Total downtime for servers behind CAR2-8.DLLSTX2 was:
0d 3h 30m 18s

This was independently monitored from the ev1 datacenter, at 90 second intervals

From the Server Matrix staff:

Outage Report
March 19, 2004

"Summary: At 23:00 CDT, ServerMatrix engineers were alerted that one of our customer VLAN routers (CAR) was not responding. It was determined that the router had crashed and it was reloaded and returned to normal service. Within several minutes 2 more customer routers crashed as well. Upon inspection, it was determined that the cause of these crashes was a vulnerability in the BlackIce personal firewall program. The details of the vulnerability are located at http://www.eeye.com/html/Research/U.../20040213.html. In order to stop the routers from crashing, Planet engineers were required to locate and disconnect all affected servers. In addition, filters were applied at our border routers to prevent further exploitation of the vulnerability. Planet engineers are continuing to monitor all of the routers to make sure the problem does not re-manifest.

Future Mitigation: The filters will be left at the border for an indefinite time to prevent future exploits. In addition, we would suggest that if you utilize the BlackIce firewall system, that you patch your software as soon as possible. If you have any questions about this, please contact our network security team at [email protected].

Thank You,

William Charnock
VP Network Engineering
The Planet Internet Services"