More that 350.000 web sites (DreamHost) back online after almost 8 hours outage

Due to broken power cable the whole Dreamhost network was not reachable for almost 8 hours today. It just came back online…

Not all the servers went down, but as DreamHost’s core routers and upstream providers are in that data center, there was no network to over 350,000 web sites during that time.

The scheduled outage time was 3 hours but due to a router failure it took much longer.

Update 20070225 8:15 PST:
There are reports that still quite some sites are not reachable.

Our weblog for today shows the first successful connection serving a web page for 6:38 PST and the last successful connection before that for 11:05 PST yesterday…

Update 20070225 9:05 PST:
More than 500 mostly angry comments have now piled up at their status page and many are still complaining that their web sites are not reachable. The status info on the site was last updated at 04:51 PST - We keep you posted -

Update 20070225 11:15 PST:
We pinged all 492 hosts IPs from the DreamHost Server IP Address List.
While we got time outs from various more servers the following IP-adresses were still not reachable:

• absinthe.dreamhost.com [66.33.205.5]
• adams.dreamhost.com [64.111.117.6]
• bundy.dreamhost.com [64.111.108.23]
• dollar.dreamhost.com [205.196.218.5]
• foothill.dreamhost.com [64.111.108.14]
• formosa.dreamhost.com [64.111.122.20]
• fresca.dreamhost.com [208.97.171.23]
• gumdrop.dreamhost.com [64.111.126.26]
• haven.dreamhost.com [64.111.126.4]
• heath.dreamhost.com [64.111.122.25]
• highland.dreamhost.com [64.111.110.22]
• holt.dreamhost.com [64.111.125.15]

It looks like they are almost done now…

Update 20070225 11:30 PST:
The DreamHost status page (now more than 600 comments) has just been updated with the following text

- UPDATE 11:21 PST -
The router is up and healthy, and the majority of our network is back up and running. However, a few of our servers are having issues because their file servers are not talking to the rest of the network correctly. There is a link down from one part of the datacenter back to the routers and that is causing the file server issues. If you are on a server whose name is a beverage, this will effect you. We are working very hard to resolve this issue, and we apologize for the inconvenience.


Update 20070225 12:05 PST:
Pinged the list of servers above again (not the whole list) and the following servers are still not reachable:

• bundy.dreamhost.com [64.111.108.23]
• fresca.dreamhost.com [208.97.171.23]

There seem to be no DNS entry for beverage.dreamhost.com i.e. name could not be resolved.

Update 20070225 12:20 PST:
More than 650 comments now at the DreamHost status page

People have started to report all sites that are still not up properly at that page. Go to the DreamHost status page and write a comment if your site is not fully up and running…

Thanks to Aaron for the tip that beverage is hi-c.dreamhost.com

Update 20070225 13:00 PST:
Could anybody please explain why this story was just buried at Digg.com?????

If you’re angry that your server is not properly running it’s not our fault!!!

Update 20070225 13:20 PST:
Only fresca.dreamhost.com [208.97.171.23] seems to continue not to be reachable by now.

About 700 comments now on the DreamHost status page with dozens of sites reported to be still not reachable after more than 13 hours

Please DIGG this story with the icon above if you want us to continue with the updates….

Update 20070225 15:00 PST:
We pinged all 493 hosts IPs from the DreamHost Server IP Address List again.

Looks like the fumbling is going on and there are some servers still not reachable (now other servers). And we are still getting time outs from various servers but it looks that most at least respond to pings (or have before). This does of course not necessary mean that your site is up and running.

dads.dreamhost.com [208.97.167.9] seems to be crawling
dollar.dreamhost.com [205.196.218.5] no response
nimitz.dreamhost.com [205.196.218.28] no response
slimy.dreamhost.com [205.196.208.18] seems to be crawling too
soda.dreamhost.com [66.33.197.10] no response
tab.dreamhost.com [208.97.167.22] no response
taffy.dreamhost.com [64.111.122.9] no response

Thanks to all of you who are still digging this page even after some have buried it on DIGG.COM and made a 2 days old story popular telling that there WILL BE an outage - I guess that we have all seen today

Update 20070225 17:50 PST:
The “Planned Power Outage” from last night is still flagged as an unresolved high level issue at the DreamHost Status Page. Looks like they are resolving a few other issues as well while at it…

And it’s getting a bit more quiet at their comment page. While people are reporting that their servers are back, sites served from these servers still have to come alive again. Looks like a very tough Sunday afternoon for the DreamHost people…

Update 20070226 06:15 PST:
Still no updates on the situation at the DreamHost web pages.

People are now merely reporting issues with services (SQL, SSH, Mail) while servers overall seem to be up. Also still some “brown-outs” meaning that sites are not responding for a few minutes…
Update 20070226 11:00 PST:
The DreamHost team has put up three new messages on the DreamHost status page reg. issues related to yesterday’s outage:
1. Mail delay on junk filtered domains
2. Residual network issues still
3. Snoopy mysql server issues

Click on the image for the jump.

With about 54 million active web sites worldwide this means that about 0.7% of the whole Internet went bye-bye due to their broken power cable.

Apologies to all our readers for not being reachable for the last hours! We are hosted on DreamHost as well.

More info and pictures at their blog.