Sandy From The West Coast
Tuesday, October 30, 2012 at 10:10AM The images are everywhere: flooded tunnels, runways, and streets, fallen trees, dark skylines.... They tell us of a horrific nightmare in a visually shocking way. Then I got an email from the founder and CEO of the company that hosts my blog—they’re great people who run a great (but small) company with great customer service.
The email starts out with “I have some unfortunate news....” Their primary data center Peer1 in Manhattan had lost power yesterday afternoon. They switched to generator power, no problem. They had enough fuel to last them “three to four days.” And then things started to go wrong.
It’s such an excellent account of just one aspect that businesses in the area have to struggle with, namely power, that I decided to share excerpts ... while I still can.
At 8:30PM yesterday, we received reports that the lobby in the data center's building was beginning to take on water. By 10:30PM, as is sadly the case in most of Lower Manhattan, Peer1's basement had experienced serious flooding. At 5AM, we learned our data center's fuel pumps and fuel tanks were completely flooded and unable to deliver any more fuel. At 8AM, they reported that the generators would be able to run for a maximum of four more hours.
The service would be “offline soon,” probably at around 10:45 AM EST. As I’m writing this, the generator should already have shut down, but I can still access the system, and my blog is still up. Maybe they were able to get some fuel to their generator.
The email assured me that there would be “no chance of data loss or other permanent effects.” I’m not worried. I assume their servers aren’t in the basement next to the fuel tanks.
I just checked their blog. New update. They’ve been working furiously to keep power up for as long as possible. The new estimate is 12:46 PM, the very minute that I’m writing this. System is still up.
It’s hard to imagine what it must be like dealing with this kind of generator problem, when people have died, streets are flooded, bridges and tunnels are closed, transportation has come to a halt, communication is difficult, and people have trouble going where they’re needed.
The email closes with these thoughts: “Our hearts go out to the many people who have lost their lives in this terrible tragedy and also to those who continue to suffer through the consequences of this historic storm.”
Thoughts that reflect exactly how we feel—and so I want to share them around the world while I still can post them, and while my blog is still up.
Update: so now it's 8 PM ET, and the generator is still running because....
We have been running hour by hour by manually carrying fuel to our generators (17 floors) with support from the building. As the night goes on, this is becoming a bit more difficult to sustain, as fuel trucks will appear more intermittently. For now, we remain online. Thank you all for your patience.
As before, we do not have a sustainable solution to the energy problem, but we persevere in our efforts to get more fuel on site and get a pump connected.
They're certainly dedicated. Small company fights the odds and the flood. Awesome!



Reader Comments (6)
(When the water is gone and life is back to normal in Manhattan, they should shop for a backup data center in another part of the country. It's called geographical redundancy. Just an idea.)
Bizarre.