Lessons in disaster recovery, tsunamis, and meltdowns

We will probably not know the full story of the nuclear reactor problems in Japan for many months, but one news story I read over the weekend suggests that the the Japanese are re-learning the lessons of the Katrina disaster. Apparently the Japanese reactors survived the initial earthquake and tsunami without much damage--but whatever was damaged caused the primary cooling pumps to fail. No big deal, as nuclear power plants have extensive back up and redundant secondary cooling systems designed to take over if the primary cooling system fails.

If the primary cooling system fails, the reactor is usually shutdown immediately, meaning no electric power. Even when the control rods are in, heat can continue to be generated for some time, hence the need for secondary cooling. So here is the scenario. Primary cooling fails. The reactor is shut down, and secondary backup cooling systems are activated. Apparently all this happened just as it was supposed to immediately after the earthquake and tsunami. The secondary cooling pumps are powered by large diesel generators, which apparently ran for about an hour, then shut down.

Why did they shut down? The fuel was contaminated by seawater.

So what was the lesson of Katrina the Japanese missed? In the New Orleans area, many telecom, radio, TV, and computer installations were thoughtfully built on upper floors of buildings so that they would be immune from flooding. But the generators and fuel tanks that were supposed to power all those systems in the event of a power outage were all installed at ground level, because it costs more money to put heavy, loud generators and diesel fuel on an upper floor. So the hurricane winds blew down power poles and the power went out first. No problem. Emergency generators started up, and everything kept running. Until the water came and flooded and the fuel tanks and generators.

One small ISP in downtown New Orleans stayed up and running throughout the entire flood because they had installed a generator on an upper floor. They managed to truck in 55 gallon barrels of diesel fuel once their initial supply ran out.

Those who fail to learn from history are doomed to repeat it.

How is your community's disaster recovery plans? Many areas of the U.S. are flood prone, but everywhere I go, I see generators now--which were very rare ten years ago. But those generators are all on the ground. Is that okay for your area? What about a ten year flood? What about a 100 year flood?

Technology News: