I’ll just say that if you couldn’t make it to NYTSL’s Spring Program last night, you missed an excellent discussion.
The program was “”Disaster Recovery and the Digital Library”, and as such, it brought a bit of real world application to the often abstract world of disaster planning. NYTSL’s guest speakers were Frank J. Monaco, a retired Army Colonel and the recently retired CIO of Pace University; and Neil H. Rambo, the Director of NYU Health Science Libraries and Knowledge Informatics.
Both men spoke about experiences that put their training, planning, and experience to the test. In Frank Monaco’s case, it was managing the school’s recovery from the 9-11 attacks.
Monaco has already written extensively on what he did that day but briefly: after he transitioned from the military to CIO of Pace U., the first part of his plan was to move the institutional data centers as far away from downtown as possible, meaning Briarcliff Manor, the site of Pace’s Westchester campus. This turned out out to be a fortuitous decision. When the Internet collapsed (literally, as data transfers relied on the Verizon facility at 140 West Street which was damaged by the 7 WTC building collapse) Pace’s CTO had to physically carry the school’s mission-critical external servers and move them to a disaster-recovery site in Hawthorne, NY. After 24 hours to allow new IP addresses to propagate, web pages and e-mail returned to functionality.
Importantly, Monaco noted that their disaster recovery plan was still incomplete at the time of the attack. He also pointed out that restoring service was a very small part of the tremendous effort exerted by Pace’s president, executive staff, administrative staff, faculty, and students.
Neil Rambo told a hair-raising tale of the events that occurred in 2012 when Hurricane Sandy heaped a few million cubic feet of water on New York City. Long story short, a 14-foot storm surge sent a wall of water powerful enough to blow steel doors off their hinges and send part of the East River into the lowermost levels of NYU’s Langone Medical Center on First Avenue. The result was a ruined library, destroyed archives, and a non-functional hospital.
Part of the problem was that NYU had previously weathered hurricane in 2011, Irene, which did minimal damage, and created a plan that expected similar damage from Sandy. After creating a response that planned for a cleanup and restoration of the medical center, library staff were enabled to work out of different facilities in a building across the street, reducing the loss of activity. It’s taken this long to determine that the library will be rebuilt into a superior environment which will devote nearly all its space to electronic resources, and is due to open later this year.
The lessons here aren’t really that surprising: make plans when things are running well, because there won’t be a chance when things break. A great plan, properly executed, is always better than an okay plan properly executed and light-years ahead of no plan at all. Optimally, the highest levels of administration need to be on board from the first stages of planning. Monaco’s advice on achieving this: “Scare the hell out of them.” Rambo’s advice was a bit more circumspect: “Imagine what would happen if you library was just gone, and work from that.”
And so I did. I’ll tell you what I figured out in the next post.