Disaster Preparedness

As we all know, disasters and emergencies happen when you least expect it. People get hurt, leave, find new jobs, or are otherwise taken out of the loop. Often, techs who are completely unfamiliar with the institution will be called in to cover tech needs in the vacuum left by the departed tech. Sometimes, the disaster will be so terrible that you won’t have an institution to go back to, in which case you had better be ready to deal with the insurance company and have a plan for rebuilding the technology infrastructure in your institution.

Working as an admin/tech support/geek of last resort in a multi-type library system, I’ve come across many situations in which the documentation for an “outside” tech (usually me) is either inadequate, nonexistent, or held hostage by yet another third party.

Library hackers, we cannot abide by this! We should be prepared!

I challenge you to have the following ready for your institution, just in case, heaven forbid, you should get hit by a bus:

• A network diagram (I like to use gliffy for this), preferably with ISP’s, IP’s, subnets, and relevant servers noted.
• Contact information for the ISP and phone service provider.
• An up to date list of usernames and passwords. No password should be kept in only your head. I make two accounts on every server: an institutional one, and my personal one. The institutional user has the same rights as mine, but this way I don’t have to document my personal password in the institutional documentation. I keep a cheapie thumb drive in the fire safe with copies of all of the SSH keys and usernames/passwords, and update it twice a year when I rotate passwords. It probably wouldn’t hurt to keep another copy offsite somewhere.
• A list of servers and what they do, where they are, their IP and/or DNS name, and what OS they run.
• A list of software keys for software your institution has purchased, and customer numbers, logins, and contact information for those vendors. You can keep this on your fire-safe-thumb-drive as well.
• Offsite backups of important data. I don’t care how you do it, just make sure you do.
• An up to date inventory of hardware you’d replace if it were lost, with serial numbers and purchase date.

I’m sure there are things that you all do that aren’t on this list, and I’d love to start a discussion of how best we can prepare our libraries for disaster. What more should we do? What do you do?