Ironic as it may sound, I was writing this post yesterday when access to our WordPress site was terminated by the Anonymous attack on GoDaddy.com. At first I thought I screwed something up but after a few minutes I convinced myself that it wasn’t my doing. I sent an e-mail to my business partner on the off-chance he was doing system maintenance. He fired back a quick “Nope” and then followed up with “All my GoDaddy sites are down.” A few minutes later the news was out that GoDaddy.com had been hacked!
It was a sobering moment. The thought of having to rebuild this site was not something I wanted to contemplate, but I was comforted by the fact that I had numerous backups on my own servers. Those servers were also backed up with local off-line copies, and older but still useful backups stored at remote off-site locations. I was confident that the site could be resurrected in a few days even if the worst happened. At the same time though, I’m sure there were thousands, perhaps millions of people around the world, who were caught short and wondering what they were going to do should the outage persist and what would become of their precious e-mails, photos, websites and blogs. What the hacker did yesterday was criminal and whoever did it should be caught and punished to the full extent of the law. On the other hand, the hacker may have done many people a big favor by reminding them of how vulnerable they are and prompting them to start thinking about their IT disaster plan. Hopefully, some of these people will actually be in a stronger position the next time something like this happens; as it surely will!
In the last few months I’ve helped a small health related business transition to a new business software package. This software runs in what is known as client-server mode. In simple terms that means that there is one central server where all the data is stored and each employee uses client software running on a workstation to enter or manage the data. In this particular case, there are workstations for billing, scheduling, patient registration as well as workstations for the health care workers to view and enter patient notes. The workstations communicate with the server over the local wired or wireless network. Overall, the new system is a dramatic improvement over their old hybrid paper / digital system and it’s making life a lot easier for everyone involved. Ultimately, the client-sever setup and new software promises to make their business more efficient and profitable.
I’m sure many of you have similar types of client-server business software packages that have become indispensable parts of your business operation. Some of you may even have your servers in the “cloud” and “workstations” in the form of netbooks, iPhones and tablets scattered all over the globe. Those of you that don’t yet use such a system yet will almost certainly find that you will eventually need one to run or grow your businesses. With that in mind, I’m going to discuss some things that you need to keep in mind as you consider the pros and cons of client-server type setups.
If you are already using a typical client-server type system you probably know that “if the server or network is down, your business is down too!” During such outages, your employees aren’t productive, your customers aren’t being served, and your business is bleeding money! Regardless of whether you already use client-server or if you are contemplating migrating to your business to a client-server setup, you owe it to yourself to think about this fact and plan to mitigate it. The companies that sell you on client-server software will focus on the great advantages of client-server and show you how their software and systems will help streamline your operation and make you more money. Indeed, many of their claims will likely be true. What they likely won’t discuss however is what happens when something breaks and your whole business is paralyzed because of a server or network failure. Don’t get me wrong, migrating to a client-server setup is usually the right call for most businesses. Just be aware that there will be added costs and issues once you make the jump.
From my perspective, one of the most important concepts you need to drill into your head for successful IT infrastructure is that of redundancy. Redundancy is something we usually try to avoid in business since it adds cost and overhead. On the other hand, redundancy can also add resilience against certain types of failures. The space shuttle computers were triple redundant. Three computers constantly did calculations critical to the flight plan. At each step, the results from each computer were compared and if one of the computers disagreed with the other two it was taken off-line and the remaining two continued to safely run the shuttle. Adding redundancy to your business IT infrastructure will allow you to keep going when you might otherwise be forced to halt operations and make repairs. Repairs and maintenance are of course inevitable and redundancy can also help reduce both planned and unplanned downtime. Redundancy is probably most critical in your backup and data recovery plan. Redundancy will make sure that you have a plan B, C or D on the day that you find your most recent backups are corrupt or infected with a virus.
If it’s not obvious by now, the key to high availability client-sever setups is redundancy. Unfortunately, it is my experience that most small businesses believe they can’t afford the luxury of redundant hardware and systems. I believe this is false economy in most circumstances!
If your entire business depends completely on client-server software, and you’ve got only one server, then I think you owe it to yourself to consider getting a spare. You might argue that you spent a lot of money on the server and you might even have spent extra for a RAID disk system so it sure as hell better work! No doubt, I can appreciate and sympathize with that sentiment. On the other hand, I shouldn’t have to remind you that nothing is perfect. I also shouldn’t have to remind you that a RAID 5 disk system can only protect you against a single disk failure! RAID 6 can weather two drive failures but your server is still dead if the RAID controller fails! So, you relent and figure it might be wise to have a spare RAID controller. Now suppose that over the weekend, the fan in the power supply dies and no one is there to hear the warning beeps. You come in on Monday morning and your server is dead. It could be the power supply, the CPU, the memory, or a motherboard issue. What do you do? Do you keep a spare of each component in the server so you can rebuild it piece by piece? A rebuild like that takes time and expertise that you and your staff might not have. Wouldn’t it be smarter to have a spare server sitting there and ready to go? That would drastically cut your down time should disaster strike. Maybe your primary server was really expensive and you truly can’t afford to duplicate it. Perhaps you should consider a spare that isn’t quite as fast and fancy? Surely a slow server is better than NO sever when you’ve got a line of customers at the front door?
With client-server you also need to consider redundancy is in your network. Routers and network switches are usually very reliable, but when they die they wreak all kinds of havoc on your business. Fortunately, these items aren’t usually as expensive as the server and thus there’s little justification for not having spares readily available. If you’re smart, you will configure the spares with the particulars of your network so that any staff member can quickly and easily do a drop in replacement in under 5 minutes. If you gamble and wait until you actually have a failure you might have to pay your IT guy premium rates to configure it in the middle of the night or over the weekend. Worse, if you live in a remote area like I do, you might end up waiting 24 to 48 hours for a delivery from Amazon before you’re back up and running! Do the math on that and a spare router looks like a bargain!
Perhaps the most important place to have redundancy is in your backups. The price of Tera-byte hard drives is so low that there’s no excuse for not having a lot of backups. You should have multiple backups at the office, as well as copies at home, and with trusted friends and family in other areas of the country. (Note: I’m assuming you’ve encrypted the backups! If they are not encrypted, you need to read my articles on Truecrypt NOW!) Even if you’ve got hard drive backups, for some files the use of cloud storage may also be a good idea. The more backups you have the better off you’re going to be when disaster (or the hacker) strikes!
I’ve only discussed some of the obvious places where redundancy can help improve reliability. I’m sure if you think through your business process and the IT infrastructure that supports it you will find more items that could be classified as potentially critical failure points. If you’re smart, you will add some redundancy at those points to help mitigate the damage should a failure occur. If any of you have more thoughts on this topic I hope you’ll consider posting them in the comments section! Thanks for reading!