Make Sure They've Done It Before

Would you agree to have an operation performed by a doctor that had never done that type of surgery before? Would you buy a car sight unseen without a test drive? This is essentially what I see people doing all the time when it comes to IT. They allow internal IT staff to implement products they have never used before, and they allow consulting firms to implement solutions that they have never deployed at other clients. There is no need for this to happen.

It is perfectly appropriate to ask internal IT staff or consultants whether or not they have ever deployed the technology in question at another job or client site. If the answer is no, don’t allow them to do it on their own. If you really like them and you want to keep them around request that they bring in a consultant that has done it before to oversee the process. Explain that it isn’t your lack of confidence in them. It’s simply that you know that the first time you do something you almost always make mistakes, and you don’t want that to happen with your critical IT infrastructure. If the answer is yes, ask a few more questions. How many times? How did it go? Can you speak to someone who used or uses the system that they deployed? This simple process can help to avoid a disaster.

We recently won a new client that had a Microsoft Exchange clustering solution implemented by another firm. Basically the solution incorporates two servers, one of the servers can fail completely and Exchange will remain up and running. If you can afford it the solution makes sense. Unfortunately the consulting firm they hired made several critical mistakes when deploying it. When it was all said and done they got an email system that was going down for hours every two days. After a month of excuses from the old consulting firm they said enough is enough and brought us in to rectify the situation. They paid $60,000 for the system and then they had to pay us an additional $10,000 to fix it.

Ask questions, require logical answers, and don’t be afraid to bring someone else in if you don’t feel comfortable.

Backup Your Data and Verify Every Quarter

Data backups are incredibly important; you probably don’t need me to tell you that. In the even t that a server fails your backups are the only place where that data still resides. If you can’t restore the data from your backup the data is lost along with all the work that was put into generating it. Given that it’s so important it’s shocking that most organizations never verify that their backups are working.

Attitudes towards painful events are always the same, it will never happen to me. Complete server failures don’t happen often, but they do happen. How much would it cost you if you lost your ability to conduct business for five days? How much would it cost you if after the five days of downtime you had no historical business data? The reality is that you can avoid problems like this with a few simple policies and services that are inexpensive when you consider what they are protecting you against. I have listed those out below.

• Create a data backup schedule that makes sense for your business. You are going to want to do daily backups at least once or twice per day. This will allow you to restore files that people delete or modify accidentally. Next, figure out how much data it is acceptable to lose in a disaster scenario. If your server went down and couldn’t be restored how far back would it be acceptable to start? One week, two weeks, one month? Once you have answered that question ensure that your data backup is taken or sent offsite at least that often.
• Ask your IT department or IT services company how much it would cost to accommodate your backup schedule. If the cost is too high modify your schedule until you reach a good balance between cost and protection.
• Test your file backups once per quarter, and don’t allow your IT department or IT services company to say “we checked and they are fine”. Give them the name of a file and what date you would like it restored from. Have them send it to you so that you can verify that it is intact and useable. If they can’t do it find out why and do a more detailed investigation. Things that get measured and verified get done, if they know that you are going to be looking for the data they will make sure that the backups are running properly.
• If you are backing up entire servers or applications have them restored from backup once per quarter or semi-annually to ensure that they are useable in the event of a server failure. There is no point in backing them up if you can’t restore them when you need them.

My first real world example involves one of our clients that runs an ERP application that is designed for apparel companies. This application houses all their data, without it their administrative operations are basically stopped. They recently had a simultaneous multiple hard drive failure on a server that was about 2.8 years old. The server data could not be recovered. Since we have a very strict process for managing client backups we were able to restore the server from backup within 24 hours.

My second example is a client that is new; they had no functional data backup when they became a client. The client’s old IT services company sold them a backup solution but they did not implement it correctly. The reason that we won the client’s business was that they had a server failure and the old IT services provider was unable to restore the server. If the client had been verifying the backup on a regular basis they would have caught this issue before it caused them problems.

Second Opinions Can Save You Money and Time

I realize that this seems like common sense, but you would be surprised how many organizations don’t get second opinions on major projects. We see it all the time. We get a phone call from an executive at a prospective client and they want us to come in to provide a second opinion on a project that their current IT consultant or internal IT staff has presented for approval. But that’s usually the third or fourth project in the last 12 months, and it’s the first one that an outside party is looking at. A lot of the time it is a case where a major project was implemented, then there was an add on due to “unforeseen circumstances” or “scope creep”, and then another. By the time the third add on rolls around the client is starting to wonder if it was a good idea to do the project in the first place.

You should always, without exception, get a second opinion on a major project. I define a major project as a project where one or both of the following conditions exist a) you are spending more money than you would be comfortable spending normally b) you are nervous about the impact it will have on the users in your organization. If you think about it you really have nothing to lose, for most projects it will take about four hours of your time and you can save yourself significant amounts of money and pain. Here’s why.

1. A company brought in to give a second opinion will happily do so for free; this could lead to new business for them. If they won’t do it for free find someone who will.
2. They’ll do all the work. All you have to do is meet with them once or twice and give them your current quote or project document without the pricing. Never give them the pricing that you already have, you want to see their price without any outside influence.
3. If you use an IT consulting firm having a lower price from a competitor gives you significant negotiating power. In fact I recommend telling them up front, before they provide the quote or project document, that you will be sending it to an outside party for a second opinion. This will keep them on their toes; they will keep the proposal lean if they believe that they are going to have to fight for the business.
4. If you have internal IT staff it will communicate to them that they better have their act together when they send you a proposal. It will come out very quickly if they haven’t done their homework regarding the technology or pricing.
5. You will often find that the company providing the second opinion recommends a different technology or implementation strategy. This will force internal IT staff or an outside firm to justify their recommendations; you can then make a choice based on the most logical argument.

One real world example that I encountered recently involved an IT consulting firm and a 150 person commercial real estate firm. The firm has no internal IT staff and uses an IT consulting firm for all their IT needs. The client was negotiating the monthly fee for their IT services at the end of their current contract and they asked us to provide a quote for the same services. Lloyd Group did an assessment of the network and quoted a price of $17,900 per month, more than $7,100 per month less than their current IT provider. The client then used this quote to negotiate a better price with their current provider since they were relatively happy with their services. In the end this client saved $6,000 per month simply by asking us to come in and provide a second opinion, and it took less than 4 hours of their time. Lloyd Group is now asked to quote all projects for this client and one day we are going to win their business.

A second example involves a 12 person sports services firm and a small IT consulting firm. The client’s consulting firm made a recommendation for a server upgrade project to address network slowness which the client signed off on. Unfortunately once the project was complete the network was slower than before the upgrade. There were 2 additional projects which the client signed off on in order to address the slowness, but neither of them fixed the issue. We were brought in to do an assessment and determine whether or not the work was done properly. We found that the problem actually resided in the configuration of the client’s network, not the hardware and infrastructure. We were hired to resolve the issue, which we did, and the client signed with us to provide all their IT services moving forward.