HOW THOUGHTLESS DECISION-MAKING & SLOPPY HOUSEKEEPING NEARLY HIJACKED A HOSPITAL’S JCAHO

In 2005, twenty servers running a critical application at the busiest hospital in Illinois were consolidated into one physical server. Instead of reaping the benefits of consolidation, disaster struck. (Its name will go unmentioned but you’ll find it out if you read on.)

Hospital management anticipated the usual benefits that virtualization brings:

1. Easier administration. Caring for one server is easier than caring for 20.
2. Greater confidence in the IT infrastructure. The storage that accompanies virtualization is likely to be more reliable than the distributed storage of standalone servers. This reliability is a product of newer technology and a more efficient design.
3. Peace of mind. Virtualized storage complements or fits well with its business continuity features. VMware’s VMotion, for instance, empowers the human administrator to migrate virtual machines to backup servers in real time.

Unfortunately these benefits did not happen. They lost data and, for a time, they risked, first, losing JCAHO accreditation and, second, punitive action from CMS.

(Click here to learn why JCAHO accreditation is important to a hospital.)

How did this happen?

After the virtual environment was created, the IT staff added standard security controls to each new virtual server. This was fine as this is standard procedure. However, some of those virtual servers lay dormant. In fact, it appears that nearly a dozen servers were created for “testing” purposes. These were not removed after they had served their purpose. (I actually think that most of them were created for the novelty of it. How do you account for servers named “Tyrone” or “Michael Jordan?”) During the months that these servers lay dormant, Microsoft and the application vendor had issued patches. When these dormant servers were reactivated, they were not updated with those patches. The servers thus turned into potholes or, worse, security vulnerabilities waiting to be compromised. It didn’t take long for that to happen. Consequently, the hospital lost data.

We were brought in to sort out the mess.

LESSONS LEARNED

What did we take away from this incident?

First, virtual servers must be managed individually and managed from their creation to their removal.

Second, management of these servers consists of staying abreast of patches, installing them as needed and meticulously documenting the patches that were installed. These steps have to be done for the virtual environment, the guest operating system and the application. These steps are crucial especially because of staff turnover.

Finally, management of the virtualized data center should be handled by capable hands. The integrator may have configured the virtual environment properly when it was created. However, we all know that things change over time. Someone has to take ownership of staying abreast of these changes. In the hospital’s situation, the virtual environment unraveled in steps. Visualize these: (1) a new appliance was installed, (2) a new server was created, (3) a new application was implemented, and (4) Microsoft issued more security patches. All of these events most likely took place. Consequently, failing to update the relevant pieces or updating the pieces incorrectly would have caused problems. Note that there are two hurdles: (1) identify the pieces that need to be updated and (2) do the updates correctly. At the end, we discovered two network links that were dead ends. We think these links had prevented two or more virtual servers from communicating.

While that was a technical AHA!, the bigger picture shows the consequences of a thoughtless decision. The hospital had stopped paying maintenance fees to the integrator. It attempted to maintain the environment on its own. This was unwise since the IT staff did not have trained personnel. The VLAN’s configuration developed potholes and compromised security. This is how a combination of thoughtless decision-making and sloppy housekeeping nearly hijacked a hospital’s JCAHO accreditation and risked punitive action from CMS. (This was a major reason. During that period, the hospital was cited for numerous violations.)

SMALL BUSINESS STORAGE GROWS!

Over breakfast with another ex-EMCer, I learned that the storage demands of small businesses continue to grow—recession or not. We worked for EMC from 2000 to 2001. EMC, like many large companies, hold periodic conventions for its employees.

At the annual EMC World Congress (or whatever it was called then) for its global sales force, I recalled then-CEO, Mr. Michael Ruettgers, proclaim how mankind’s thirst for more storage would fuel EMC’s growth for the decades to come. Shortly after that, the dot-com bubble burst, a recession started, and spending in IT slowed. Today, investment in storage is apparently back on track.

What, we asked ourselves, was fueling the demand by small business for more storage demands? We identified three trends responsible for this.

First, most e-mails aren’t deleted. And many e-mails contain attachments. Second, many files are being saved even after their original purpose is over. And third, files have become larger. For small businesses, more documents are being scanned and stored in image or PDF format. PDF files are nearly as large as image files. Think of how much more widely images are circulated on the Internet, for example. Image files are vastly larger than a Microsoft Word document, for example. Movies (think YouTube) are even larger.

SOLUTIONS

There are several ways to help small businesses cope with this demand. One technology solution would be the VMotion capability of VMware. This capability is specifically designed to facilitate infrastructure growth. One benefit: it transforms the smaller configuration into a high-availability infrastructure.

The first trend we identified was e-mail proliferation. The best solution to this problem requires a combination of teaching best practices and using technology to rein in the e-mail monster. Most people don’t know how to shrink the sizes of their image files. Cameras typically produce photos in the multi-megabyte range. Music files are similar. A minute of song is about 750kb. Shrinking files of any kind before attaching them will reduce e-mail storage requirements tremendously. That’s the best practice part. File deduplication technology identifies and removes file duplicates. That’s the technology part but before we leave this topic, let me ask you: if a person attaches a file and sends that email to his co-workers, how many copies of that file attachment are created?

Storage appliances are another solution. NASs and even SANs have their place too. The former, as you can see from this selection, is especially well suited for smaller IT infrastructures.

Another way that small businesses can cope is through the adoption of an old strategy called Hierarchical Storage Management. HSM works this way: files are stored and archived in different kinds of media depending upon the business’ policy such as the file’s importance, its frequency of access, or any other. For example, in an architectural firm, all files that relate to a project are stored together, separately from the rest of the company’s files. HSM uses tiered storage. Files that are used regularly are kept on primary storage. Files that are accessed less often are transferred to secondary storage. Primary storage allows instant access. Secondary takes longer; the storage media has to be located and then mounted before the file can be retrieved. The key word is “policy.” The business should state its policy towards each type of file. E-mail goes here, image files stay there, and so forth. SearchStorage.com has a concise but comprehensive definition of HSM.

SUMMARY

During these tight economic times, small businesses require more storage. We think that’s because of three trends: (1) uncontrolled e-mails, (2) unnecessary storage of obsolete files, and (3) larger file sizes. We also thought of these solutions: (1) VMware’s Vmotion, (2) best practices, (3) file deduplication, (4) storage appliances, and (5) hierarchical storage management.


Image courtesy of Dell Ireland