How This Company Kept Running During a 59-Day Power Outage

How This Company Kept Running During a 59-Day Power Outage2018-12-04T18:31:30+00:00

Project Description

The Incident

“It was December 18, a nice, bright Monday morning and I was just arriving at the office,” Diane Hawkins, vice president of people and performance at Meeting Professionals International, said. “When I pulled into the parking lot, it was filled with fire trucks from about four different fire departments and everyone had been required to evacuate the building.”

Meeting Professionals International (MPI), a worldwide meeting and event industry association, had suffered a fire at their headquarters in Dallas.

“All we knew at that point was that there was some sort of fire,” Hawkins said. “But I don’t think we knew the magnitude of it at that particular moment.”

An electrical fire on the building’s tenth floor had burned out its core switching, resulting in a building-wide power outage that kept MPI out of their offices for 59 days. After the power was shut down and the fire was contained, employees were allowed back inside to retrieve critical items—purses, keys, laptops—and were told to work from home for the rest of the day.

“We had no clue what to expect,” Hawkins said. “We kept hearing, ‘The building is going to be closed down and won’t open until 2 o’clock this afternoon.’ And then it was, ‘It’ll be closed all day.’ Some employees didn’t even go up and get their laptops because they thought that they would be getting back into the building.”

MPI was able to accomplish a few things that first day. However, it soon became clear that the incident would turn out to be more than a minor setback. From phones and computers to their backend operating systems—the fire brought several parts MPI’s business offline.

“IT is everything that we do and touch,” Michael Crumrine, vice president of information technology, said. “So, you can imagine, when we get to talking about the incident, there’s not an area of IT that it didn’t touch.”

Engaging the Business Continuity Plan

Aldridge’s remote monitoring and management (RMM) team sprang into action as they received a series of alerts about MPI’s infrastructure. Already aware of the fire and subsequent power outage, Aldridge quickly engaged MPI’s business continuity plan.

“I contacted Aldridge because we knew we were going to be taking our systems down,” Crumrine said. “They had to do remote shutdowns of our servers before the power was just turned off.”

Aldridge’s service team immediately began restoring critical files for MPI.

“Initially, they expected to get back into their building within a week or so,” Bill Taylor, Aldridge’s Dallas CIO, said. “As time went on and we saw they were going to be out of their offices for a longer period of time, we restored their data to their cloud which allowed end users to be able to access it from wherever they had internet connectivity.”

MPI lost power to several servers in their local data center. Aldridge worked around the clock during the outage to keep their business up and running.

“We had to depend on off-site backups and that’s where our partnership with Aldridge really ended up paying off,” Crumrine said. “Aldridge had to start rebuilding those backups to get us our files and some of our critical data that we needed to keep the business running.”

Still out of their offices and working remotely a week after the fire, MPI’s leadership began discussing options for temporary offices.

“We were being told that the whole core electrical system of the building was burned out,” Hawkins said. “They had to rebuild it and replace the entire system, then the building had to be re-inspected and given clearance. We weren’t sure if we were going to be out 30 days, 60 days or 90 days because we were kind of at the mercy of contractors, building management, and the electrical company as well as the building inspectors.”

Two weeks after the initial incident, MPI moved operations to the offices of one of their business partners.

“The first 50 of those 59 days, the building was without power altogether,” Crumrine said. “But we felt it was critical to end 2017 and start 2018 having people in a spot where they were working together, seeing each other eye to eye, solving problems and getting things done.”

While MPI moved 55 of their employees into the space provided by their partner, Aldridge worked in tandem with that company to set up temporary workstations and phones.

“When we made the decision to move into our partner area to work, Aldridge sent talent over there to make sure people knew how to get connected,” Crumrine said. “It was really a partnership. Whatever we needed as far as IT support, they were jumping in and helping us problem solve.”

Taylor said Aldridge’s strategy and support teams held daily “checkpoint” calls with Crumrine and the rest of MPI’s leadership to confirm the company’s needs were being met. Even as the holiday season approached, Aldridge continued to work 24/7 on MPI’s environment.

“I ended up working with someone from almost every area of Aldridge at some point,” Crumrine said. “It was really all hands on deck. This happened on the 18th of December and we were heading into holiday hours, but even with that, I had access to the people that I needed.”

Lessons Learned

Crumrine said that while MPI did have a business continuity plan in place, he never imagined that it would be put to the test during a two-month outage.

“We have a VoIP phone system provided through Aldridge, we have cloud hosting for our website and database servers, and our critical ones are hosted off-site,” Crumrine said. “We’re an Office 365 client with SharePoint, so I would say most of our infrastructure can be accessed regardless of where we are. But business continuity, for us, was thinking much shorter term.”

While MPI’s business continuity plan worked as intended, Crumrine said the outage highlighted a few areas—particularly communication protocols—that needed improvement.

“The whole communication plan is so critical,” Hawkins said. “We wanted to keep our employees up to speed, but in that initial week or so, it was clear that we needed to look for different ways of communicating.”

“We’ve recently implemented [mass] texting for our employees,” Hawkins said. “If something like this happens again and they can’t access email, at least we’ll have a way of getting in touch with them through their mobile devices.”

On February 14, 59 days after the initial incident, MPI returned to their offices.

“Once they were able to get back in, we powered them up,” Taylor said. “None of their equipment was damaged, there was no data loss. Once they moved back in, we resynced the data that had changed in the cloud back to their physical systems.”

According to Crumrine, the disaster made MPI more aware as an organization and he said he couldn’t imagine going through the outage without Aldridge.

“We had access to people, knowledge and resources,” Crumrine said. “Without [Aldridge], we would have kind of had to take that on ourselves. That’s really been the value of Aldridge, even beyond just the outage that we had, it was the access to experts as an extension of our team.”

About MPI

Meeting Professionals International (MPI) is the largest meeting and event industry association worldwide. MPI has a global community of 60,000 meeting and event professionals, including more than 17,000 members with more than 90 chapters in 19 countries.

About Aldridge

At Aldridge, our team of IT professionals can provide your business with the resources it needs to reach its potential. We specialize in service and support, and can deliver a strategy that effectively manages your IT. Our approach relieves clients of the hassle of technology management so they can focus on running their business.