BYLINE: Tom Field
Nasdaq was back up and trading only six days after the World Trade Center catastrophe. CIO Gregor Bailar tells us how it was done.
Where were you on Tuesday morning, Sept. 11?
Bailar I was in Greenwich Village [about 10 blocks above the World Trade Center] heading north in a hired car. We didn’t hear the crash, but we stopped, turned and saw the smoke. It wasn’t clear what was going on. One Liberty Plaza, which is across the street from the World Trade Center, is Nasdaq’s headquarters, with mostly executive staff and sales staff, not operational staff. Our offices are on the 49th and 50th floors of that building; they saw the first plane hit. I immediately called headquarters, and they said, “Everything seems OK.” Next I tried to call my wife. I had trouble but eventually got a message through. Then I called Connecticut, where our primary data center is. It appeared that we didn’t have any issues.
Then the second plane hit. I thought: two planes, direct center of two different towers. Time to batten down.
I jumped in the car and told the driver, “We’ve got a major problem here; they just took out our two financial center buildings. Let’s get out of here and get to our Times Square site, where I know we have a lot of security staff, and let’s get this market locked down.” What was happening at 1 Liberty? They began evacuating after the first plane hit. Our security guards on their own accord evacuated our floor at least, so most of our people were on the ground when the second plane hit.
What did you do next?
We drove to Nasdaq’s market site, a building where we have a press studio and broadcast studio in Times Square. At the market site a group from the press was asking, “What are you going to do with this building? Shouldn’t we evacuate?” We needed to keep it up as our command center, so I was not interested in shutting it down. At 9:15, I determined with the rest of our staff that it would be necessary to keep the market closed at least until 10. Later we made the decision to shut down the market for the whole day.
Halting the market wasn’t a step you could take lightly.
We’ve never done that in the securities market, period. I remember that moment vividly because there were a lot of people on the phone call — 40 or 50 people — including senior management for Nasdaq, the New York Stock Exchange, the SEC. I was the ranking executive for Nasdaq, and my cell phone was fading in and out. They were saying, “Gregor, have you made the decision?” and I was saying, “Yes, halt!” They didn’t hear me. They said, “Gregor, we need your confirmation that we need to pull the markets.” I pressed some phone buttons so that the tones would stop the conversation. Everybody stopped talking, and I said, “Yes, halt the market.”
How did the command center operate?
When the first tower fell, it took out the power in Lower Manhattan, and in fact Nasdaq had a telephone switch there that was serving the market site in Times Square. So our market site telephones went down. We moved essential staff over to the Marriott Marquis [a hotel in Times Square] and sent the rest home.
At the Marriott, we went through a progression of information gathering. The first thing we had to understand was our personnel situation. Then we broadened the investigation to learn who was affected among our traders. Then we had to understand the situation from a physical perspective: Did we lose a building? Did we lose a data center? Did we lose connectivity? What have we got in the way of physical damage that’s going to take a long time to restore? Next we needed to know the regulatory situation: Are people trading today? What’s the landscape of the trading industry? It was literally in that order.
How did you handle those tasks?
In our Maryland center, they immediately set up a central depository for information about where all of our people were and who was unaccounted for. We had two people we couldn’t account for who we thought could have been in one of the towers; but it turned out they never got there. We also used this depository to track where all our executives were. So if you needed to get in touch with any executive, you would call that center — it was the executive locator from that moment on. Every hour or every two hours a listing would come through saying here’s where Frank is, here’s where Gregor is, with phone numbers. That was very important because a lot of us were on the move from site to site.
We also had a group in our Maryland center focused on the regulatory processes. Nasdaq trades from 8 a.m., so things had already occurred that day. We had to figure out how that would be affected if we restarted trading or kept it closed.
As for our trading companies, we could see by our network monitoring that there were certain sites that were not responding. We were pretty sure that there were 40 to 70 firms that had problems based on that monitoring.
Some of your traders were in trouble, but Nasdaq’s systems were all up?
Nasdaq is highly redundant, so just because we may have an office gone or they turn off the electricity in Lower Manhattan didn’t mean we were knocked out. We have servers in different buildings. Every single one of our traders is connected to two different Nasdaq points of presence or connection centers, and there are four connection centers alone in downtown Manhattan, none of which were really affected.
Including Manhattan, there are 20 connection centers around the United States, and every single server connects to two of those centers through two different paths, and often through two different vendors.
How did you help your traders get back online?
Obviously, this was a big factor in reopening the markets. By noon on Wednesday we had confirmed that if we reopened, 60 percent of the Nasdaq trading volume would be ready to trade. By the end of the day we called more than 300 firms, and of that only 30 were not going to be operational by the next day. We sent nearly 100 people from our centers to work with these folks. We were also given resources by Cisco, Dell and WorldCom. On Wednesday we had about 10 trading companies that were going to have to operate out of their backup centers. One of those had to be built entirely from scratch. We knew it would take at least until Friday for a couple of the companies to provision their backup sites. Meanwhile, on Wednesday I went to several meetings organized by Mayor [Rudolph] Giuliani specifically to look at the telecom and infrastructure of Lower Manhattan and discuss how we would deal with access and security. By Thursday’s discussions, we saw that the New York telecommunications infrastructure just wasn’t getting there. Access to the downtown buildings wasn’t available yet. That was when Monday was decided for the reopening.
How did you prepare for Monday?
We started industrywide testing on Saturday at 7 or 8 in the morning, and by 11:30 that morning, we had achieved 98 percent of the volume. And then on Sunday we did a half-day of retesting with people who wanted to add a little more volume capability. Then we sent people home, saying, “Get some sleep guys. We’ve got a big day tomorrow.” And people were in at 6:30 Monday morning to make sure everybody was connected and up and running. On Monday we had amazing volume — a 2.7 billion share day. There were a couple of firms that had more volume than they thought they could handle, so there may have been a couple of phone calls that didn’t get answered. We traded something near 5 billion shares that day between Nasdaq and the NYSE, which is a healthy volume on any day. To have that come six days after you literally were in the crosshairs of a terrorist attack…people were pretty happy. People were screaming happy.
What did Nasdaq lose over the downtime and what did it cost to get back up?
We have interruption insurance, so we hope to recover most of it, but it’s in the millions, and it could crest tens of millions.
What were the disaster recovery lessons for Nasdaq?
We learned that distributed systems are really good. You have to think about how your business has concentrated people or operational centers in certain places. You’ve got to consider if it’s the wisest distribution. We feel we were lucky having some folks in Connecticut and some in Maryland. Even if we had lost some of our senior management at 1 Liberty Plaza, we would have still had a senior team.
This event validates the distributed organization.
It certainly validates caution in how you concentrate your operations and staff. If you couldn’t get access to southern Manhattan, and that’s the only way you could open your doors, then you were in trouble. The other validation is one of network topology — to know that there are two network topologies, never just one — and we’ve been religious about that.
Were there other lessons? Are you going to change anything?
We did have extensive [crisis management] rehearsals for Y2K and for our conversion to decimals, and in these cases, the securities markets really worked together. So it was a natural thing for me to set up a nationwide teleconference for all the exchanges in the U.S. at noon every single day and say, “Hey folks, we want to talk about what’s going on.” We made decisions as an industry, and our openness of communication was critical.
One thing we’re going to try is to formalize more of these communication forums. We’ll want to trigger those automatically with the SEC. Trading is still a New York-centered industry, so there are also some things that we want the New York City government to do around communications. But you can’t script the rest. You have no clue what the next disaster is going to look like. Will it hit a power substation? Will it hit the trains coming in so none of the traders can get in? You really need to be able to communicate and have a place for people to talk about what’s going on. We already require many of our executives to carry two cell phones, each with different service providers. We are focusing on making sure the network is resilient and that the design is able to endure and that we can communicate.
After living through this, what would you advise other CIOs to consider?
This was a true test of people’s backup strategies. Did you ever test your backup strategy? Have you worked out of your backup center? Do you know how to get people there? Do you know the critical phone numbers? A lot of people don’t have phone numbers as part of their continuity of business plan. I think people will have to look very carefully at their backup strategies and see whether they can communicate with everybody easily, whether the phone numbers are not stored in that same building that could experience the disaster, and whether they’ve got hot backups. Hot backups are going to be much more popular than they have been in the past.
Any last thoughts about your experience?
I’m very proud of Nasdaq and their response, and I’m proud of the industry because it was an incredibly tough time. People were caught up in emotions you can’t imagine. Many of our staff have been going to memorial services almost daily. Yet throughout that they’re coming to work. Not with smiles on their faces but with determination and with no negativity. Everybody got counseling the first day. Giuliani visited with a lot of our people at different locations, and there was just an outpouring of community you don’t get too often in normal daily life.
How did this affect you personally?
The emotional hit didn’t come until Friday night, when I got to my house [in suburban Washington, D.C.] for the first time since Tuesday. My kids were asleep by the time I got home. My wife had to go out, so I was alone there. That’s when it hit me.
Editor’s note: As CIO went to press, Gregor Bailar accepted the CIO position at credit card company Capital One, where he was scheduled to start Oct. 24.