Hospitals, schools and a number of government agencies in western Colorado were left without Internet service for more than six hours two weeks ago because of a failure in Qwest’s fiber-optic network.
The outage forced some officials to examine their contingency plans and left others wondering about the reliability of the company’s network.
“Qwest was supposed to put in a redundant network,” said Rick Smith, director of general services for the city of Cortez, whose schools, hospital and city office were knocked out. “You’re talking about hospitals relying on the Internet to trade over X-ray images and things like that. It’s real important.”
Qwest said its network is redundant – meaning it has immediate backup in case of an outage. But in this case, both the primary and backup networks failed at roughly the same time, causing the extended outage, said Qwest’s vice president of network operations Amador Lucero.
“This is the first time in my experience where I have seen (the primary and backup networks) go down at the same time,” said Lucero, who has more than 30 years of experience in the field.
Qwest said a fallen tree in a remote area in the state took out the primary fiber ring, which was above-ground because of the mountainous conditions.
Eighty-seven minutes after that network shut down, its backup failed because of an open connector – which connects two ends of a fiber ring, Lucero said.
“I don’t know what caused it to open up,” Lucero said Friday. “We are still smacking our heads against the wall trying to figure out what the actual cause is.”
The outage took out Internet and cellphone service in western Colorado, including Grand Junction, Durango, Cortez and Montrose, from about 8 a.m. to 2:30 p.m. on Aug. 20.
Cellphone service from Verizon, which routes its calls through Qwest’s network in that region, was also impacted by the outage. The problems didn’t affect landline and 911 phone services.
“This, by far, was the longest outage we’ve had,” said Eric Pierson, information-systems manager for the city of Durango.
Pierson said the city’s Internet comes via the state’s Multi-Use Network, or MNT. Colorado teamed with Qwest on the MNT project to build a high-speed fiber-optic network linking rural and urban Colorado. The network was completed in 2006.
Don Randolph, a computer technician for Southwest Memorial Hospital in Cortez, said outlying clinics weren’t able to access the hospital’s patient records because of the outage. He said the hospital’s pharmacists couldn’t order medicine online, so they had to spend more time ordering over the phone.
“No explanation”
“We’ve gotten no explanation,” Randolph said. “We’re puzzled.”
He said the hospital is reviewing its options, which include satellite Internet service.
Julie Postlethwait, a spokeswoman for the Colorado Department of Personnel and Administration, said 24 state agencies were affected by the network outage. She called it a minor annoyance.
She said state officials are scheduled to meet this month with Qwest, which provides local phone service in 14 states in addition to operating a nationwide fiber-optic network, to get more details about the problem.
David Bruzzese, a spokesman for Mercy Regional Medical Center in Durango, said the hospital’s Internet was down, but its patient care wasn’t affected.
He said certain nonclinical employees, such as those in accounts receivable, left work early during the outage because their systems were down.
Qwest’s Lucero said the company is working to ensure that the problem doesn’t occur again.
“We are chasing down every one of the connectors in the 14 states to ensure that we don’t have this kind of situation come up again,” Lucero said.
The tree-damaged fiber ring and the open connector were located in Colorado.
“We consider service diversity and survivability of the network as extremely important,” said Qwest spokesman Jon Lentz.
Staff writer Andy Vuong can be reached at 303-954-1209 or avuong@denverpost.com.
This article has been corrected in this online archive. Originally, due to a reporting error, it incorrectly stated David Bruzzese’s affiliation. Bruzzese is the spokesman for Mercy Regional Medical Center in Durango.
