If they could do it all over again, Major League Baseball’s Web managers might have paid closer attention to how the Colorado Rockies planned to sell their World Series tickets.
Perhaps they would have pushed the team to use a ticket lottery to avoid the rush of millions of requests at one time. Maybe they would have grilled Paciolan, the California-based company that handled online operations, on their capability to manage the big moment.
As it is, things didn’t turn out so well.
“It flat-out didn’t work,” said Bob Bowman, chief executive of MLB Advanced Media, which operates websites for all 30 teams.
“Maybe we should have been more persistent,” he said. “Did we collectively not do enough? Yes. It’s our fault as much as anyone else’s.”
The Rockies’ first attempt to sell tickets over the Internet on Oct. 22 crashed shortly after it started, prompting a public-relations mess that the next day’s quick sellout did little to fix.
Anecdotal evidence fueled suspicion among empty-handed fans wary of the team’s claim of a “malicious attack” through cyberspace that foiled the initial ticket offering. Theories ranged from technical incompetence to a greed-driven conspiracy to funnel tickets to scalpers.
“Brokers immediately had hundreds, probably thousands, available,” said Michael Quinlisk, a 52- year-old engineering program manager from Colorado Springs. “How many people do you know who went online and got tickets? I know one.”
Intense demand served to ratchet emotions sky-high as local fans confronted a major shift in the way sports franchises do business. Namely, teams have found a way to take a cut from what they now politely call the “secondary market” – essentially a dressed- up, high-tech version of the system angrily decried as “scalping.”
But the story of the Rockies’ troubled ticket plan began with the decision to create an online-only marketplace – a decision that included team brass from president Keli McGregor to vice president of ticket sales Sue Ann McClaren to vice president for communications Jay Alves.
“The idea was to get the most tickets to the most fans,” Alves said. “We’d had such success with the tiebreaker game, distributing so many tickets quickly and efficiently online, we thought it was the best way.”
So what went wrong?
Series of unfortunate events
According to the Rockies, their technical partners and outside experts, the first attempt to sell tickets failed because a series of events unfolded – either simultaneously or separately – to disable the site and render the purchasing experience slow and frustrating.
David Butler, chief executive of Irvine, Calif.-based Paciolan, said troubles began on that Monday when the team’s website was first hit with “three different levels of attacks.”
He described two of them as denial- of-service attacks, in which a hacker sends millions of automated requests to overwhelm and eventually stifle the servers.
Those attempts succeeded Monday. But by Tuesday, Paciolan was prepared for the same kind of assault.
“For the first type of denial-of-service attack, we were hit 336,741,975 times (Tuesday) – that is a gargantuan attack,” Butler said. “That allowed them to tie up connections on computers. We’re not making this up.”
Paciolan said it doesn’t have clear numbers for Monday’s denial-of-service attacks, but said those attacks were “comparable” to Tuesday’s.
MLB’s Bowman said the crash also was related to the way Paciolan handled invalid online requests for tickets – storing the bad information instead of scrapping it.
“Add all that up, and in a short period of time, it got to be too much,” Bowman said. “It’s like a circuit breaker. It gets shut down.”
He said that wasn’t involved in the ticket-buying process until it crashed that Monday.
“We talked with Paciolan, and we thought Paciolan could handle the rush. We told them what to expect,” he said. “It flat-out didn’t work.”
Paciolan’s Butler said the second type of attack hit the purchasing system at the same time on Tuesday – 32,919,988 times.
“They were purely attempting to flood the site to bring it down,” he said. “We’re prepared for denial-of-service attacks, but we had never seen … this variant.”
However, computer-security experts say that while denial-of-service attacks of this scale do happen, the hackers’ motives usually involve financial gain – which doesn’t seem to be the case here.
“It sounds like the first two could have been really aggressive ticket-buying programs,” said Johannes Ullrich, chief researcher for the SANS Institute, a group that trains network-security administrators and runs the Internet Storm Center that monitors new security threats.
“If you have millions of automated attempts, they end up blocking each other from sending information through,” he said. “The denial of service is a side effect of these gold-rush moments.”
Two hours after tickets went on sale Oct. 22, Paciolan shut down the system and began working around- the-clock to repair its wounded servers and prevent similar attacks.
“We had a massive amount of servers,” Butler said. “The problem was that we had so many bad guys, people couldn’t get to the tickets.”
As far as automated programs – called “bots” – buying up tickets for scalpers and brokers, Paciolan recorded 3,360 automated purchase attempts on Oct. 23. The company said it was able to thwart these attacks before the bots, which mimic human behavior, were able to purchase tickets.
Butler said what distinguished these attacks from the others was the tremendous speed with which the bots got to the ticket-selection area of the site.
Paciolan’s logs of all the IP addresses that came to the site are being investigated by the FBI in Los Angeles to determine the source of the attack. The FBI last week confirmed that it received reports from Paciolan of a denial-of-service attack.
For the do-over on Oct. 23, Paciolan and MLB Advanced Media said it took about 20 minutes to “get things running smoothly,” with changing the flow of traffic to Paciolan’s site and Paciolan balancing the load on its servers.
Ullrich said that while it’s possible to do load testing in advance, controlling a single-day spike in traffic is “a hard and expensive problem” in terms of adding more bandwidth and protecting against attacks.
“You have to buy yourself out of these denial-of-service attacks,” he said. “It’s one of the hardest problems in the security industry to prevent.”
Easier for fans elsewhere?
Paciolan wouldn’t disclose how much money it spent on enhancing its network for the ticket sales, but said that overall it spends “hundreds of thousands” of dollars on its systems.
Butler said that more than 80 percent of the people who purchased tickets Tuesday were from Colorado, despite anecdotal reports that fans in other cities – particularly Boston – had an easier time getting through the website.
In the end, officials concluded, there was simply more demand than anticipated.
“We only sold 50,000 tickets in 17,000 transactions,” Bowman said. “Those are the odds.”
At Wednesday’s rally to honor the team’s late-season and postseason runs, 33-year-old James Nesbit proudly wore his Rockies jacket but still struggled to reconcile those long odds with his own experience.
“I don’t think they had true Rockies fans in mind,” he said. “I know people who’ve been going to games since they played at Mile High, and they weren’t able to get tickets.”
Online ticket selling is hardly a new concept, but fans entered an unusually furious marketplace with mixed feelings. The technical ease of purchase carries the side effect of an expansive geographic market.
“Why is a guy sitting in Duluth as interested as a guy in Denver?” asked shutout fan Quinlisk. “Because he can resell them. Local fans are due some consideration a bit above and beyond the general public that has Internet and a mouse.”
That’s a quaint notion but not grounded in the new realities of the primary- and secondary-ticket markets, said Stephen Happel, an economics professor at Arizona State University.
He submits that there was nothing wrong with the Rockies’ approach to selling World Series tickets – neither their online sale nor the subsequent surge of resale availability. And even the technical glitch was cleaned up a day later.
“This is one of these things where you have a strong sense of fairness – that you should have the right to see the World Series if you’re a fan – running into the realities of the world,” Happel said. “All these people arguing that the true fan gets screwed are really just arguing that they want to pay lower prices. They just can’t afford to get into the game. Well, I’m sorry, but this is life in the fast lane.”
It also illustrates how more and more sports franchises are embracing the secondary market. MLB Advanced Media recently entered a five-year deal with StubHub, the online ticket marketplace, to be its official online provider of tickets on the secondary market.
Teams’ piece of the action
After generations of sports owners fighting ubiquitous ticket scalpers, the online model has presented them with an opportunity to get a piece of the action. StubHub pays MLB Advanced Media a percentage of its take.
“We decided that instead of trying to fight and compete, let’s combine one global area and send those that want tickets on the secondary market to the same place,” said MLB Advanced Media’s Bowman. “It’s pervasive and growing dramatically. If you can’t fight it or beat it, embrace it and manage it.”
Individual teams have the option of whether to sign on with StubHub as their official secondary marketplace. The Rockies haven’t decided yet, said Alves.
But on its website, the team already offers season-ticket holders a chance to resell tickets for face value, although the team adds a 30 percent “fee” that’s paid by the buyer.
While rankling to would-be buyers shut out of the Rockies’ sale, the secondary market – which offered Game 3 tickets for hundreds above the high-end $250 face value – more often drives down prices, said StubHub spokesman Sean Pate.
But there’s going to be a period of adjustment.
“People not used to shopping for tickets online are looking for reasons they didn’t get a ticket,” said Pate. “There’s no God-given right to a face-value World Series ticket. In the supply-demand model, people get frustrated looking for a scapegoat, and they direct their negativity to the aftermarket.”
Even online competitors such as TicketsNow see the StubHub deal as creating a more vibrant market for everyone who cross-checks prices.
“There will be continued blurring of lines between primary and secondary markets,” said Mark Hodes, TicketsNow’s senior vice president. “Electronic ticketing will be more the norm than the exception, and you’ll see big sports leagues and even concert promoters begin to actively participate in the secondary market and legitimize it even more than today.”
Meanwhile, the Rockies continue to re-evaluate their missteps on the ticketing front to restore fans’ confidence.
“We need them to trust us again,” Alves said. “We need to let them know we’ve reviewed the process. Again, we’re sorry. We do need to make it better.”
Kimberly S. Johnson: 303-954-1088 or kjohnson@denverpost.com
