The Washington Post
WASHINGTON — Administration officials are preparing to announce Sunday that they have met their Saturday deadline for improving HealthCare.gov, according to government officials, in part by expanding the site’s ability to handle 50,000 users at once. But they have yet to meet all their internal goals for repairing the federal health-care site, and it will not become clear how many consumers it can accommodate until more people try to use it.
As of Friday night, federal officials and contractors had achieved two goals, according to government officials who spoke on the condition of anonymity in order to discuss ongoing operations. They had increased the system’s capacity and reduced errors. On the other hand, the site’s pages do not load as fast as they want, officials said, and they are working to ensure large numbers of consumers can enter the site.
An official at the Centers for Medicare and Medicaid Services, the agency overseeing the federal health insurance exchange, said the site’s true capacity is somewhat murky because they need to see how it performs under “weekday traffic volumes” when demand is at its peak.
Federal employees and IT contractors were expected to work through the night Friday to try to reach one of the remaining targets: how many people per hour should be able to register and log onto the site. An earlier attempt to make the fix failed several days ago.
Nov. 30 was not originally intended to be a key date for the online enrollment system, but it took on outsized political and public importance when administration officials announced five weeks ago that the “vast majority of users” would be able to sign up for insurance through the site by that day. A combination of federal employees, outside contractors and a handful of technical and management experts have worked at breakneck speed for five weeks to improve the website’s performance as the White House has come under withering criticism from its political opponents and some consumers.
In an interview with Barbara Walters that aired on ABC on Friday night, President Barack Obama said he was confident that, in time, Americans would come to embrace his controversial health-care law.
“I continue to believe, and I am absolutely convinced, that at the end of the day, people are going to look back at the work we’ve done to make sure that in this country you don’t go bankrupt when you get sick, that families have that security,” Obama said. “That is going be a legacy I am extraordinarily proud of.”
Administration officials have said for several weeks they define success as having “the vast majority of users” be able to navigate the site and sign up for insurance. While they initially did not define what that meant, White House press secretary Jay Carney said earlier this month that the administration’s aim was to have 80 percent of users enroll through the site. Those working on the project have set speed and error rates as a way of measuring that goal.
A significant upgrade in the website’s capacity was carried out early this week, according to the government officials, which allowed HealthCare.gov to accommodate more people at once without causing the site to malfunction. For instance, many more consumers can simultaneously be on a section of the site designed to let people compare available health plans.
Even so, how much load the site can withstand depends on how much consumers are asking it to do. For example, simply looking around on the “learn” part of the site – where people can get basic information and a rough sense of prices – puts less strain on the system than the areas where people actually apply for coverage.
The upgrade that workers were planning to attempt at midnight Friday is intended to allow more people to use the initial stages of seeking insurance on the site. But the upgrade also meant that the site would be unavailable overnight from 9 p.m. Eastern time on Friday until 8 a.m. Saturday.
Administration officials have established a capacity goal of 80,000 consumers per hour being able to register and 320,000 people per hour who already have accounts being able to log in, according to federal officials familiar with this aspect of the project who spoke on the condition of anonymity to describe internal details.
The team has not yet attained its target of cutting the average load time across the site to no more than half a second, officials said, though it is running faster than before.
The blitz to rehabilitate the site has entailed an unusual public and private partnership over the past five weeks. The White House rejected the idea of turning over the work of fixing the site mainly to Silicon Valley allies, some of whom served as the president’s campaign operatives, deciding instead to selectively tap a handful of them to help steer the project.
Michael “Mikey” Dickerson, a Google employee who is on leave and volunteered on both of Obama’s campaigns, has led the twice daily calls in which the entire team has discussed ongoing problems and how to solve them. Civis Analytics chief technology officer Gabriel Burt, who served as lead analytics engineer and product manager for Obama’s 2012 reelection campaign, has also been deeply involved.
Jeffrey Zients, the former acting director of the Office of Management and Budget who was designated by the White House in late October to oversee the project, told reporters during a briefing this week the administration “brought in a fresh set of eyes” the week of Oct. 21 to examine the system. He said that survey concluded within a few days that the website could be fixed and that it did not need to be thrown out and a new one installed – but that the fix would require “additional talent and expertise” and a different way of approaching the site’s defects.
The work on HealthCare.gov took place primarily in two places: in a command center dubbed the “Exchange Operation Center,” or XOC, in Columbia, Md., and in the Tysons Corner area of Fairfax County, Va. QSSI President Bikram Bakshi – whose company developed the system’s data hub and became the project’s general contractor on Oct. 25 – said in the briefing “other business analysts, system analysts, developers and testers ⅛were€ scattered across the D.C. metro area.”
Since starting work, the team has “knocked through well over 300 bugs,” Zients said, and each evening the group has either put in hardware upgrades or fixed a glitch during the four- to five-hour maintenance window it has to work on the site.
The 25 people working in the command center, who represent all the major contractors that helped build the site, could evaluate HealthCare.gov’s real-time performance through 15 screens capturing its overall response time, error rate, volume of traffic and visitor makeup and which functions users are performing at a given time. When they see a disparity in the readings, Bakshi said, “then the team springs into action” to try to fix it.
John Engates, chief technology officer for the cloud computing company Rackspace, toured the project’s command center Monday with a handful of other private tech experts and said he was encouraged that they have streamlined the decision-making. “There isn’t a lot of ‘Let’s run it up the flagpole.’ That’s what’s helping things run more smoothly,” he said.
Before touring the command center, Engates, along with representatives from Salesforce.com, IBM and other firms, met in the Situation Room with senior White House officials, including Zients, chief of staff Denis McDonough, chief technology officer Todd Park and chief information officer Steven VanRoekel.
Salesforce.com President Marc Benioff, who hosted a fundraiser with Obama at his home Monday in San Francisco, said in a phone interview he has dispatched two employees to the White House to provide what he described as “coaching and mentoring.”
“A friend in need is a friend indeed. Silicon Valley is a friend of President Obama’s, and we’re going to help the government get through this,” Benioff said, noting his employees were taking no compensation for their efforts.
But there are also limits to this support. Benioff said he had “offered to rebuild the entire HealthCare.gov site for free and run it for five years at no charge” in direct talks with the White House, but the administration declined the offer.
A White House official said the administration could not have accepted the offer, because it ran afoul of federal contracting rules.