In this Dec. 22, 2019 photo, Boeing, NASA and U.S. Army personnel work around the Boeing Starliner spacecraft shortly after it landed in White Sands, New Mexico. (Bill Ingalls/NASA via AP)

In this Dec. 22, 2019 photo, Boeing, NASA and U.S. Army personnel work around the Boeing Starliner spacecraft shortly after it landed in White Sands, New Mexico. (Bill Ingalls/NASA via AP)

Boeing didn’t perform full test of its astronaut capsule

A Dec. 20 mission missed critical maneuvers and flew into the incorrect orbit.

By Chabeli Carrazana / Orlando Sentinel

ORLANDO, Fla. — In the weeks since Boeing flew its astronaut capsule on an ill-fated demo flight, questions about the company’s testing procedures prior to the mission have started to emerge — putting safety at the center of a debate on the future of human spaceflight.

NASA is on the verge of sending astronauts back to space from U.S. soil for the first time in almost a decade, but it’s doing it with commercial companies who are taking the lead on key decisions when it comes to flying with a crew. Now it seems some of those decisions are raising flags among safety experts.

Boeing and NASA officials are expected to release the results of an independent investigation into the set of issues that occurred during Boeing’s late December test of Starliner, its astronaut crew capsule, within the next week. But speaking to the Orlando Sentinel, members of NASA’s safety advisory panel expanded on some of the testing decisions Boeing made that drew questions about whether Starliner was ready to fly.

Critically, the panel learned early this month that Boeing did not perform a full, end-to-end integrated test of Starliner in a Systems Integration Lab with ULA’s Atlas V rocket. The test typically shows how all the software systems during each component of the mission would have responded with each other through every maneuver — and it could potentially have caught the issues Boeing later experienced in the mission.

“It’s pretty exhaustive. You gotta do that,” said Christopher Saindon, a former member who ended his tenure on the panel in mid-February. “That was somewhat surprising to us on the panel. There were certainly gaps in the test protocol.”

It was software that ultimately did fail Boeing when it flew Starliner on a Dec. 20 mission intended to dock with the International Space Station. The capsule’s internal clock was 11 hours ahead, causing it to miss critical maneuvers and fly into the incorrect orbit. Then, communication issues potentially caused by cell towers in the area blocked Boeing from sending a command to rectify the orbit. Starliner, the company determined, wasn’t going to be able to reach the space station.

But in the process of bringing it back down and re-checking its software, the company caught yet another issue that could have caused Starliner to collide with its service module when the two separated prior to the capsule’s return to Earth. Teams were able to correct the issue before to the capsule’s return on Dec. 22, but the multitude of problems have led NASA to call for a full re-verification of Boeing’s software — a process that will take analyzing about a million lines of code.

Software issues are also plaguing another arm of Boeing, which is dealing with the fall out of problems with its 737 Max airplanes that led to the deaths of 346 people and has grounded the planes.

Still, he added, “that doesn’t mean it’s not a business right? And they’re trying to do things efficiently and cost-effectively.”

Boeing said it followed all of the testing procedures NASA required of it prior to the Starliner test flight. The agency gave Boeing the green light to proceed after a flight readiness review prior to the mission.

During a call with reporters earlier this month, John Mulholland, vice president and program manager for Boeing Commercial Crew program, said teams ran through a series of integration tests to check different parameters before Starliner went into formal qualification testing. He didn’t speak specifically at the time about the end-to-end full system integration test the panel is concerned about.

“When we look at our software verification process, we’re looking at that end to end process. And obviously, at a minimum, these two code errors made their way fully through that process that was designed to detect and correct,” Mulholland said. “And so we’re looking at at each of those work streams and identifying ways to make it more robust.”

Because of the nature of the program, commercial providers have more flexibility in choosing how they will test their vehicles. NASA just has to approve that the testing was sufficient or request higher-level certification requirements, said George Nield, a current member on the panel and the former associate administrator for commercial space transportation at the Federal Aviation Administration.

“Software is a challenging (and potentially expensive) thing to test in a realistic environment,” Nield said. ” … You would think you would always want to test the software with the computer connected to all the real hardware, but the problem is, if it is designed to control the system during launch, or in space, you won’t really be testing the real situation if the hardware is just sitting on the ground.”

The panel expects to hear more from Boeing soon on why it decided to skip full end-to-end, integrated testing. Until then, experts can only guess at what impact that may have had when Starliner took flight.

“Since the two noted problems (during the flight) occurred at system interfaces, one would have to speculate that there was some weakness in the integrated testing,” said Don McErlean, a current panel member and former director of aerodynamics, mechanical engineering and industrial design at L-3 Technologies’ platform integration division. “The reason for that weakness is one of the things (NASA’s Independent Review Team) is investigating.”

Saindon said the advice the panel gave NASA was to “look really hard now” and “get more involved” in the testing processes of its providers on the program, called Commercial Crew. Each provider, Boeing and SpaceX, is expected to perform a test flight without crew followed by a test flight with astronauts on board.

Boeing was initially awarded $4.2 billion to perform the feat, while SpaceX got $2.6 billion. SpaceX’s test flight in March 2019 was successful without any major issues. Elon Musk’s rocket company is expected to fly with crew onboard its capsule, Crew Dragon, later this year.

Because of the issues Boeing experienced, it’s still unclear whether the company will have to repeat its uncrewed test.

“If you decide the next flight will be crewed, you have to look really hard and make sure that all the integration testing has been done,” Saindon said.

During Boeing’s fourth-quarter call with investors in late January, the company said it has taken on a $410 million charge in case it did have to perform another test flight without humans onboard Starliner.

With the return of human spaceflight imminent for the U.S., the pressure to ensure that Boeing and SpaceX are prepared to fly humans is peaking.

Both providers still have outstanding work on parachute designs, an area of repeated challenges for Boeing and SpaceX. NASA is also now conducting a more thorough workplace safety culture study with Boeing following the recent Starliner challenges. The agency already completed a similar study with SpaceX.

The lingering question is what, if any, other issues teams may not have caught with the capsules. With Boeing in particular, one of the questions now is: “What are the unknowns that are still there with the Starliner capsule and its integration with the ULA rocket?” Saindon said.

NASA and Boeing have both previously acknowledged that both their teams failed to catch issues at multiple points in the review process. McErlean, whose area of expertise was certification, said particular focus should have been put at the points where the systems interact with each other — the kind of thing that an end-to-end full integration test could have caught.

“More than likely, where one system has to interface with the other system,” McErlean said, “is the garden in which those problems grow.”

Talk to us

More in Herald Business Journal

Everett to consider allowing three more pot shops in city

After months of economic, planning and public safety review, the city council could vote next month.

Economic Alliance and Lynnwood offer new business grants

The grants are derived from the federal Coronavirus Assistance, Recovery and Economic Security (CARES) Act.

Paine Field gets $5M grant to remedy a CARES Act oversight

Shortchanged earlier, the Snohomish County airport is the recipient of a new federal grant.

Amazon’s buying spree of used planes goes against green pledge

Airlines are being spurred to hasten the retirement of their oldest, fuel-guzzling aircraft.

An update: We’re proud and humbled by our readers’ support

The Daily Herald investigative fund has grown, and now we’re working to expand environmental coverage.

Commentary: The 737 Max debacle won’t be the end of Boeing

The plane may actually be the bright spot in Boeing’s airliners business.

Panel blasts Boeing, FAA for ‘horrific culmination’ of failures

Investigators found that the company had a financial incentive to avoid more pilot training.

Marysville offers another round of CARES Act grants

Funds are available for those who need help paying for housing or business expenses amid COVID-19.

Port again wins millions in grant money for mill site revamp

The Port of Everett successfully reapplied for federal funding after losing $15.5 million last year.

‘Better with Boeing’ campaign aims to keep 787 assembly here

A new marketing effort hopes to persuade the company to keep Dreamliner work in Everett.

737 Max engineer didn’t know details of flight control system

The program’s leaders only assumed pilots would react differently to the triggering of the MCAS.

Everett company faces $230,000 fine for safety violations

State inspectors allege that Chilos Builders exposed workers to hazards at area construction sites.