A bug in SpaceX’s communication system kept the company in the dark about potential satellite collision

On Monday, a European satellite changed its position in orbit to avoid a potential collision with one of SpaceX’s Starlink satellites — one of 60 probes the company launched in May to beam internet coverage down to Earth. The European Space Agency (ESA), which operates the satellite, performed the maneuver after calculating a higher-than-usual probability that the two satellites might run into each other. SpaceX did not move its satellite, blaming a computer bug that prevented proper communication with ESA.

Maneuvers like this aren’t uncommon. Every now and then, satellite operators will slightly alter a spacecraft’s position if they calculate an uncomfortable chance that their vehicle might hit another vehicle. No one wants a collision, especially since these satellites are moving through space at several thousands of miles per hour. At those speeds, an impact can cause spacecraft to break apart into hundreds of pieces. The resulting high-speed junk could potentially run into other satellites, possibly creating more dangerous debris.

This particular scenario with ESA raises some concerns since SpaceX’s probes are the first of nearly 12,000 internet-beaming satellites the company intends to put into a low orbit around Earth. The sheer size of the planned Starlink constellation has prompted many space experts to speculate how these vehicles might increase the chances of collisions in space. If satellites are already having to move out of the way of a Starlink satellite, how often is this going to happen when there are thousands of these vehicles in orbit?

Another worry revolves around SpaceX’s decision to not move the Starlink satellite. ESA officials said that they did not have the best communication with SpaceX leading up to the maneuver, and the agency ultimately made the decision on its own to move its satellite without SpaceX’s input. Initial reports claimed that SpaceX had “refused” to move the Starlink satellite, but SpaceX says the bad communication was not intentional and that a bug in the company’s “on-call paging system” prevented the Starlink team from getting additional email correspondence from ESA.

“SpaceX is still investigating the issue and will implement corrective actions,” a company spokesperson said in a statement. “However, had the Starlink operator seen the correspondence, we would have coordinated with ESA to determine best approach with their continuing with their maneuver or our performing a maneuver.”

This situation started last week when ESA realized that its Aeolus satellite — an Earth-observing spacecraft that launched in August 2018 — might come close to a Starlink satellite that was in a relatively low orbit, about 198 miles (320 kilometers) up. Most of the Starlink satellites have raised their orbits to around 342 miles (550 kilometers) up, but SpaceX decided to lower a couple of them to test the process of taking these vehicles out of orbit. One of these two satellites was coming into close proximity with Aeolus.

Officials at ESA said they contacted SpaceX about a week ahead of the potential collision to see if the company planned to move the Starlink satellite. “Every now and then, the other object is actually an operational spacecraft,” Klaus Merz, with ESA’s Space Debris Office, tells The Verge. “And then of course, we try to coordinate. The very basic thing is you want to know if the other one has maneuvering plans already.” On August 28th, SpaceX informed ESA via email that the company did not intend to move the Starlink probe, according to Merz. “They said at that point in time they had no plans,” says Merz. SpaceX confirmed that it exchanged the initial email with ESA. At that time, the probability of collision was about 1 in 50,000, according to SpaceX, which is too low to require any preventive action.

As the potential collision date drew closer, Merz says that ESA gathered data from the US Space Surveillance Network, an array of ground-based telescopes that track objects in orbit, and combined it with their internal data about the size and shape of the Aeolus spacecraft. Using that information, the ESA experts calculated even higher probabilities that the two satellites might hit one another — eventually getting to a more than 1 in 1,000 chance of an impact. For the satellite community, that’s high. Typically, satellites make moves when there is a 1 in 10,000 chance of collision.

Merz says ESA contacted SpaceX daily about its evolving calculations, but the agency did not receive any additional replies after the original email response. SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action.

Typically, the United States Air Force, which monitors space traffic, will issue warnings, or conjunction data messages (CDMs), if there is a high probability of a collision. The threshold for sending out a warning is when there is a probability of more than 1 in 10,000 of an impact. The Air Force’s 18th Space Control Squadron, or 18 SPCS, which issues these warnings, confirmed to The Verge that it sent nine updates to both ESA and SpaceX in the 72 hours leading up to the closest approach of the two satellites.

A spokesperson also noted that both SpaceX and ESA have an agreement with US Space Command that allows them to receive additional advanced warnings of these threats up to seven days beforehand (rather than the standard three days). “Both ESA and SpaceX also submit their [data] to 18 SPCS on a regular basis so they received additional CDMs that supported possible maneuver planning,” a spokesperson for Air Force Space Command said in a statement to The Verge. “Totaling messages for this event, ESA received 32 CDMs, and SpaceX received 29.” SpaceX did not say if the company received these alerts from the Air Force or if the bug prevented the company from seeing them. The Air Force acknowledged that the CDMs are sent via email.

Even if ESA hadn’t moved the Aeolus satellite, it’s possible the Starlink spacecraft may have moved on its own if it detected a potential impact. SpaceX CEO Elon Musk claims that each Starlink satellite is programmed with its own autonomous space debris tracking system, which it will use to move out of the way of potential debris if necessary. SpaceX confirmed to Quartz that its Starlink satellites have made a total of 16 autonomous maneuvers in space, but did not say when they occurred.

Ultimately, the entire situation illustrates a need for better communication, argues Merz, especially as more satellites are launched into space by SpaceX and others. Companies like OneWeb and Amazon have also proposed sending hundreds to thousands of satellites into orbit to beam internet connectivity to the Earth below. “I think trying to solve these things via email is not the future,” says Merz. “We should have something that is getting more efficient.”

Update September 4th, 1:30PM ET: This post was updated to include a statement and additional information from the US Air Force.