Warnings ignored: Telstra failed to update technology | Stockhead
Telstra executives have admitted the nationwide outage would not have happened had the company completed a warned software upgrade. Telstra executives have admitted the nationwide outage would not have happened had the company completed a warned software upgrade.
Telstra chief executive Vicki Brady has declared the recent network failure as “not acceptable”. Pic: Getty ImagesBy Jared Lynch and Jack QuailTelstra was repeatedly warned to update a 15-year-old piece of equipment responsible for a nationwide network outage, prompting the telco’s executives to concede under questioning at a parliamentary inquiry that the calamitous system failure would not have happened if it had completed a necessary software upgrade. Chief executive Vicki Brady, chief financial officer Michael Ackland and other executives last Friday fronted a Senate inquiry into the meltdown, where it was revealed the $56bn telecommunications giant was told by a supplier as late as January that a server, that helped keep time across its network, needed to be upgraded. But it was never done. Nor did Telstra replace the ageing server, with the company’s resilience boss Gerald Tracey eventually admitting after persistent quizzing that, if it had, the outage – which shut down train networks, digital payments and resulted in more than 600 calls failing to connect to triple-0 – would not have happened. Declaring the network failure “not acceptable”, Brady told the inquiry that Telstra’s “controls were not good enough”. She also conceded the company, which most recently posted an interim half-year profit of $1. 2bn, couldn’t guarantee there would not be another outage. She said the complexity of modern telecommunications networks meant no provider could promise uninterrupted service. “I would love to be able to sit here and say there will be zero outages,” she said. “The reality of a complex network environment with fast-evolving technology, you just can’t. No telco around the world could guarantee that. “But absolutely, we accept here that it wasn’t good enough. Our controls and our processes definitely let us down, and that impacted a lot of people. And we are deeply sorry for that. ”The revelations prompted a fiery response from Communications Minister Anika Wells, who said Telstra’s explanation was “just the beginning” of the answers it needed to provide. “Telstra still has further questions to answer about how this happened, what could have been done to stop it, and how they will prevent this happening again,” Wells said. ACMA investigatingShe pointed to the Australian Communications and Media Authority’s investigation of the outage. ACMA chair Nerida O’Loughlin told the inquiry the watchdog would be examining a number of factors, including whether Telstra met its obligations to conduct welfare checks on customers unable to contact emergency services during the outage. “Our investigation will cover a range of matters, including what Telstra knew when on the morning of the 8th of July, whether it alerted its customers and its key stakeholders effectively, including the custodian and us,” O’Loughlin said. “Then we’ll also look at what Telstra knew in terms of the risks and vulnerabilities in its voice networks, and whether it has taken appropriate steps to meet its obligation to maintain its network to facilitate triple-0. ”Greens communications spokeswoman Sarah Hanson-Young attempted to link Telstra’s increasing profits to a growing number of outages that have afflicted the network, pointing out that the number of outages had risen from 3641 in 2024 to 5221 last year. Brady responded by saying the number of minutes customers had been impacted by outages had fallen. “Our customer impacting minutes from major incidents are actually down 91 per cent,” she said. “Our times are down 91 per cent and our time to restore is down 22 per cent. So our internal measurement of this would show the investment and steps we’re taking are improving the resilience of our network. ”Customer compensationAddressing the matter of customer compensation, Brady was unable to say what the ultimate cost would be, suggesting most people would be given some kind of credit. “I would expect a majority of customers will be in credit, but that will vary depending on some of the organisations we’re engaging with,” she said. Hanson-Young asked why “credit is better than cash”, to which Brady responded that credit was the most effective and quickest method to compensate customers. “I just don’t know how much people will want credit if they can’t rely on their mobile phone,” the Senator retorted. It was also revealed at the inquiry that Labor’s new triple-0 custodian only discovered the mass network outage after the duty officer realised her own phone had lost reception, rather than through a notification from the telco. “The duty officer was first alerted to a potential issue just after 7 because her Telstra phone wasn’t working,” the triple-0 custodian’s first assistant secretary Clare Chapple told the committee. “So she discovered it herself, checked Down Detector, and actually text-messaged some Telstra contacts to confirm if what we were seeing was an indication of a major outage, and it kind of proceeded from there. ”How it occurredExplaining how the outage occurred, Telstra’s executives told senators that an undocumented design change and a software update that was never applied reset its network clocks back to 2006, sparking a cascading national outage. It began at 3. 38am on Wednesday, July 8, when technicians restored power after swapping a faulty chassis on one of three network time protocol servers that keep the telco’s mobile network in sync. Brady said Telstra was told by supplier Scientific Devices that there was a risk of failure on a key piece of equipment, which cost about $30,000, that eventually sparked the outage. “We know that our teams did get the alert of a software update. They assessed that,”Brady said. “This is not an excuse, but putting in context, there are many software updates and changes getting made across the network. Any software update creates a risk and we put it into testing. “In this case, they thought, given the design that was in place, it was not relevant, because we were not using this equipment in the way that would trigger this risk. ”Brady said had Telstra completed the software update when it was warned, the “outage may not have occurred”. And Tracey said that, had Telstra replaced it entirely, it also would have avoided the meltdown. “A newer piece of hardware? Operating in the same design that we intended to? The issue wouldn’t have happened,” he said. This article first appeared in The Australian asWarnings ignored: Telstra failed to update technology.