149
One line of code caused AT&T to lose $60 million
(engineercodex.substack.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 14 Nov 2023
149 points (96.9% liked)
Programming
17314 readers
168 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 2 years ago
MODERATORS
If one single line of code can make you lose $60M, surely you'll ensure due review processes and independent QA and clear requirements and regular audits and a middle management not only doing KPI monitoring for a failing upper management. Right? Rrrright?
AT&T made 120.74 billion in 2022. They can afford a lot of bad code.
~~Revenue is not profit.~~
Edit: Jesus. Their profit was $70b in FY 2022?! On $120b in revenue? Maybe I’m the one misunderstanding.
With a +/-0.1% margin of error, the difference between $60M and $70B is $70B.
Do you think their profits will be dented by a $60 million loss?
“60,000 people lost full phone service, half of AT&T's network was down, and 500 airline flights were delayed”
I'm sure AT&T care about that due to their humanitarian nature.
AT&T is not humanitarian non-profit company. It should worry about increasing its profits by providing people with good product so that people choose them over competition.
That hasn't seemed to be their overall strategy considering how shitty their service is.
I personally had better experience with them than Verizon. But whatever. The fact that they have customers today does meant that they provide competitive service. Today, you can easily switch provider.
AT&T does a lot more than provide phone service.
It is still a provider.
Okay? And?
I can ask the same about your post. My point is that you can change it.
Which would matter if that was AT&T's only source of revenue. Now tell me how you can change ISPs from AT&T easily. Especially in a smaller city or town.
Natural monopoly and anti-competitive behavior would like to have a little chat with you in that dark alley over there...
Yes, anti-monopoly laws are important to provide competitive market.
Hahahaha, you're kidding right? I shit you not, I've literally seen a single line change almost cost a company £150MM during testing because "we need to test in prod because the guy we need to run the test hasn't got access to the QA environment"
Best part was the actual change, there was a bug where a number that should've been divided by 100 was being multiplied by 100, the dev somehow managed to implement the fix in such a way that the number was multiplied by a further 100.
Knight Capital Group lost $440 million in just 45 minutes due to a repurposed feature flag.
https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html?m=1
Kinda makes the att one seem tiny in comparison.