149
One line of code caused AT&T to lose $60 million
(engineercodex.substack.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 14 Nov 2023
149 points (96.9% liked)
Programming
17207 readers
218 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 1 year ago
MODERATORS
If one single line of code can make you lose $60M, surely you'll ensure due review processes and independent QA and clear requirements and regular audits and a middle management not only doing KPI monitoring for a failing upper management. Right? Rrrright?
Hahahaha, you're kidding right? I shit you not, I've literally seen a single line change almost cost a company £150MM during testing because "we need to test in prod because the guy we need to run the test hasn't got access to the QA environment"
Best part was the actual change, there was a bug where a number that should've been divided by 100 was being multiplied by 100, the dev somehow managed to implement the fix in such a way that the number was multiplied by a further 100.
Knight Capital Group lost $440 million in just 45 minutes due to a repurposed feature flag.
https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html?m=1
Kinda makes the att one seem tiny in comparison.