174
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 18 Feb 2024
174 points (95.3% liked)
Ask Lemmy
26904 readers
625 users here now
A Fediverse community for open-ended, thought provoking questions
Please don't post about US Politics. If you need to do this, try !politicaldiscussion@lemmy.world
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
founded 1 year ago
MODERATORS
One time I was deleting a user from our MySQL-backed RADIUS database.
DELETE * FROM PASSWORDS;
And yeah, if you don’t have a WHERE clause? It just deletes everything. About 60,000 records for a decent-sized ISP.
That afternoon really, really sucked. We had only ad-hoc backups. It was not a well-run business.
Now when I interview sysadmins (or these days devops), I always ask about their worst cock-up. It tells you a lot about a candidate.
Always skeptical of people that don't own up to mistakes. Would much rather they own it and speak to what they learned.
This is what I was told when I started work. If you make a mistake, just admit to it. They most likely won't punish you for it if it wasn't out of pure negligence
It's difficult because you have a 50/50 of having a manager that doesn't respect mistakes and will immediately get you fired for it (to the best of their abilities), versus one that considers such a mistake to be very expensive training.
I simply can't blame people for self-defense. I interned at a 'non-profit' where there had apparently been a revolving door of employees being fired for making entirely reasonable mistakes and looking back at it a dozen years later, it's no surprise that nobody was getting anything done in that environment.
Incredibly short-sighted, especially for a nonprofit. You just spent some huge amount of time and money training a person to never make that mistake again, why would you throw that investment away?
Exactly!
I was a sysadmin in the US Air Force for 20 years. One of my assignments was working at the headquarters for AFCENT (Air Forces Central Command), which oversees every deployed base in the middle east. Specifically, I worked on a tier 3 help desk, solving problems that the help desks at deployed bases couldn't figure out.
Normally, we got our issues in tickets forwarded to us from the individual base's Communications Squadron (IT squadron at a base). But one day, we got a call from the commander of a base's Comm Sq. Apparently, every user account on the base has disappeared and he needed our help restoring accounts!
The first thing we did was dig through server logs to determine what caused it. No sense fixing it if an automated process was the cause and would just undo our work, right?
We found one Technical Sergeant logged in who had run a command to delete every single user account in the directory tree. We sought him out and he claimed he was trying to remove one individual, but accidentally selected the tree instead of the individual. It just so happened to be the base's tree, not an individual office or squadron.
As his rank implies, he's supposed to be the technical expert in his field. But this guy was an idiot who shouldn't have been touching user accounts in the first place. Managing user accounts in an Airman job; a simple job given to our lowest-ranking members as they're learning how to be sysadmins. And he couldn't even do that.
It was a very large base. It took 3 days to recover all accounts from backup. The Technical Sergeant had his admin privileges revoked and spent the rest of his deployment sitting in a corner, doing administrative paperwork.
BEGIN TRAN
ROLLBACK TRAN
This. My comment was going to be “what kind of maniac uses auto commit?”
I always put the where clause first since a fuck up in my early 20s lost a loans company £40k of business.
My trick is writing it as a
SELECTstatement first, making sure it's returning the right number of records, and then switching out theSELECTforDELETE. Hasn't steered me wrong yet.This.
The hero we don't deserve.
I worked for a company where the testing database was also the only backup.