733

Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.

Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.

Rolled back to the backup before I made it public and now I have a security checklist.

top 50 comments
sorted by: hot top controversial new old
[-] punkwalrus@lemmy.world 164 points 1 week ago

Basic setup for me is scripted on a new system. In regards to ssh, I make sure:

  • Root account is disabled, sudo only
  • ssh only by keys
  • sshd blocks all users but a few, via AllowUsers
  • All 'default usernames' are removed, like ec2-user or ubuntu for AWS ec2 systems
  • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn't make it "more secure" but damn, it reduces the script denials in my system logs, fight me.
  • Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.

My systems are not "unhackable" but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.

[-] feddylemmy@lemmy.world 69 points 1 week ago
  • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn't make it "more secure" but damn, it reduces the script denials in my system logs, fight me.

Gosh I get unreasonably frustrated when someone says yeah but that's just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening.. But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.

Also does anyone still port knock these days?

[-] kernelle@0d.gs 19 points 1 week ago

Also does anyone still port knock these days?

Enter Masscan, probably a net negative for the internet, so use with care.

load more comments (2 replies)
load more comments (3 replies)
[-] kibiz0r@midwest.social 74 points 1 week ago

One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.

“Hey, why is this Valheim server running like shit?”

“Wtf is xrx?”

“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”

So anyway, now I use NixOS.

[-] pageflight@lemmy.world 16 points 1 week ago

Good point about a default deny approach to users and ssh, so random services don't add insecure logins.

[-] dadabean@feddit.org 52 points 1 week ago

Interesting. Do you know how it got compromised?

[-] Tablaste@linux.community 73 points 1 week ago* (last edited 1 week ago)

I published it to the internet and the next day, I couldn't ssh into the server anymore with my user account and something was off.

Tried root + password, also failed.

Immediately facepalmed because the password was the generic 8 characters and there was no fail2ban to stop guessing.

[-] lud@lemm.ee 93 points 1 week ago

Don't use passwords for ssh. Use keys and disable password authentication.

[-] Voroxpete@sh.itjust.works 52 points 1 week ago* (last edited 1 week ago)

More importantly, don't open up SSH to public access. Use a VPN connection to the server. This is really easy to do with Netbird, Tailscale, etc. You should only ever be able to connect to SSH privately, never over the public net.

[-] troed@fedia.io 29 points 1 week ago

It's perfectly safe to run SSH on port 22 towards the open Internet with public key authentication only.

load more comments (8 replies)
load more comments (3 replies)
[-] PotatoesFall@discuss.tchncs.de 27 points 1 week ago

wow crazy that this was the default setup. It should really force you to either disable root or set a proper password (or warn you)

[-] jatone@lemmy.dbzer0.com 12 points 1 week ago

Most distributions disable root by default

[-] satans_methpipe@lemmy.world 9 points 1 week ago

Which ones? I'm asking because that isn't true for cent, rocky, arch.

[-] TheEntity@lemmy.world 12 points 1 week ago

Mostly Ubuntu. And... I think it's just Ubuntu.

load more comments (2 replies)
load more comments (4 replies)
load more comments (5 replies)
[-] cm0002@lemmy.world 11 points 1 week ago* (last edited 1 week ago)

because the password was the generic 8 characters and there was no fail2ban to stop guessing

Oof yea that'll do it, your usually fine as long as you hardened enough to at least ward off the script kiddies. The people with actual real skill tend to go after...juicer targets lmao

[-] Tablaste@linux.community 11 points 1 week ago

Haha I'm pretty sure my little server was just part of the "let's test our dumb script to see if it works. Oh wow it did what a moron!"

Lessons learned.

load more comments (13 replies)
[-] mlg@lemmy.world 46 points 1 week ago

Lol you can actually demo a github compromise in real time to an audience.

Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.

[-] Irelephant@lemm.ee 30 points 1 week ago

I search commits for "removed env file" to hopefully catch people who don't know how git works.

[-] raspberriesareyummy@lemmy.world 13 points 1 week ago* (last edited 1 week ago)

--verbose please?

edit: never mind, found it. So there's dumbasses storing sensitive data (keys!) inside their git folder and unable to configure .gitignore...

[-] Irelephant@lemm.ee 12 points 1 week ago

yeah, I just tried it there, people actually did it.

load more comments (2 replies)
[-] spicehoarder@lemm.ee 11 points 1 week ago

You gremlin lmao

[-] electric_nan@lemmy.ml 40 points 1 week ago

Do not allow username/password login for ssh. Force certificate authentication only!

[-] LordCrom@lemmy.world 10 points 1 week ago

If it's public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.

That's just off the top of my head.

There's a lot you can do to harden a host.

load more comments (2 replies)
[-] Hozerkiller@lemmy.ca 39 points 1 week ago

I've gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn't know better, instead of pure salt like another site I won't mention.

[-] Tablaste@linux.community 28 points 1 week ago

I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.

We all make mistakes!

[-] LordCrom@lemmy.world 13 points 1 week ago

There's a 40 year I.T. veteran here that still suffers imposter syndrome. It's a real thing I've never been able to shake off

load more comments (1 replies)
[-] communism@lemmy.ml 38 points 1 week ago

How are people's servers getting compromised? I'm no security expert (I've never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don't do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is password123 then maybe but I'm surprised I've never been pwned if it's so easy to get got...

[-] cmnybo@discuss.tchncs.de 27 points 1 week ago

By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.

load more comments (1 replies)
[-] nsrxn@lemmy.dbzer0.com 13 points 1 week ago

glad my root pass is toor and not something as obvious as password123

load more comments (1 replies)
load more comments (3 replies)
[-] DaCrazyJamez@sh.itjust.works 30 points 1 week ago

As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),

Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP

[-] psivchaz@reddthat.com 26 points 1 week ago

The other poster gave you a lot. If that's too much at once, the really low hanging fruit you want to start with is:

  • Choose an active, secure distro. There's a lot of flavors of Linux out there and they can be fun to try but if you're putting something up publicly it should be running on one that's well maintained and known for security. CentOS and Debian are excellent easy choices for example.

  • Similarly, pick well maintained software with a track record. Nginx and Apache have been around forever and have excellent track records, for example, both for being secure and fixing flaws quickly.

  • If you use Docker, once again keep an eye out for things that are actively maintained. If you decide to use Nginx, there will be five million containers to choose from. DockerHub gives you the tools to make this determination: Download number is a decent proxy for "how many people are using this" and the list of updates tells you how often and how recently it's being updated.

  • Finally, definitely do look at the other poster's notes about SSH. 5 seconds after you put up an SSH server, you'll be getting hit with rogue login attempts.

  • Definitely get a password manager, and it's not just one password per server but one password per service. Your login password to the computer is different from your login to any other things your server is running.

The rest requires research, but these steps will protect you from the most common threats pretty effectively. The world is full of bots poking at every service they can find, so keeping them out is crucial. You won't be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some, so try not to make too many enemies.

load more comments (1 replies)
[-] horse_battery_staple@lemmy.world 19 points 1 week ago* (last edited 1 week ago)

Paranoid external security. I'm assuming you already have a domain name. I'm also assuming you have some ICANN anonymization setup.

This is your local reverse Proxy. You can manage all this with a container called nginx proxy manager, but it could benefit you to know it's inner workings first. https://www.howtogeek.com/devops/what-is-a-reverse-proxy-and-how-does-it-work/

https://cloud9sc.com/nginx-proxy-manager-hardening/

https://github.com/NginxProxyManager/nginx-proxy-manager

Next you'll want to proxy your IP address as you don't want that pointing to your home address

https://developers.cloudflare.com/learning-paths/get-started-free/onboarding/proxy-dns-records/

Remote access is next. I would suggest setting up wireguard on a machine that's not your webserver, but you can also set that up in a container as well. Either way you'll need to punch another hole in your router to point to your wire guard bastion host on your local network. It has many clients for windows and linux and android and IOS

https://github.com/angristan/wireguard-install

https://www.wireguard.com/quickstart/

https://github.com/linuxserver/docker-wireguard

Now internally, I'm assuming you're using Linux. In that case I'd suggest securing your ssh on all machines that you log into. On the machines you're running you should also install fail2ban, UFW, git, and some monitoring if you have the overhead but the monitoring part is outside of the purview of this comment. If you're using UFW your very first command should be sudo ufw allow ssh

https://www.howtogeek.com/443156/the-best-ways-to-secure-your-ssh-server/

https://github.com/fail2ban/fail2ban

https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands

Now for securing internal linux harden the kernel and remove root user. If you do this you should have a password manager setup. keepassx or bitwarden are ones I like. If those suck I'm sure someone will suggest something better. The password manager will have the root password for all of your Linux machines and they should be different passwords.

https://www.makeuseof.com/ways-improve-linux-user-account-security/

https://bitwarden.com/help/self-host-an-organization/

https://keepass.info/

Finally you can harden the kernel

https://codezup.com/the-ultimate-guide-to-hardening-your-linux-system-with-kernel-parameters/

TLDR: it takes research but a good place to start is here

https://www.digitalocean.com/community/tutorials/recommended-security-measures-to-protect-your-servers

[-] psivchaz@reddthat.com 9 points 1 week ago

Correct, horse battery staple.

load more comments (1 replies)
[-] ptz@dubvee.org 25 points 1 week ago* (last edited 1 week ago)

And this is why every time a developer asks me for shell access to any of the deployment servers, I flat out deny the request.

Good on you for learning from your mistakes, but a perfect example for why I only let sysadmins into the systems.

[-] Tablaste@linux.community 11 points 1 week ago

You're not wrong! Devops made me lazy

load more comments (2 replies)
load more comments (1 replies)
[-] recklessengagement@lemmy.world 24 points 1 week ago

This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted

[-] ramius345@sh.itjust.works 24 points 1 week ago* (last edited 1 week ago)

You should turn off ssh password logins on external facing servers at a minimum. Only use ssh keys, install fail2ban, disable ssh root logins, and make sure you have a firewall limiting ports to ssh and https.

This will catch most scripted login attempts.

If you want something more advanced, look into https://en.m.wikipedia.org/wiki/Security_Technical_Implementation_Guide and try to find an ansible playbook to apply them.

load more comments (5 replies)
[-] nonentity@sh.itjust.works 21 points 1 week ago

Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.

The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.

load more comments (6 replies)
[-] ohshit604@sh.itjust.works 21 points 1 week ago* (last edited 1 week ago)

I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.

load more comments (5 replies)
[-] sommerset@thelemmy.club 20 points 1 week ago

I'm confused. I never disable root user and never got hacked.

Is the issue that the app is coded in a shitty way maybe ?

[-] Xanza@lemm.ee 12 points 1 week ago

You can't really disable the root user. You can make it so they can't login remotely, which is highly suggested.

load more comments (4 replies)
load more comments (1 replies)
[-] phx@lemmy.ca 20 points 1 week ago

Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.

When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.

load more comments (2 replies)
[-] DavidGA@lemmy.world 14 points 1 week ago

Although disabling the root user is a good part of security, leaving it enabled should not alone cause you to get compromised. If it did, you were either running a very old version of OpenSSH with a known flaw, or, your chosen root password was very simple.

load more comments (5 replies)
[-] otacon239@lemmy.world 11 points 1 week ago

I’ve always felt that if you’re exposing an SSH or any kind of management port to the internet, you can avoid a lot of issues with a VPN. I’ve always setup a VPN. It prevents having to open up very much at all and then you can open configured web portal ports and the occasional front end protocol where needed.

[-] possiblylinux127@lemmy.zip 8 points 1 week ago

I like to spin up a public facing server and run tcpdump

load more comments (3 replies)
load more comments
view more: next ›
this post was submitted on 10 Feb 2025
733 points (99.3% liked)

linuxmemes

22622 readers
888 users here now

Hint: :q!


Sister communities:


Community rules (click to expand)

1. Follow the site-wide rules

2. Be civil
  • Understand the difference between a joke and an insult.
  • Do not harrass or attack users for any reason. This includes using blanket terms, like "every user of thing".
  • Don't get baited into back-and-forth insults. We are not animals.
  • Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
  • Bigotry will not be tolerated.
  • These rules are somewhat loosened when the subject is a public figure. Still, do not attack their person or incite harrassment.
  • 3. Post Linux-related content
  • Including Unix and BSD.
  • Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of sudo in Windows.
  • No porn. Even if you watch it on a Linux machine.
  • 4. No recent reposts
  • Everybody uses Arch btw, can't quit Vim, <loves/tolerates/hates> systemd, and wants to interject for a moment. You can stop now.
  • 5. 🇬🇧 Language/язык/Sprache
  • This is primarily an English-speaking community. 🇬🇧🇦🇺🇺🇸
  • Comments written in other languages are allowed.
  • The substance of a post should be comprehensible for people who only speak English.
  • Titles and post bodies written in other languages will be allowed, but only as long as the above rule is observed.
  •  

    Please report posts and comments that break these rules!


    Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't remove France.

    founded 2 years ago
    MODERATORS