62
CPU load over 70 means I can't even ssh into my server
(lemmy.world)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 29 Aug 2023
62 points (95.6% liked)
Selfhosted
40198 readers
382 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
It's sounds like it could be an IO wait issue, system load will climb a ton without showing much CPU usage.
Make sure you're not running out of RAM and going into swap space, it doesn't sound like it though.
iotopmight show something useful. And inhtopyou can add the 'PERCENT_IO_DELAY" column which can be useful.My money is also on IO. Outside of CPU and RAM, it's the most likely resource to get saturated (especially if using rotational magnetic disks rather than an SSD, magnetic disks are going to be the performance limiter by a lot for many workloads), and also the one that OP said nothing about, suggesting it's a blind spot for them.
In addition to the excellent command-line approaches suggested above, I recommend installing netdata on the box as it will show you a very comprehensive set of performance metrics without having to learn to collect each one on the CLI. A downside is that it will use RAM proportional to the data retention period, which if you're swapping hard will be an issue. But even a few hours of data can be very useful and with 16gb of ram I feel like any swapping is likely to be a gross misconfiguration rather than true memory demand... and once that's sorted dedicating a gig or two to observability will be a good investment.
And I know OP mentioned not using much ram, but almost everytime I see a server load that high, it's usually because the server is swapping heavily causing the iowait.
Yeah I figured I would mention it since OP does describe symptoms like that.
Does top show unpaged memory too? I've had an application with a memory leak before that would fill up unpaged memory and it would look like nothing was using ram when I looked in the task manager, even though usage was 99%.
Yep. IO.
OP, this might be overkill for you but it might be worth standing up a grafana/prometheus stack.. You'd be able to see this stuff a lot faster and potentially narrow in on a root cause.
That is definitely an interesting idea! Much, much better than the stupid
dashdotcontainer I am running now :-D