38

Hello everyone,

Recently I have returned to managing a kubernetes cluster in my homelab with Ansible on RHEL distros. Since I haven't touched to the installation stages since quite a long time I started to look for tutorials from the base installation to the cni configuration, MetalLB setup and metrics server installation.

In every single tutorial, I have seen major issues that made me pull my hair:

  • First and the worst, most tutorials obviously have the firewall disabled or tells you to deactivate it. Just. No. I know deactivating it makes everything much easier and many issues disappear as soon as you run a systemctl stop firewalld. But if you want to teach correcty, you wouldn't recommend something that would make you fired on the spot.

  • CNI installations are straight forward but miss important information for troubleshooting. Stuff like putting flannel interfaces in the internal zone or adding some direct forwarding rules to firewalld can be necessary but again, everyone and their mothers have their firewall off so they never talk about it.

  • In MetalLB, the configMap used by the speakers is not created automatically by the official manifest. Missing it is impossible as the speaker straight up do not start and the logs are straightforward. Yet I have never seen one tutorial mention it.

  • Again in metalLB, if the controller is on a worker node, webhooks are not accessible and you cannot configure the load balancer. It's rare-ish and easy to fix but again, never seen any mention of that

  • While Flannel, MetalLB, Weave, ... clearly state which ports you need to open for their solutions, tutorials never do (firewall? Someone?)

  • The metrics server has some ... Particularities (like the need to modify the startup arguments or the dnsPolicy). Those are easily found in the github issues due to how frequent they're but I can never seem to find a tutorial mentionning those extra configuration to do.

  • Various basic stuff like a worker node + a cni being needed for coreDNS and the master node to become ready. Or how to verify your deployment of ingress/cni/metalLB is working correctly. If you are familiar with Kubernetes, it's not too hard to find the solution to those but when most of your audience, it should be explicit to at least share a random nginx manifest to test if everything is good.

This is mainly a rant because it is crazy to see that a tutorial that is supposed to explain the documentation but faster is utterly useless because of course, you won't get any forwarding issues between interfaces if your device is an open bar.

And that most of them are like this.

So to everyone who also tried to follow tutorials for the set up of their clusterw what was your experience with them? Were they also useless or did you find a gem that didn't simply copy pasted the documentation and took screenshots of an working cluster setup without trying their guide?

you are viewing a single comment's thread
view the rest of the comments
[-] notfromhere@lemmy.one 6 points 1 year ago

You bring up a lot of great points. I disabled the firewall on my bare metal cluster nodes and didn’t give it another thought. I had to go digging to figure out how to encrypt secrets, and NFS StorageClass is not very great security wise either. Not to mention lack of isolation for privileged containers. I found kata containers a good solution to that. Then there’s wireguard between workers I don’t know if I got working correctly because I can’t figure out how to really test it.

this post was submitted on 26 Aug 2023
38 points (91.3% liked)

Selfhosted

40330 readers
353 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS