61
submitted 10 months ago* (last edited 10 months ago) by Edgarallenpwn@midwest.social to c/selfhosted@lemmy.world

I am currently running most of my stuff from an unraid box using spare parts I have. It seems like I am hitting my limit on it and just want to turn it into a NAS. Micro PCs/USFF are what I am planning on moving stuff to (probably a cluster of 2 for now but might expand later.). Just a few quick questions:

  1. Running arr services on a proxmox cluster to download to a device on the same network. I don't think there would be any problems but wanted to see what changes need to be done.

  2. Which micro PCs are you running? I am leaving towards HP prodesk or Lenovo 7xx/9xx series around 200 each. I don't really plan on getting more than 2-3 and don't run too many things, but would want enough overhead if I switch stuff over to home assistant and windows and Linux VMs if needed.

  3. Any best practices you recommend when starting a Proxmox cluster? I've learned over time it's best to set it up correctly than try to fix stuff when it's running. I wish I could coach myself from 7 years ago now. Would of saved a lot of headaches lol.

top 50 comments
sorted by: hot top controversial new old
[-] eerongal@ttrpg.network 7 points 10 months ago

Running arr services on a proxmox cluster to download to a device on the same network. I don’t think there would be any problems but wanted to see what changes need to be done.

I'm essentially doing this with my set up. I have a box running proxmox and a separate networked nas device. There aren't really any changes, per se, other than pointing the *arr installs at the correct mounts. One thing to make note of, i would make sure that your download, processing, and final locations are all within the same mount point, so that you can take advantage of atomic moves.

[-] archomrade@midwest.social 2 points 10 months ago

I second this. It took me a really long time how to properly mount network storage on proxmox VM's/LXC's, so just be prepared and determine the configuration ahead of time. Unprivilaged LXC's have differen't root user mappings, and you can't mount an SMB directly into a container (someone correct me if i'm wrong here), so if you go that route you will need to fuss a bit with user maps.

I personally have a VM running with docker for the arr suite and a separate LXC's for my sambashare and streaming services. It's easy to coordinate mount points with the compose.yml files, but still tricky getting the network storage mounted for read/write within the docker containers and LXC's.

[-] phanto@lemmy.ca 6 points 10 months ago

Do two NICs. I have a bigger setup, and it's all running on one LAN, and it is starting to run into problems. Changing to a two network setup from the outset probably would have saved me a lot of grief.

[-] possiblylinux127@lemmy.zip 2 points 10 months ago

Can you explain what benefit that would bring?

[-] Edgarallenpwn@midwest.social 1 points 10 months ago

So dual NIC on each device and set up another lan on my router? Sorry it seems like a dumb question but just want to make sure.

[-] fuckwit_mcbumcrumble@lemmy.world 1 points 10 months ago

Why would you need two nics unless you’re planning on having a proxmox Vm being your router?

[-] atzanteol@sh.itjust.works 3 points 10 months ago

I haven't done it - but I believe Proxmox allows for creating a "backplane" network which the servers can use to talk directly to each other. This would be used for ceph and server migrations so that the large amount of network traffic doesn't interfere with other traffic being used by the VMs and the rest of your network.

You'd just need a second NIC and a switch to create the second network, then staticly assign IPs. This network wouldn't route anywhere else.

[-] fuckwit_mcbumcrumble@lemmy.world 1 points 10 months ago

In proxmox there's no need to assign it to a physical NIC. If you want a virtual network that goes as frast as possible you'd create a bridge or whatever and assign it to nothing. If you assign it to a NIC then since it wants to use SR-IOV it would only go as fast as the NIC can go.

[-] stown@sedd.it 1 points 10 months ago

Security. Keeping publicly accessible and locally accessible on different networks.

[-] DeltaTangoLima@reddrefuge.com 2 points 10 months ago* (last edited 10 months ago)

Hmmm - not really any more. I have everything on the same VLAN, with publicly accessible services sitting behind nginx reverse proxy (using Authelia and 2FA).

The real separation I have is the separate physical interface I use for WAN connectivity to my virtualised firewall/router - OPNsense. But I could also easily achieve that with VLANs on my switch, if I only had a single interface.

The days of physical DMZs are almost gone - virtualisation has mostly superseded them. Not saying they're not still a good idea, just less of an explicit requirement nowadays.

[-] FiduciaryOne@lemmy.world 1 points 10 months ago

I think two NICs is required to do VLANing properly? Not 100% sure.

[-] DeltaTangoLima@reddrefuge.com 4 points 10 months ago* (last edited 10 months ago)

Nope - Proxmox lets you create VLAN trunks, just like a physical switch.

Edit: here's one of my Proxmox server network configs.

[-] FiduciaryOne@lemmy.world 2 points 10 months ago

Huh, cool, thank you! I'm going to have to look into that. I'd love for some of my containers and VMs to be on a different VLAN from others. I appreciate the correction. 😊

[-] DeltaTangoLima@reddrefuge.com 2 points 10 months ago

No worries mate. Sing out if you get stuck - happy to provide more details about my setup if you think it'll help.

[-] FiduciaryOne@lemmy.world 2 points 10 months ago

Thanks for the kind offer! I won't get to this for a while, but I may take you up on it if I get stuck.

[-] monkinto@lemmy.world 2 points 10 months ago

Is there a reason to do this over just giving the nic for the vm/container a vlan tag?

[-] DeltaTangoLima@reddrefuge.com 2 points 10 months ago

You still need to do that, but you need the Linux bridge interface to have VLANs defined as well, as the physical switch port that trunks the traffic is going to tag the respective VLANs to/from the Proxmox server and virtual guests.

So, vmbr1 maps to physical interface enp2s0f0. On vmbr1, I have two VLAN interfaces defined - vmbr1.100 (Proxmox guest VLAN) and vmbr1.60 (Phsyical infrastructure VLAN).

My Proxmox server has its own address in vlan60, and my Proxmox guests have addresses (and vlan tag) for vlan100.

The added headfuck (especially at setup) is that I also run an OPNsense VM on Proxmox, and it has its own vlan interfaces defined - essentially virtual interfaces on top of a virtual interface. So, I have:

  • switch trunk port
    • enp2s0f0 (physical)
      • vmbr1 (Linux bridge)
        • vmbr1.60 (Proxmox server interface)
        • vmbr1.100 (Proxmox VLAN interface)
          • virtual guest nic (w/ vlan tag and IP address)
        • vtnet1 (OPNsense "physical" nic, but actually virtual)
          • vtnet1_vlan[xxx] (OPNsense virtual nic per vlan)

All virtual guests default route via OPNsense's IP address in vlan100, which maps to OPNsense virtual interface vtnet1_vlan100.

Like I said, it's a headfuck when you first set it up. Interface-ception.

The only unnecessary bit in my setup is that my Proxmox server also has an IP address in vlan100 (via vmbr1.100). I had it there when I originally thought I'd use Proxmox firewalling as well, to effectively create a zero trust network for my Proxmox cluster. But, for me, that would've been overkill.

[-] Live2day@lemmy.sdf.org 2 points 10 months ago

No, you can do more than 1 VLAN per port. It's called a trunk

load more comments (2 replies)
[-] TCB13@lemmy.world 4 points 10 months ago

It's 2024, avoid Proxmox and safe yourself a LOT of headaches down the line.

You most likely don’t need Proxmox and its pseudo-open-source bullshit. My suggestion is to simply with with Debian 12 + LXD/LXC, it runs VMs and containers very well. Proxmox ships with an old kernel that is so mangled and twisted that they shouldn’t even be calling it a Linux kernel. Also their management daemons and other internal shenanigans will delay your boot and crash your systems under certain circumstances.

What I would suggest you to use use instead is LXD/Incus.

LXD/Incus provides a management and automation layer that really makes things work smoothly - essentially what Proxmox does but properly done. With Incus you can create clusters, download, manage and create OS images, run backups and restores, bootstrap things with cloud-init, move containers and VMs between servers (even live sometimes).

Another big advantage is the fact that it provides a unified experience to deal with both containers and VMs, no need to learn two different tools / APIs as the same commands and options will be used to manage both. Even profiles defining storage, network resources and other policies can be shared and applied across both containers and VMs.

I draw your attention to containers (not docker), LXC containers because for most people full virtualization isn't even required. In a small homelab if you can have containers that behave like full operating systems (minus the kernel) including persistence, VMs might not be required. Either way LXD/Incus will allow for both and you can easily mix and match and use what you require for each use case.

For eg. I virtualize the official HomeAssistant image with LXD because we all know how hard is to get that thing running, however my NAS / Samba shares are just a LXD Debian 12 container with Samba4, Nginx and FileBrowser. Sames goes for torrent client that has its own container. Some other service I've exposed to the internet also runs a full VM for isolation.

Like Proxmox, LXD/Incus isn’t about replacing existing virtualization techniques such as QEMU, KVM and libvirt, it is about augmenting them so they become easier to manage at scale and overall more efficient. I can guarantee you that most people running Proxmox today it today will eventually move to Incus and never look back. It woks way better, true open-source, no bugs, no BS licenses and way less overhead.

Yes, there's a WebUI for LXD as well!

[-] lazynooblet@lazysoci.al 12 points 10 months ago

Can someone explain the benefits of LXD without the opinionated crap?

load more comments (1 replies)
[-] node815@lemmy.world 9 points 10 months ago

Since you didn't include a link to the source for your recommendation:

https://github.com/canonical/lxd

I've been on Proxmox for 6 or so months with very few issues and have found it to work well in my instance, I do appreciate seeing another alternative and learning about it too! I very specifically like Proxmox as it gives me an actual IP on my router's subnet for my machines such as Home Assistant. So instead of the 192.168.122.1 it rolls a nice 192.168.1.X/24 IP which fits my range which makes it easier for me to direct my outside traffic to it. Does this also do this? Based on your screenshots, maybe not, IDK.

[-] jgkawell@lemmy.world 7 points 10 months ago

Thanks for the link! I've been running Proxmox for years now without any of the issues like the previous commenter mentioned. Not that they don't exist, just that I haven't hit them. I really like Proxmox but love hearing about alternatives. One day I might get bored and want to set things up new with a different stack and anything that's more free/open is better in my book.

[-] TCB13@lemmy.world 1 points 10 months ago

it gives me an actual IP on my router’s subnet for my machines

Yes you configure LXD/Incus' networking to use a bridge and it will simply delegate the task to your router instead of proving IPs itself. One of my nodes actually runs the two setups at the same time, I've a bunch of containers on an internal range and then my Home Assistant VM getting an IP from my router.

[-] possiblylinux127@lemmy.zip 5 points 10 months ago* (last edited 10 months ago)

Your comment is wrong in a few ways and suggests using a LXC which is way slower than docker or podman and lacks the easy setup.

Proxmox is good because it makes it easy to create VMs and setup least access. It also has as new of kernel as stable Debian so no, its not terribly out of date.

If you want to suggest that someone install Debian + Docker compose that would make more sense. This isn't a good setup for more advanced setups and it doesn't allow for a not of flexibility.

load more comments (1 replies)
[-] drkt@feddit.dk 3 points 10 months ago

I'm going to experiment with this! I would love to get rid of Proxmox, it has so many problems and I only run containers anyway.

Is there an easy way to migrate containers? I'm not well versed in LXC despite using it for years.

load more comments (4 replies)
[-] MangoPenguin@lemmy.blahaj.zone 3 points 10 months ago

How well does it handle backups, and are they deduplicated incremental ones like proxmox backup server makes?

[-] TCB13@lemmy.world 1 points 10 months ago

I do regular snapshots of my containers live and sometimes restore them, no issues there. De-duplication and incremental features are (mostly) provided by the storage backend, if you use BTRFS or ZFS for your storage pool every container will be a volume that you can snapshot, rollback, export at any time. LXD also provides tools to make those operations: https://documentation.ubuntu.com/lxd/en/latest/howto/instances_backup/

[-] MangoPenguin@lemmy.blahaj.zone 3 points 10 months ago

That makes sense, but no remote backups over the network? Local snapshots I don't really count as backups.

[-] DeltaTangoLima@reddrefuge.com 3 points 10 months ago* (last edited 10 months ago)

I have two Proxmox hosts and two NASes. All are connected at 1Gbps.

The Proxmox hosts maintain the real network mounts - nfs in my case - for the NAS shares. Inside each CT that requires them, these are mapped to mount points with identical paths in each, eg. /storage/nas1 and /storage/nas2.

All my *arr (and downloader) CTs are configured to use the exact same paths.

It's seamless. nzbget or deluge download to the same parent folders that my *arr CTs work with, which means atomic renames/moves are pretty much instant. The only real network traffic is from the download CTs to the NASes.

Edit: my downloader CTs download directly to the NAS paths - no intermediate disk at all.

[-] atzanteol@sh.itjust.works 3 points 10 months ago

Use ZFS when prompted - it opens up some features and is a bitch to change later. I don't understand why it's not the default.

load more comments (3 replies)
[-] Lem453@lemmy.ca 3 points 10 months ago

I have a setup similar to what you want.

My nas is a low powered atom board that runs unraid.

My dockets run on a ryzen CPU with proxmox. I don't have a cluster, just 1.

In proxmox I run a VM that runs a all my dockets.

I use portainer to run all my services as stacks. So the arr stack has all the arrs together in a docker compose file. The docker compose files are stored in gitea (one of the few things I still run on unraid) and Everytime I make a change to the git, I press one button on portainer and it pulls down the latest docker compose.

For storage, on proxmox I use zfs with ssds only. The only thing that needs HDDs is the media on my unraid.

When a docker needs to access the media it uses an NFS mount to the unraid server.

Everything else is on my zfs array on proxmox. I have auto zfs snapshots every hour. Borg backup also takes hourly incremental backups of the zfs array and sends it to the unraid server locally and borg base for off-site backup.

The whole setup works very well and it very stable.

The flexibility of using proxmox means that things that work better in a VM (HaOS) I can install as a VM. Everything else is docker.

[-] possiblylinux127@lemmy.zip 2 points 10 months ago

For your Proxmox cluster shoot for three devices. With three devices you can do high availability which is a bonus but not something I though to do when I built my setup.

[-] SeeJayEmm@lemmy.procrastinati.org 2 points 10 months ago

And you don't have quorum issues any time a system is down. (I regret making mine a cluster.)

[-] nem@sopuli.xyz 1 points 10 months ago

You can set up a qdev on a pi or something.

[-] possiblylinux127@lemmy.zip 2 points 10 months ago

Can you? That would be really cool

[-] nem@sopuli.xyz 1 points 10 months ago

Yeah, you can run it on anything and its great for even numbered clusters.

[-] possiblylinux127@lemmy.zip 1 points 10 months ago

Can you explain how?

load more comments (3 replies)
[-] Decronym@lemmy.decronym.xyz 2 points 10 months ago* (last edited 10 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
Git Popular version control system, primarily for code
HTTP Hypertext Transfer Protocol, the Web
IP Internet Protocol
LXC Linux Containers
NAS Network-Attached Storage
SSD Solid State Drive mass storage
ZFS Solaris/Linux filesystem focusing on data integrity
k8s Kubernetes container management package
nginx Popular HTTP server

8 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.

[Thread #411 for this sub, first seen 8th Jan 2024, 20:15] [FAQ] [Full list] [Contact] [Source code]

[-] tristan@aussie.zone 1 points 10 months ago* (last edited 10 months ago)

My current setup is 3x Lenovo m920q (soon to be 4) all in a proxmox cluster, along with a qnap nas with 20gb ram and 4x 8tb in raid 5.

The specs on the m920q are: I5 8500T 32gb ram 256gb sata SSD 2tb nvme SSD 1gbe nic

On each proxmox machine, I have a docker server in swarm mode and each of those vm all have the same NFS mounts pointing to the nas

On the Nas I have a normal docker installation which runs my databases

On the swarm I have over 60 docker containers, including the arr services, overseerr and two deluge instances

I have no issues with performance or read/write or timeouts.

As one of the other posters said, point all of your arr services to the same mount point as it makes it far easier for the automated stuff to work.

Put all the arr services into a single stack (or at least on a single network), that way you can just point them to the container name rather than IP, for example, in overseerr to tell it where sonarr is, you'd just say http://sonarr:8989 and it will make life much easier

As for proxmox, the biggest thing I'll say from my experience, if you're just starting out, make sure you set it's IP and hostname to what you want right from the start... It's a pain in the ass to change them later. So if you're planning to use vlans or something, set them up first

Pic of my setup

[-] Cooljimy84@lemmy.world 1 points 10 months ago

With arr services try to limit network throughput and disk throughput on them, as if either are maxed out for too long (like moving big linux iso files) it can cause weird timeouts and failures

[-] Edgarallenpwn@midwest.social 1 points 10 months ago

I believe I would be fine on the network part, I am just guessing writing them to an SSD cache drive on my NAS would be fine? Im currently writing to the SSD and have a move script run twice a day to the HDDs

[-] Cooljimy84@lemmy.world 1 points 10 months ago

Should be fine, I'm writing to spinning rust, so if I was playing back a movie it could cause a few "dad the tv is buffering again" problems

load more comments
view more: next ›
this post was submitted on 08 Jan 2024
61 points (96.9% liked)

Selfhosted

40330 readers
263 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS