41
submitted 5 months ago* (last edited 5 months ago) by TankieTanuki@hexbear.net to c/technology@hexbear.net

Your answer could potentially help me un-fuck my website!

Details here.


Edit: The total capacity is currently 160 GB, so no amount of pruning will free up 1 TB. I need to mount an external share or something.


Edit2: I'm trying this but the command isn't working despite zero exit status.

top 34 comments
sorted by: hot top controversial new old
[-] CyberSyndicalist@hexbear.net 10 points 5 months ago

Do you have your own physical server or are renting one?

[-] TankieTanuki@hexbear.net 9 points 5 months ago

Renting a VPS in the Netherlands.

[-] fox@hexbear.net 7 points 5 months ago

You may be able to just buy more capacity from the vendor by writing to their support. It's all containers

[-] TankieTanuki@hexbear.net 6 points 5 months ago* (last edited 5 months ago)

Storage isn't one of the things the sell on their business page but yeah, maybe their team can rig up something for a fee. It's worth a shot.

[-] Awoo@hexbear.net 6 points 5 months ago* (last edited 5 months ago)

Some of these companies can be really good, others can be like "no". InmotionHosting support were the primary reason I used them despite being slightly more expensive because their support would literally help with any issue even ones no other company would, and they would do stuff like give free storage for a problem like this. At one point I was using like 21TB of space on their service and really testing the limitations of "unlimited storage" they offered on their plan and they were totally cool with it even though I was clearly taking the piss a little. The only thing they held me to was that all things being stored on it had to be frontend accessible.

[-] TankieTanuki@hexbear.net 3 points 5 months ago

frontend accessible

What does that mean, not encrypted at rest?

[-] Awoo@hexbear.net 2 points 5 months ago* (last edited 5 months ago)

No like physically used as part of the website. Not just sitting on the server that the website was on without actually being in use on the website itself.

At the time what we were doing was archiving all pictures and video from a very very large and active discord. These were then searchable, you could see who posted what pictures and video, in what order historically, even sort by total number of reacts.

I don't like the fact that discords buries all media for guilds and communities behind a god awful chat. So many screenshots and videos and all kinds of media are just buried in discord, extracting JUST the media and making it viewable and searchable by user etc preserves genuinely emotionally sentimental content. Things like that time a group of friends defeated their first dungeon together and stuff like that deserves to not get forgotten about never to be found ever again somewhere in the depths of a discord server.

Also I am a hoarder.

[-] PaX@hexbear.net 4 points 5 months ago

Idk if you pay for bandwidth or anything but if you do that might be cheaper than exchanging 1 TB of traffic or more over their network agony-deep

[-] TankieTanuki@hexbear.net 3 points 5 months ago* (last edited 5 months ago)

Bandwidth in and out of my VPS is free. Object storage ingress and egress is what will cost me.

[-] Tabitha@hexbear.net 3 points 5 months ago

IDK I'm drunk right now but maybe they meant "rent a 2TB VM in the same datacenter for a few days"

[-] TankieTanuki@hexbear.net 3 points 5 months ago* (last edited 5 months ago)

Unfortunately the storage on their largest VM is only 320 GB.

[-] PaX@hexbear.net 5 points 5 months ago* (last edited 5 months ago)

I have been in similar situations before and there is no easy answer agony-wholesome

I have no idea how Peertube works but if you're on Linux you might be able to turn on filesystem compression.... but if the data you're storing is already compressed (like video) that probably won't help very much

You could try sshfs like you mentioned in your post but it might be unreliable :( , maybe you could try NFS or even mount remote storage over Plan 9 protocol

[-] TankieTanuki@hexbear.net 5 points 5 months ago* (last edited 5 months ago)

To mount an sshfs share I think I need to shell into my PC from the webserver (not the other way around) and I can't do that.

[-] PaX@hexbear.net 5 points 5 months ago

Unlimited destruction upon NAT qin-shi-huangdi-fireball

Do both hosts have internet connection via IPv6? You could try this over IPv6, usually no NAT, but your ISP might do some kind of firewalling

You could also probably put up a Wireguard (VPN) connection between them since the webserver is not behind NAT

Or also "forward a port" if you have access to the NATing router between your PC and webserver

[-] TankieTanuki@hexbear.net 5 points 5 months ago

I might be able to do IPv6 but I would need to drop my VPN (IPv4 only).

I know very little about Wireguard.

[-] PaX@hexbear.net 5 points 5 months ago

Sorry, had to log off for a bit

It seems you need some kind of NAT traversal mechanism. Wireguard is pretty easy to set up, it's built-into the Linux kernel these days. I've heard good things about Tailscale as well.

Sorry I wish I could help more :(

[-] JoeByeThen@hexbear.net 4 points 5 months ago

I just setup tailscale on my home Ubuntu jellyfin server the other day for the first time and it was a totally painless experience. Been working great.🤞

[-] Tabitha@hexbear.net 3 points 5 months ago

I could totally see tailscale working in this scenario lol

[-] What_Religion_R_They@hexbear.net 5 points 5 months ago

regarding your cgnat idea you can try to work with an ssh tunnel between your PC and the VPS to bridge them and do what you were planning to by mounting

[-] Findom_DeLuise@hexbear.net 3 points 5 months ago* (last edited 5 months ago)

You probably have some archival shit under /var/log that you can prune. I will usually go level-by-level running du -sh {PARENT_PATH}/* to narrow things down one level at a time until I find the offender. Check under /opt and user home dirs, too. Otherwise, you're probably stuck adding another disk or pushing data out to an S3 bucket.

[-] TankieTanuki@hexbear.net 3 points 5 months ago

Can I make a BackBlaze bucket and mount that on the filesystem?

[-] Findom_DeLuise@hexbear.net 3 points 5 months ago

Depends on if you can free up enough space on / to install any packages you may need for the filesystem drivers. I've never had to do it either way; if you have a local rig that you can test with, that might be a start.

[-] What_Religion_R_They@hexbear.net 3 points 5 months ago

I don't know what peertube can do but can you move stuff to /dev/null as a storage device?

[-] comrade_pibb@hexbear.net 3 points 5 months ago

There's a cloud option if you don't want to self host: https://devnull-as-a-service.com/

[-] TankieTanuki@hexbear.net 4 points 5 months ago

lol reminds me of the odd/even API.

[-] comrade_pibb@hexbear.net 3 points 5 months ago

npm and its consequences etc etc

[-] TankieTanuki@hexbear.net 1 points 5 months ago

That's a neat idea, but when peertube goes to put the files back into remote storage it would fail to find them locally.

[-] farting_weedman@hexbear.net 2 points 5 months ago

Stop the service that’s doing the thing causing you a problem.

Fix the thing you screwed up

Restart the fixed service that was causing the problem

What could possibly go wrong?

In this case, and I’m making wild assumptions because I don’t have a peertube instance to play with, stop peertube, connect your local system up to it with a vpn to traverse your nat, mount some volume in the local system as the remote systems target for the file system move, then start peertube back up and see if it starts filling your local drive with some data.

[-] TankieTanuki@hexbear.net 1 points 5 months ago* (last edited 5 months ago)

When I made the wrong command, the application registered hundreds of move jobs in its database. The developer said there is no way to cancel the jobs. If I restart peertube.service then it will notice the incomplete jobs and immediately resume them. I think I'm going to use btrfs next time so I can take a system snapshot before executing risky server commands.

connect your local system up to it with a vpn to traverse your nat

I need a little more detail on this. Here's my takeaway:

  1. Host an (e.g.) OpenVPN server on the webserver, after creating the proper certificate infrastructure.
  2. Connect as client from home PC.
  3. VPN Success. Now I can access each other on the local subnet that starts with a 10. or something.
  4. Mount home PC's NFS share on the webserver.
[-] farting_weedman@hexbear.net 3 points 5 months ago

Yeah that’s what I was thinking. Standing up openvpn especially is pretty well documented.

[-] farting_weedman@hexbear.net 2 points 5 months ago

Alternate solution: use atop sorting by fs access and kill the process doing it.

[-] Sickos@hexbear.net 1 points 5 months ago

So if I'm reading this right, the issue is going to be that when you revive peertube, it's going to resume trying to move stuff? Is it running in a container or directly on that machine?

[-] Sickos@hexbear.net 1 points 5 months ago* (last edited 5 months ago)

If I'm understanding this correctly, and keep in mind I know nothing of peertube, I'm just an old coder tearing through their source, you could break your config. Configure it not to use object storage and restart. If can't find the storage it's trying to move stuff from, it can't move that stuff.

OBJECT_STORAGE: { ENABLED: config.get<boolean>('object_storage.enabled'),

[-] Sickos@hexbear.net 1 points 5 months ago

Or make the target folder/file read only

this post was submitted on 15 Jun 2024
41 points (100.0% liked)

technology

23313 readers
123 users here now

On the road to fully automated luxury gay space communism.

Spreading Linux propaganda since 2020

Rules:

founded 4 years ago
MODERATORS