265

Just started self hosting this instance. Nothing on the docs mentioned anything about storage considerations.

top 50 comments
sorted by: hot top controversial new old
[-] ruud@lemmy.world 271 points 1 year ago

This is lemmy.world after 4 weeks:

58G	pictrs
34G	postgres

Considering this is going to be around a 5 user instance at most I think I'll be good for awhile. Thanks!

[-] manitcor@lemmy.intai.tech 57 points 1 year ago* (last edited 1 year ago)

im running 50 users right now, subbed to A LOT of communities, seeing db growth of about 100mb per day.

[-] Pleonasm@programming.dev 19 points 1 year ago

That seems high when you extrapolate that to 10000 users, like a larger instance might have.

[-] terribleplan@lemmy.nrd.li 69 points 1 year ago* (last edited 1 year ago)

It's all about how many communities your user(s) subscribe to since your instance basically acts as a mirror for those.

My instance has been running for 23 days, and I am pretty much the only active local user:

7.3G    pictrs
5.3G    postgres

edit: I may have a slight ~~Reddit~~ Lemmy problem

[-] Pleonasm@programming.dev 9 points 1 year ago

So if you're the only user (let's assume for ease) then, that represents all the updates (posts, comments, votes) from each community that you are subscribed to?

[-] terribleplan@lemmy.nrd.li 16 points 1 year ago

Yeah, and I purposely subscribe to (or sometimes have a dedicated "federation helper bot" account I run subscribe to) most of the most popular communities on the most popular instances so I can get a decent sampling of what's going on in the fediverse on the "All" feed. So I assume my storage usage is maybe a bit higher than what an "average" single-user instance may be...

[-] bdonvr@thelemmy.club 4 points 1 year ago

lmao same here. I have a spare account that I use to sub to everything worth subbing to. I haven't automated it yet though.

load more comments (2 replies)
load more comments (3 replies)
[-] Stubborn9867@lemmy.jnks.xyz 22 points 1 year ago

It won't scale linearly. A lot of those users will be subscribed to subs the instance is already replicating. It would only be new subs that would add to the growth.

[-] bdonvr@thelemmy.club 4 points 1 year ago

And only active subs. And even then, it's just text and tiny thumbnails.

[-] Dran_Arcana@lemmy.world 16 points 1 year ago

Question if you know: does a lemmy instance have to be publically accessable to work? Like, if I make an instance on my homelab can the instance "fetch" content and serve it faster locally? Could I reply to a post and have others see it? Etc

load more comments (1 replies)
[-] HappyHam@lemm.ee 18 points 1 year ago

Now I wonder how viable it would be to support video hosting. The answer is almost certainly "God no!"

load more comments (1 replies)
[-] Grimr0c@lemmy.world 17 points 1 year ago

Honestly, Less than I thought!

[-] mruczek@lemm.ee 15 points 1 year ago

Interesting, I thought it would be waaayyy more

[-] BigWigglyStyle@lemmynsfw.com 17 points 1 year ago

At the end of the day the vast majority of what needs to be saved is text. If media content is embedded, the the server just has to save the path to the file not the file itself.

[-] bdonvr@thelemmy.club 11 points 1 year ago* (last edited 1 year ago)

Yeah lemmy seems to use just about nothing for data storage.

[-] myersguy@lemmy.simpl.website 11 points 1 year ago

Wow, that is surprisingly not bad given the size of the instance!

[-] lightrush@lemmy.ca 5 points 1 year ago* (last edited 1 year ago)

Feels like this will benefit from some sort of fuzzy deduplication in the pictrs storage. I bet there are a lot of similar pics in there. E.g. if one pic or a gif is very similar to another, say just different quality or size, or compression, it should keep only one copy. It might already do this for the same files uploaded by different people as those can be compared trivially via hashing, but I doubt it does similarity based deduplication.

[-] NettoHikari@social.fossware.space 63 points 1 year ago

This is my small instance with way fewer users than lemmy.world.

11G	pictrs
5.2G	postgres
[-] Molecular0079@lemmy.world 9 points 1 year ago

Out of curiosity, how long has your instance been up? Just want to get a sense of how fast storage is increasing for you.

[-] Kushan@lemmy.world 7 points 1 year ago

How has your Lemmy experience been on a self hosted instance? I'm currently using lemmy.world and it's very error prone, would self hosting reduce those errors at the expense of anything? Does federation take long or do you find you're getting federated content quickly enough?

[-] NettoHikari@social.fossware.space 12 points 1 year ago

The experience has been pretty good, to be honest. No instability, easy updates, etc. I find federated content quite quickly, because I use this script to populate the "All" feed.

[-] Kushan@lemmy.world 5 points 1 year ago

I didn't make it! :) I think, @fmstrat@nowsci.com made it.

[-] RedWizard@lemmygrad.ml 4 points 1 year ago

Oh this is very cool

[-] bdonvr@thelemmy.club 7 points 1 year ago

You won't get any old content, so that's a downside. You'll only get content after you start federating. Unless someone votes or comments on old content.

Other than that the only downside is spending time maintaining and updating it.

[-] russjr08@outpost.zeuslink.net 29 points 1 year ago

My instance has 13 users, and has been up for 2 months now:

1.5G    ./pictrs
3.4G    ./postgres
[-] A10@kerala.party 26 points 1 year ago

Is there any way to purge old data?

[-] holycrap@lemmy.world 35 points 1 year ago

I really hope it doesn't get purged if lemmy is to be a Reddit replacement. A lot of the value Reddit had was obscure knowledge and making google searches actually usable.

[-] Molecular0079@lemmy.world 22 points 1 year ago

I think as long as the original community the post is in doesn't purge the data, it's fine for other instances to purge if necessary.

[-] punio7@lemmy.world 3 points 1 year ago

Exactly, when dealing with big data, you need a strategy to archive old data. You can't just store everything in one DB. Smaller instances may not feel like keeping all the date from all the time. Even big instances should have a mechanism to move old data do different databases.

[-] HKayn@dormi.zone 4 points 1 year ago

Are you planning on donating to instances that don't purge old data?

476M ./postgres 1.1G ./pictrs

After 3 weeks

load more comments (1 replies)
[-] tugg@lemmyverse.org 16 points 1 year ago

Small instance with about 3 users and myself online for about 2 weeks.

pictrs   930M
postgres 1.4G
[-] key@lemmy.keychat.org 15 points 1 year ago* (last edited 1 year ago)

Depends. If you have a lot of users posting a lot of pictures and you use pictrs out of the box config, then a lot. If you are just running a few users with finite communities being synced then a lot less. The number is going to vary a lot as lemmy grows and gets older so hard to document realistic expectations. But docker images are probably going to take up more disk space than actual contents unless you get quite big. I just threw my PG volume into a tgz to move servers and it's less than a gig.

[-] bdonvr@thelemmy.club 6 points 1 year ago* (last edited 1 year ago)

The lemmy.world admin said above that their instance currently takes up less than 100GB

[-] holycrap@lemmy.world 10 points 1 year ago

Unless they changed all of the comment and post ids to bigints that'll probably bring the site down before it runs out of storage. In defense of the lemmy developers they have been receptive to feedback, so I don't think it'll take long for that to be fixed if it hasn't already.

[-] hitagi@ani.social 9 points 1 year ago

My instance eats up almost 100MB everyday. It mostly depends on what your users subscribe to. It was barely growing on my first few days until I invited a couple of friends over to try it out.

lol lemmy died almost immediately after i posted this time to figure out what the hell caused that

it was because i set a damn server icon

[-] b3nsn0w@pricefield.org 4 points 1 year ago

lmao just how powerful is your server icon?

load more comments (2 replies)
[-] HKayn@dormi.zone 7 points 1 year ago* (last edited 1 year ago)

My instance dormi.zone has been running for around 3½ weeks now, has a 3-digit amount of users and hosts a community with little more than 1000 subscribers. Here's how much storage it currently takes up:

  • 6.2 GiB postgres
  • 4.9 GiB pictrs

In the default Ansible configuration, storage will mostly be accumulated by log files that are automatically generated by Docker and deleted whenever you restart the Docker containers.

[-] lemmy@endlesstalk.org 7 points 1 year ago

After hosting my own instance with just me for ca. 2 weeks:

1.99Gi pictrs

5.21Gi postgres

[-] lightrush@lemmy.ca 5 points 1 year ago

How many cans-of-beans.jpg can you store?

load more comments (1 replies)
load more comments
view more: next ›
this post was submitted on 03 Jul 2023
265 points (98.2% liked)

Selfhosted

40383 readers
363 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS