Edited: this post to be the Lemmy.World federation issue post.

We are now ready to upgrade to postgres 16. When this post is 30 mins old the maintenace will start

Updated & Pinned a Comment in this thread explaining my complete investigation and ideas on how the Lemmy app will & could move forward.

This is terribly hard to write. If you flushed your cache right now you would see all the newest posts without images. These are now 404s, even thought the images exist. In 2 hours everyone will see this. Unfortunately there is no going back, recovering the key store for all the "new" images.

What happened?

After the picture migration from our local file store to our object storage, i made a configuration change so that our Docker container no longer reference the internal file store. This resulted in the picture service having an internal database that was completely empty and started from scratch 😔

What makes this worse is that this was inside the ephemeral container. When the containers are recreated that data is lost. This had happened multiple times over the 2 day period.

What made this harder to debug was our CDN caching was hiding the issues, as we had a long cache time to reduce the load on our server.

The good news is that after you read this post, every picture will be correctly uploaded and added to the internal picture service database! 😊 The "better" news is the all original images from the 28th of June and before will start working again instantly.

Timeframe

The issue existed from the period from 29th of June to 1st of July.

Resolution

Right now. 1st of July 8:48 am UTC.
From now on, everything will work as expected.

Going forward

Our picture service migration has been fraught with issues and I cannot express how annoyed and disheartened by the accidents that have occurred. I am yet to have provided a service that I would be happy with.

I am very sorry that this happened and I will strive to do better! I hope you all can accept this apology

Tiff

Welcome to everyone joining up so far!

Housekeeping & Rules

Reddthat's rules are defined here (& are available from the side bar as well). When in doubt check the sidebar on any community!

Our rules can be summed up with 4 points, but please make sure you do read the rules.

• Remember that we are all humans
• Don’t be overtly aggressive towards anyone
• Try and share ideas, thoughts and criticisms in a constructive way
• Tag any NSFW posts as such

Funding & Future

How we pay for all the services is explained in our main funding and donation post. We are 100% community funded, and completely open about our bills and expenses.
The post may need some updating over the course of the next day or two as we deal with an increased user base.
If you enjoy it here and want to help us keep the lights on and improve our services. Please help us out. Any amount of dollarydoos is welcome!

I'm lost and completely new to the fediverse!

That's fine, we were all there too! Our fediverse friends created a starting guide here which can be read to better understand how everything works.

As part of the fediverse you can follow and block any community you want to as well as follow and interact with any other person, on any other server. That means even if you did not choose Reddthat as your home server, you still could have interacted with us regardless!
So I'd like to say thanks for picking our server to come and experience the fediverse on.

Reddthat Communities:

• Announcements: !reddthat@reddthat.com (Also where you are reading this now!)
• Support: !reddthatsupport@reddthat.com
• Community !community@reddthat.com (for all off-topic chats)

Other Communities

Communities can be viewed by clicking the Communities section at the top (or this link). You will be greeted by 3 options. Subscription, Local, All.
They are pretty self explanatory, but for a quick definition:

• Subscription: Communities that you personally are subscribed too (can exist anywhere on the fediverse, on any server)
• Local: Communities that were created on Reddthat
• All: Every community that Reddthat knows about

All is not every community on the whole fediverse, all is only the ones we know about. Also known as "federated" against.
There is the fediverse browser which attempts to list every community on the fediverse.
Just because a community exists elsewhere, doesn't mean that you have to join it. You can create your own! We welcome you to create any community on Reddthat that your heart desires.

Good luck on your adventures and welcome to Reddthat. Cheers,
Tiff

      2 18:00
     18 19:00
      7 20:00
     20 21:00
     27 22:00
     76 23:00
     90 00:00
     84 01:00
     62 02:00
     47 03:00
     32 04:00
     28 05:00
     44 06:00
     43 07:00
     43 08:00
     38 09:00
      5 10:00
     42 11:00
     53 12:00
     49 13:00
     48 14:00
     50 15:00
     16 16:00
     24 17:00
     62 18:00
     40 19:00
     41 20:00
     20 21:00
     16 22:00
     18 23:00

So for those of you who were refreshing the page and looking at our wonderful maintenance page it took way longer than we planned! A full write up I'll do after I've dealt with a couple time out issues.

Here is a bonus meme.

So? How'd it go...

Exactly how we wanted it to go... except with a HUGE timeframe.
As part of the initial testing with object storage I tested using a backup of our files. I validated that the files were synced, and that our image service could retrieve them while on the object store.

What I did not account for was the latency to backblaze from Australia, how our image service handled migrations, and the response times from backblaze.

• au-east -> us-west is about 150 to 160ms.
• the image service was single threaded
• response times to adding files are around 700ms to 1500ms (inclusive of latency)

We had 43000 files totaling ~15GB of data relating to images. If each response time is 1.5 seconds per image, and we are only operating on one image at a time, yep, that is a best case scenario of 43000 seconds or just under 12 of transfer time at an average of 1s per image.

The total migration took around 19 hours as seen by our pretty transfer graph:

So, not good, but we are okay now?

That was the final migration we will need to do for the foreseeable future. We have enough storage to last over 1 year of current database growth, with the option to purchase more storage on a yearly basis.
I would really like to purchase a dedicated server before that happens and if we continue having more and more amazing people join our monthly donations on our Reddthat open collective, I believe that can happen.

Closing thoughts

I would like to take this opportunity to apologise for this miscalculation of downtime as well as not fully understanding the operational requirements on our usage of object storage.
I may have also been quite vocal on the Lemmy Admin matrix channel regarding the lack of a multi-threaded option for our image service. I hope my sleep deprived ramblings were coherent enough to not rub anyone the wrong way.
A big final thank you to everyone who is still here, posting, commenting and enjoying our little community. Seeing our community thrive gives me great hope for our future.

As always. Cheers,
Tiff

Edit:

Instability and occasional timeouts

There seems to be a memory leak with Lemmy v0.18 and v0.18.1 which some other admins have reported as well and has since been plaguing us. Our server would be completely running fine, and then BAM, we'd be using more memory than available and Lemmy would restart. These would have lasted about 5-15 seconds, and if you saw it would have meant super long page loads, or your mobile client saying "network error".

Temporary Solution: Buy more RAM.
We now have double the amount of memory courtesy of our open collective contributors, and our friendly VPS host.

In the time I have been making this edit I have already seen it survive a memory spike, without crashing. So I'd count that as a win!

Picture Issues

This leaves us with the picture issues. It seems the picture migration had an error. A few of the pictures never made it across or the internal database was corrupted! Unfortunately there is no going back and the images... were lost or in limbo.

If you see something like below make sure you let the community/user know:

Also if you have uploaded a profile picture or background you can check to make sure it is still there! <3 Tiff

Hello Reddthat! It is I, the person at the top of your feed.

First off. Welcome all new users! Thank you for signing up and joining our Reddthat community.

Bot Defence

Starting yesterday after going through 90+ registration applications, I couldn't do it anymore. I felt compelled to give people a great experience with instantly getting let in and kept my phone on me for over 24 hours, checking every notification to see if that was another registration application I needed to quickly click accept on.

I want to quickly say thank you to the people who obviously read all the information and for those that didn't I'm keeping a close eye on you... 😛
I found a better solution to our signup problems.

As we use cloudflare for our CDN I have turned on their security system for the signup page. ~~Now when anyone goes to the signup page, they will be given a challenge that needs to be solved.
That means any bots that cannot pass cloudflare's automated challenge cannot signup.
A win until we get our captcha back working. ~~ Well I did not check the signup process correctly. It doesn't act as I thought it would, so I'll disable it after the migration.

Downtime / Migration to object storage

Today in the fediverse we have successfully confirmed that object storage will be an acceptable path forward but will not operate as initially hoped.
I initially hoped to offload everything via our CDN, but the data still needs to go through our app server. The silver lining is that we can still cache it heavily on our CDN to ensure that the pictures will be served fast as possible for you.

So it may be slightly pricier than we initially planned for when moving to object storage, but in the end we still benefit, functionally and monetarily. The reason is we were not going to be billed for egress (fetching/displaying images), where as now we will be. The fees are very low and still should be covered by our wonderful monthly donators.

We will have about 15-20GB of storage that needs to be moved and unfortunately our image service is incapable of running at the same time the migration is done, which means we need to turn it off while the migration happens. To top it all off we have... 43000+ (and counting) small image files. If you haven't worked with large swarms of small images before, the one that I can tell you is that transferring small images, sucks.

So we can do two things:

1. Turn off everything

• Dedicate all CPU and bandwidth to the migration
• Ensuring continuity and reducing the risk of something going wrong

2. Turn off the picture service

We can run Reddthat without the picture service & uploads while we perform the migration, but the migration will have an impact in server performance.

• This will amount to having any picture we host (that isn't cached) return a 404.
• Any uploads will timeout during that period, and return an error popup.
• Pages will be slightly slower to respond.
• Something else might break 🤷

Because of the risks associated with running only half our services, I've decided to continue with our planned downtime and go with option 1, turning off everything while we perform the migration.

Date: 28th June (UTC)

• Start Time: 0:05
• End Time: 6:00 (Expected)

It will probably take the whole 6 hours. In our testing, it did 150 items in 10 minutes... I will put up a maintenance page and will keep you all updated during that time frame especially if it is going to take longer, but unfortunately it will take however long it takes.

This will be the last announcement until we do the migration.
Cheers,
Tiff

PS. Like what we are doing? Become a contributor on our Open Collective to help finance these changes!

Hello Everyone!

Lets start with the good, we recently just hit over 700 accounts registered! Hello! I hope you are doing well on your fediverse journey!

We have also hit... (15 when I posted this) total individual contributors! Every time I see an email saying we have another contributor it makes me feel all warm and fuzzy inside! The simple fact that together we are making something special here really touches my soul.

If you feel like joining us and keeping the server online and filled to the brim with coffee, you can see our open collective here: Reddthat Open Collective.

0.18 (& Downtime?)

As I have said in my community post (over here!), I wanted to wait for 0.18.1 to come out so I did not have to fight off a wave of bots because there is no longer a captcha, & I didn't want to enable registration applications because that just ruins the whole on-boarding experience in my opinion.

So, where does that leave us?

I say, screw those bots! We are going to use 0.18.0 and we are going to rocket our way into 0.18.1 ASAP. 🚀
and.... it's already deployed! Enjoy the new UI and the lack of annoyances on the Jerboa application! If you are getting any weird UI artifacts, hold Control (or command) and press the refresh button, as it is a browser cache issue.

~~I'm going to keep the signup process the same but monitor it to the point of helicopter parenting. So if we get hit by bots we'll have to turn on the Registration Application which I'm really hoping we won't have to. So anyone out there... lets just be friends okay?~~

Well... it looks like we cannot be friends! Application Registration is now turned on. Sorry everyone. I guess we can't have nice things!

Weren't we going to wait?

Moving to 0.18 was actually forced by me (Tiff) as I upgraded the Lemmy app running on our production server from 0.17.4 to 0.18.0. This updated caused the migrations to be performed against the database. The previous backup I had at the time of the unplanned upgrade was from about an hour before that. So rolling the database back was certainly not a viable option as I didn't want to lose an hour worth of your hard typed comments & posts.

The mistake or "root cause" was caused by an environmental variable that was set in my deploy scripts. I utilise environment variables in our deployments to ensure deployments can be replicated across our "dev" server (my local machine) and our "prod" server (the one you are reading this post on now!).

This has been fixed up and I'm very sorry for pushing the latest version when I said we were going to wait. I am also going to investigate how we can better achieve a rollback for the database migrations if we need to in the future.

Pictures (& Videos)

The reason I was testing our deployments was to fix our pictures service (called pictrs). As I've said before (in a non-announcement way) we are slowly using more and more space on our server to store all your fancy pictures, as well as all the pictures that we federate against. If we want to ensure stability and future expansion we need to migrate our pictures from sitting on the same server to an object storage. The latest version of pictrs now has that capability, and it also has the capability of hosting videos!
Now before you go and start uploading videos there are limits! we decided to limit videos to 400 frames, which is about 15 seconds worth of video. This was due to video file-sizes being huge compared pictures as well as the associated bandwidth that comes with video content sharing. There is a reason there are not hundreds of video sharing sites.

Object Storage Migration

I would like to thank the 5 people who have donated on a monthly recurring basis because without knowing there is a constant income stream using a CDN and Object storage would not be feasible.

Over the next week I will test the migration from our filesystem to a couple object storage hosting companies and ensure they can scale up with us. Backblaze being our first choice.

Maintenance Window

• Date: 28th of June
• Start Time: 00:05 UTC
• End Time: 02:00 UTC
• Expected Downtime: the full 2 hours!

If all goes well with our testing, I plan to perform the migration on the 28th of July around 00:05 UTC. We currently have just under 15GB of images so I expect it to take at maximum 1 hour, with the actual downtime closer to 30-40 minutes, but knowing my luck it will be the whole hour.

Closing

Make sure you follow the !community@reddthat.com for any extra non-official-official posts, or just want to talk about what you've been doing on your weekend!

Something I cannot say enough is thank you all for choosing Reddthat for your fediverse journey!

I hope you had a pleasant weekend, and here's to another great week!
Thanks all!
Tiff

After updating to the new pictrs container, all image uploads are (randomly?) failing.

~~We are looking into it.~~

https://github.com/LemmyNet/lemmy-ansible/pull/90 broke it as they updated the pictrs container from 0.3.1 to 0.4.0-rc.7

The 0.4.0-rc.7 intermittently worked for small images, and even videos! (which is the next change as part of the 0.4.x version that will be coming out.)

I've rolled back to v0.3.3 and its back and working, See the comments for a gif, that gets converted to a mp4, and a jpg, and a png.

Enjoy your weekends everyone!

Hello! It seems you have made it to our donation post.

Thankyou

We created Reddthat for the purposes of creating a community, that can be run by the community. We did not want it to be something that is moderated, administered and funded by 1 or 2 people.

Current Recurring Donators:

Current Total Amazing People:

Background

In one of very first posts titled "Welcome one and all" we talked about what our short term and long term goals are.
In 7 days since starting, we have already federated with over 700 difference instances, have 24 different communities and over 250 users that have contributed over 550 comments. So I think we've definitely achieved our short term goals, and I thought this was going to take closer to 3 months to get these types of numbers!

We would like to first off thank everyone for being here, subscribing to our communities and calling Reddthat home!

Funding

Open Collective is a service which allows us to take donations, be completely transparent when it comes to expenses and our budget, and allows us some form of idea of how we tracking month-to-month.

Servers are not free, databases are growing, images are uploading and the servers are running smoothly.

This post has been edited to only include relevant information about total funds and "future" plans. Because sometimes, we reach the future we were planning for!

Current Plans:

Database

The database service is another story. The database has grown to 1.8GB (on the file system) in the 7 days. Some quick math, makes that 94GB in 1 year's time. Our current host allows us to addon 100GB of SSD storage for $100/year, which is very viable and will allow us to keep costs down, while planning for our future.

Annual Costings:

Our current costs are:

• Domain: 15 Euro
• Server: $118.80 Aud
• Server Ram Upgrade: $54.50 Aud
• Server: $528.00 Aud
• Wasabi Object Storage: $72 Usd
• Total: ~$830 Aud per year (~$70/month)

That's our goal. That is the number we need to achieve in funding to keep us going for another year.

Cheers
Tiff

PS. Thank you to our donators! Your names will forever be remembered by me:

• Guest x6
• Dave_r
• MentallyExhausted
• Stimmed
• pleasestopasking
• RIPSync
• Incognito
• Siliyon
• Ed
• Alexander
• hexed
• muffin

The following rules governs reddthat.com and all communities hosted on reddthat

Ground Rules

We are a NSFW enabled instance, which means there may be NSFW content in the Local and All categories if you leave NSFW enabled in your account. If you do not wish to view NSFW content you are still welcome here but make sure you toggle off Show NSFW content under your Settings

Another option is to select "Block Community" from the sidebar, when on the specific community. Once you have done that you will never see it again!

Our Instance Rules

• General Code of Conduct
• Any NSFW post must be tagged as NSFW. Failure to do so will be given one warning only
  • Anything that you wouldn't want your boss or coworkers to see, needs to be tagged NSFW
  • NSFW also acts as "Content Warning" outside of the specific NSFW communities
• Remember the human! (no harassment, threats, etc.)
• No racism or other discrimination
• No endorsement of hate speech
• No self-advertisements or spam
• No link-spamming
  • Recommended 2 links per community per day unless noted in Community Rules
• No content against Australian Law

Any posts or comments that are in breach of these rules will be dealt with, and remediation will occur. Whether that be a warning, temporary ban, or permanent ban.

(TLDR) The crux of it boils down to:

• Remember that we are all humans
• Don’t be overtly aggressive towards anyone
• Try and share ideas, thoughts and criticisms in a constructive way
• Tag any NSFW posts as such

Moderation

We agree with the Code of Conduct for Moderation and will due our best to uphold it.

Lemmy provides the Moderation Log for everyone to see as well. This is a federated log where actions taken by other moderators and other instance admins are shown. As such the posts that exist in the log come with a content warning

You can see the modlog in the bottom of the sidebar. Here is the link if you wish to see it.

References

As both links above reference a domain outside of our control (join-lemmy). Here is the git commit and the wayback link:

• this git commit.
• web.archive.org link

Feedback

If you take issue with these rules, or think a rule should be modified, please add a comment on this post with answers to the following as well as any extra information you can think of:

• What rule would you like to change?
• Why you would like to change it?
• What are the benefits to this rule change?
• Have there been any issues and can you reference them?

Thank you for your continued support.

• Tiff

Transparency log:

2023-06-13 - Added "Ground Rules" section, Added Feedback section, `Instance Rules` -> `Our Instance Rules`.  
2023-06-11 - Post Creation

Thank you all for letting me wake-up this morning to find even more future friends have joined our little community. At last count it was only 30 users now we have over 80!

Since the last update we've been approved for our OpenCollective initiative. Where you lovely users can ensure that reddthat stays online for as long as the funding allows. All of our funding goals are currently there as well as our stretch goal of paying our moderation and administration team for their hard work.

We've reached another milestone in that we have now federated with over 310 different instances. That means there 310 different other servers out there that we are connecting too and sharing posts with!

Thank you all, and remember the human!

Tiff

So. You used to reddit, now you've reddthat.

Welcome!

Please feel free to explore some of our communities, and make sure you explore other instances communities as well!

We are community funded via OpenCollective & Liberapay currently.

Short-term Goals: (3-6m)

• Gather some like minded people to have fun with and share intellectual posts, and silly memes
• Federate with as many of the great communities as we can

Long-term Goals: (6-12m)

• Make progress to 50 or 100% funding goal
• Create a new skin to make reddthat unique
• Integrate a "Supporter" badge for all supporters

For anyone joining up to this instance <3 For anyone federating with this instance <3 Together we can make a great community, run by the community, for our community.

It's official. We are live.