[-] TheDude@sh.itjust.works 4 points 2 months ago
129
SJW upgrade to v19.5 (sh.itjust.works)
submitted 2 months ago* (last edited 2 months ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

Hello sh.itjust.works community,

It’s been a while since my last update, and I owe you all a more detailed one soon. However, today I’m here to give you a heads-up about the upcoming upgrade.

Over the next 7 days, I’ll be upgrading the instance to v19.5. While this update doesn’t strictly require it, I’ve decided to take this opportunity to upgrade our Postgres database from version 15 to 16 as well. This process can be time-consuming and have take up to 2 hours to complete on other instance.

I'm optimistic that we'll complete it in less time, but to err on the side of caution, I've scheduled the maintenance window for 120 minutes.

To minimize downtime, I've split the upgrade into two phases: Phase one will cover the database update, and Phase two will handle the Lemmy code upgrade.

Note: If the database upgrade is completed in under 60 minutes, I’ll proceed with the Lemmy code upgrade during the same maintenance window.

Here's the tentative timeline:

Task Date Expected Downtime
Upgrade of Postgres Database Friday September 20 2024 @ 8:00PM ET 60-120 Minutes
Upgrade to V19.5 ~~Monday or Tuesday September 23 or 24 @ 8:00PM ET~~ ~~Up to 30 Minutes~~

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

edit: 9:23EDT - Hey all, the database and lemmy codebase has been upgraded to v19.5. The database upgrade took a lot less time than expected so I went right into upgrading the codebase. A whole bunch of background tasks are still running which might make your experience slower than usual. Things should be back to normal in the next couple of hours.

[-] TheDude@sh.itjust.works 6 points 3 months ago

Hi @Blaze@sopuli.xyz

It's coming. This instance is deployed using lemmy-ansible but with some slight modifications. I need to review a few things to make sure the transition is smooth. As you also pointed out, it's also summer time which isn't helping with my free time situation.

I have a long term solution that will speed up upgrades but I only expect to get it in place by the fall. In the meanwhile, I'm planning on getting this instance upgraded to 19.5 in the coming weeks.

[-] TheDude@sh.itjust.works 61 points 4 months ago

Hey,

I'm due to make a post about the instances finances soon. I'll get one posted in the next bit.

I'm still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don't thank them enough but they are the true heroes of this instance.

As some may know, I've been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that's what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.

In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I'm OK paying the costs to keeping the lights on until we can transition to donations.

[-] TheDude@sh.itjust.works 8 points 8 months ago* (last edited 8 months ago)

Yes, I'll get this done

UPDATE: here you go https://oldsh.itjust.works/

[-] TheDude@sh.itjust.works 24 points 8 months ago

There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I'll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.

[-] TheDude@sh.itjust.works 28 points 9 months ago

The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.

557
submitted 9 months ago* (last edited 8 months ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

  • Dual Xeon 2.9Ghz CPUs (32 cores total)
  • 256GB ram
  • 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
  • 10gbit networking

While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.

  1. Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
  2. There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.

Here's the tentative timeline:

Task Date Expected Downtime
Migration to new server Tuesday February 27 2024 @ 8:00PM ET 90 Minutes
Upgrade to V19.3 Thursday February 29 2024 @ 8:00PM ET Up to 120 Minutes
  • If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
  • If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

[-] TheDude@sh.itjust.works 7 points 9 months ago

I've started the preparation steps to minimize the maintenance window needed for this upgrade. These operations are running some background tasks and once they are done I'll make an announcement to give everyone a few days heads up of the anticipated upgrade time and date. Should be soon.

[-] TheDude@sh.itjust.works 5 points 1 year ago

Hi there

Sent you a message on the matrix chat. This should now be fixed.

Thank you

[-] TheDude@sh.itjust.works 8 points 1 year ago

Hey,

I did some research on this.

I will not share the list of accounts as I feel its everyone right to vote and to have their own opinions whether they align with yours or not.

I will however say, Its mostly the same accounts that are down voting you. The majority of the accounts are over 2 months old with content generated by them (they do not seem like bot account). There are a few that are just 10 days old. 99% of the down votes are from other instances with feddit.uk and feddit.nl being the primary source of them.

[-] TheDude@sh.itjust.works 24 points 1 year ago

The big email email providers are not liking the amount of email coming from the server ip. The permanent fix is to use a mail sending service. I do plan on making that adjustment soon.

In the meanwhile send me a dm with your username and email address used and I’ll verify the account for you.

Thanks

54
SJW Rammy Statement (sh.itjust.works)
submitted 1 year ago* (last edited 1 year ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

Hey all,

Similar to the lemmy.world announcement, we have defederated from Rammy.

Want to discuss this more? I created a [Discussion] post for us to do just that.

[-] TheDude@sh.itjust.works 4 points 1 year ago

Hey there,

Send me a PM with what you want to create and i’ll create and transfer it over to you. Disabled it for now until i add additional functionality to the mod bot.

[-] TheDude@sh.itjust.works 60 points 1 year ago

Hey all, I’m always excited to upgrade to the latest versions that become available. Unless the upgrade is related to security I will tend to wait a few days to give other instances a chance to share their experiences. Before upgrading I usually go over the issues that have been submitted to help understand the risks and solutions to mitigate them.

That being said the new version does bring in a whole bunch of performance gains (mostly related to the database) that i’m looking forward to.

Lemmy.world is due to upgrade later today and I plan on upgrading shortly after them (either Monday or Tuesday).

I will put up a notification up prior to the upgrade.

Thanks

2
submitted 1 year ago* (last edited 1 year ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

This instance has been upgrade to 0.18.0.

The lemmy developers have removed the built in captcha so I had to enable email verification. They will be re-implementing this feature in the next update.

Lots of new improvements in this version. The Link i wanted to share is on the lemmy.ml instance and they appear to be down at the moment. Hopefully not related to this update.

Stay tuned.

Edit: Lemmy.ml back up, Here is the post: link of the update

6

Hey all

Today, I am thrilled to share two significant developments with you that will determine our shared future.

Firstly, the issue of donations. Since the inception of this instance, your most frequent request has been the ability to make contributions to support my initiative. While initially, I had never intended to accept donations, I've come to realize the value this brings in ensuring our platform's sustainability. In response to your requests, within the next week, I will be introducing several options for those of you who wish to donate. I want to emphasize that these donations are entirely optional and will directly support our instance's operational necessities - dedicated hardware, colocation fees, email services, and more.

The second announcement needs a full disclosure: it could be an extraordinary idea or potentially not so. Since the beginning of this instance, I've seen a number of insightful posts, recommendations, and ideas that the community has put forward, often superior to what I could have conceptualized myself. Even our instance's name was born from one of your suggestions.

Frequently, I'm approached by users seeking clarity on our rules and guidelines or expressing their thoughts on existing rules. In reflecting upon this, it became clear to me that I've been attempting to determine what's best for the community. But who am I to make these decisions? Just two weeks ago, I was a user among you all. Hence, I'm moving away from the traditional role of decision-making.

Instead, I want to hand over the reins of decision-making to you - the community. I'm excited to announce the creation of a community called agora where you can express your desires for the future of this instance. It's up to you to come together, discuss, and reach a consensus. If you wish to add, remove, or modify a rule, make a post, garner support from other members, and I'll implement the change. This invitation extends beyond our immediate community - I welcome input from everyone across the fediverse. Again to be clear, I gave an example of modifying rules but this applies to anything that I have the ability to do on this instance.

As the instance owner, I only have one caveat to add - any decisions made should not jeopardize the instance's existence or result in legal complications. Aside from that, I'm eager to see where this new direction takes us.

Thank you for your ongoing support, creativity, and engagement.

5
submitted 1 year ago* (last edited 1 year ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

Like many of you, I woke up this morning to discover that our instance, along with lemmy.world, had been unexpectedly added to the beehaw block list. Although this development initially caught me off guard, the administrators at beehaw made an announcement shedding light on their decision.

The primary concern raised was our instance's policy of open registration. Given my belief that the fediverse is still navigating its early stages, I believe that for it to mature, gain traction, and encourage adoption, it is crucial for instances to offer an uncomplicated and direct route for newcomers to join and participate. This was one of the reason I decided to launch this instance. However, I do acknowledge that this inclusive approach brings its unique challenges, including the potential for toxicity and trolls. Despite these hurdles, I maintain the conviction that our collective strength as a community can overcome these issues.

After this happened, the beehaw admins and I had a good chat about their decision. While our stances on registration policies might diverge, we realized that our ultimate goals are aligned: we both strive to foster communities that thrive in an atmosphere of safety and respect, where users can passionately engage in discussions and feel a sense of belonging.

Although the probability of an immediate reversal are slim given the current circumstances, I believe we have managed to identify common ground. It's evident that, even in separation, we can unite to contribute positively to the broader fediverse community.

In the coming weeks or months, we plan to collaborate with other lemmy instance administrators to suggest enhancements and modifications to the lemmy project. Primarily, our proposals will concentrate on devising tools and features that empower us, as instance administrators, to moderate our platforms effectively.

In the meantime, while I understand may not be ideal for everyone, users who choose to participate on the beehaw instance will be required to register a separate account on their instance.

Thank you all for continuing to make this community great!

1

Common Fortigate, do better. I really love your firewall.

1

"Give it to me!" She yelled.
"I'm so fucking wet, give it to me now!
She could scream all she wanted.
I was keeping the umbrella.

** Full disclosure, not my joke but still one I love to share

0
submitted 1 year ago* (last edited 1 year ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

Good morning,

Some of you may have experienced a small hiccup this morning (eastern time). I went ahead and upgraded our lemmy instance to 0.17.4. I also took this opportunity to make some small adjustments to the instance that required a restart of services. These changes were mainly related to increasing Federation worker count in hopes to decrease synchronization times with other instances. A small fix to help slow down the storage growth was also applied (limiting log sizes that get generated).

Everything should be back up now. In the future my goal is to be more transparent with scheduling downtime.

edit: updated link

1
submitted 1 year ago* (last edited 1 year ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

I'm aware that some of you have been getting some errors loading this instance. This was a configuration that I needed to be adjusted which has since been.

Do be patient if we run into other issues, I'll be continuously working in the back-end with others to better your overall experience.

Still a lot of capacity available!

edit: For those that are interested in the configuring changes. I was using the default lemmy configuration for the nginx worker_connections value. This value needed to be raised. Shout out to @ruud@lemmy.world over at lemmy.world for helping me out.

2

Hey all, I created this community thinking I would have time to mod and post in it but its pretty clear now that this won't be the case. If anyone is interested in taking over this community and has the time to properly moderate it please let me know!

1

Hello everyone!

I'd like to first start by welcoming all the new users who have decided to chose sh.itjust.works as their lemmy home instance. It's wild to think that its only been 4 days since I've decided to spin up this instance and we are already nearing 1000 registered users.

During these last few days I've welcomed users from all parts of the world who collectively have created nearly 50 new communities on this instance alone.

Today I want to be transparent and share some of my thoughts, clarify some rules, share some concerns, answer some questions and most importantly introduce the two new instance admins that have been added over the last couple of days. So let's get started!

New Admins

  • @smorks@lemmy.ca - Some of you may recognized this name from the lemmy.ca instance. This is in fact the same person. I've asked smorks to help out on this instance and he accepted. As the userbase grows having someone whos living some of the similar issues will be bring great value to this instance. We are stronger together!
  • @donut@sh.itjust.works - donut is a personal friend, new to the fediverse but has agreed to give me a hand should things get out of control. He's a little quiet right now but he'll open up soon enough.

Rules

As you may have noticed, there aren't a whole lot of rules. This might need to change in the future but my goal as of right now is to keep it this way. Being respectful to all users no matter which instance they come from, part of the world they grew up in, personal believe or personal opinion is a absolute must. Our world is already in shambles, we don't need another place for all of us to fight against one another.

  • Be respectful
  • No bigotry
  • No Ads or Spamming
  • No pornography

You'll may have noticed that a new rule that has been added. No pornography. When pornography is introduced a whole bunch of legal implications need to be considered amongst other things like proper moderation, DMCA notices processes, etc. 4 days ago I would not have thought this to be a problem however I am not in a position to hire legal counsel and implement the elements required to not get this instance shutdown. So for now, no pornography. If the demand grows for this in the future and there is an opportunity to fill this gap I promise I will revisit this rule.

Donations

I've had a few users ask me whether they could start donating. I'll admit I was caught off guard when that happened, thinking of how thoughtful people can be. When I started this instance 4 days ago, I did it because I wanted people to have a home away from reddit, a place where they could come and share interesting topics, debate, discuss and connect.

I'm by no means rich but I have chosen a career that puts me in proximity of things needed to run an instance like this one. It also does help that I really like servers and while some people might really like the latest limited edition sneakers from X company or the latest iPhone/Android device, etc..., I really really like servers, datacenters and networking.

I don't have a plan for accepting donations today but seeing how this community is growing, I will need to consider it soon. You know what's more expensive than running a lemmy instances for thousands of users? a failed marriage.

There are some other options that might be available such as looking for sponsors but I do want to keep this instance about the users and not about corporate so I'll need to think hard about this one.

Whatever is decided I'll make another post with the details.

Concerns

Let's quickly talk about concerns. Most of you already know that many of the reddit mods are orchestrating a protest by blacking out many popular subreddits. Most of us anticipate a large spike in traffic towards our instances during the period of June 12 to 14th. If any of you were around during the Digg days, you may remember how the migration from digg to Reddit went. The early reddit days was filled with outages, performance issues, and frustration.

A lot of people people are looking for what might be the next reddit. We don't knows if lemmy will be it but a lot of people will surely be giving it a shot. During that time many instances including this one will likely deal with some growing pains.

From a resources perspective my biggest worry for this instance is related to the storage. I have CPUs, I have RAM but when it comes to storage, well its an element that has been growing much faster than I had anticipated. I will be keeping a close eye on this however want to wave a little yellow flag so that everyone is aware until I work out a solution.

Q&A

What communities allowed to be homed here?

  • As long as it follows the rules above and doesn't get any DMCA requests or jeopardize this instance from being shutdown or me in trouble. Go nuts!

What communities allowed to be federated over/subscribed to?

  • Same as the above and follow rules.

What am I allowed to link to in communities or text posts?

  • Same as the above and follow rules.

What kind of images am I allowed to upload to the integrated image host?

  • Storage is something that will likely become a problem in the foreseeable future. You all upload a lot! That being said, don't upload copyrighted materials or anything that will get me in trouble!
view more: next ›

TheDude

joined 1 year ago
MODERATOR OF