How could I miss the opportunity to use this picture!
It definitely felt like that at times.
Haha, I couldn't resist. Great post though.
I was really hoping this was going to be a rant about clouds in the sky.
As someone who burns with the faintest touch of stellar light, I love clouds in the sky.
So the whole thing is well worth a read IMO, and addresses a lot of the issues I have with cloud as the solution for everything.
My main point here is that individuals and organizations that require all the flexibility that cloud services offer are a (tiny) minority. This means that for the majority of us, all the complexity necessary to provide this flexibility ends up being purely a complication or worse, a liability.
There are absolutely companies who need the scaling. But it's a fucking lot of overhead if you don't.
Let's repeat it one more time: complexity hides and creates security issues.
This is similar to all the LLM code stuff. If you don't actually fully understand what your code does, bad stuff happens.
This premise has the consequence that Cloud systems are a big puzzle. The pieces of the puzzle are the Cloud products. Engineers working with Cloud systems essentially need to understand the abstraction but not necessarily the underlying, ultimate working mechanism of what those abstractions do. For example, a cloud expert might know everything about the difference between NACLs and Security Groups, all the details about how to configure them, their limitations etc., but the main idea is that such expert doesn't need to know anything below that (e.g., how the traffic is filtered).
Ultimately my perspective, and I appreciate it's a very personal one, is that building and working with the Cloud makes me feel like a glorified application administrator. My job becomes researching how the Cloud solved the problem that I need to solve, and compose the solution in the way the Cloud provider imagined it should be solved, rather than solving the problem
I was going to bring up basically this point:
because vendor-lock is not something that has only to do with infrastructure. It has also to do with the skills of the engineers involved. Cloud knowledge, for the most part, is not portable. You are a wizard of IAM policies in GCP? Good job, this is completely useless if you go to Azure. Oh, you are a guru of VPCs and private endpoints? Well done, this is completely useless if you move to a different cloud.
But you covered it pretty well. Abstractions are great. Proprietary abstractions that are more focused on how they can bill you than real, useful, functional categories? Not so much.
Despite the efforts means something which is ironic: many companies which run on Cloud, at some point, will have one or more teams whose main purpose is understanding how they are spending money in the Cloud and to reduce those costs. If this sounds conflicting with the idea of reducing personnel, well, it is. The digital infrastructure of my organization is not that huge. Give or take 2000 compute instances (some very small). Something that 200 servers could easily provide. Cloud bills are more than $15 millions/year. I checked a server builder for example, and an absolute beast (something like 2x Xeon platinum processor, 200TB of NVME disks, 1TB of RAM etc.) would still stay comfortably under $250k. 100 servers this powerful will probably be a multiple of our computing power, and cost almost a third if we consider a lifetime of 3 years, which is very low. A more realistic estimation of 5 years leads to a saving of ~$50 millions over 5 years. Completely insane! This is of course if you want to buy hardware. Powerful servers rented run you for $500-1000/month. Assuming a cost of $1000/month, my company could rent more than 1000 powerful servers, and still save money compared to Cloud costs, leaving plenty for additional services such as networking, storage, premium support (remote hands) or actual engineers salary
So there's a level of rent seeking behind all the software moving to subscriptions, and them wanting to lock you in just like their service providers are doing to them. But I have to think the massive costs of cloud junk also pay a role in stuff like a calendar charging double digit annual fees for something that takes very little storage and very little computation (and you of course can't just buy software any more).
I have no words for multi-cloud. Even like a Facebook or YouTube scale site, are you really going to double (or more for some reason?) your storage costs (plus whatever intercommunication between the two), just in case the provider goes down for a couple hours (which is extremely rare, and you won't be the only site impacted, so people won't really blame you for.) Plus that architecture sounds like the shitshow to end all shitshows.
Agreed on it all.
I think a big driver for cloud clients is bean counters - cloud is an expense, while having your own systems is capital investment.
They'd rather have the waste of leasing too much compute than have to pay taxes on systems plus the cost of staff to run it.
We won't really see this get addressed until companies have to truly own the risks they take on (see all the hacks that happen on a daily basis because CIO won't pay for the security that IT management is screaming to build). When fines for these breaches are meaningful, cloud will be less interesting.
Thanks!
But I have to think the massive costs of cloud junk also pay a role in stuff like a calendar charging double digit annual fees for something that takes very little storage and very little computation (and you of course can’t just buy software any more).
Absolutely agree. I did not even think about this aspect, but I think you are absolutely spot on. Building something with huge costs is something that ultimately gets passed to the users in addition to the rent-seeking aspect.
I have no words for multi-cloud.
You and me both. I have to work with it and the reality is, there is nobody who actually understands the whole thing. The level of complexity (and fragility, I might add) of it all is astonishing. And all of this to mitigate some (honestly) low risk of downtime from the cloud provider. I have lobbied a little bit against at work, but ultimately it has become a marketing tool to sell to customers, so goodbye any hope of rational evaluation...
Great post, a quick nitpick if you don't mind, introduce or use an abbreviation's full words before using its abbreviated form
Granted that the article is geared towards sysadmins and cloud developers, others who may want to read it may have a hard time doing so. As an example, reading through the first technical point, I saw "IAMs" and "Network ACL", I don't understand what those abbrs mean
Thanks, that is a very good observation! I will try to sneak an edit later today where I can add some appendix about acronyms and abbreviations.
Edit:
While it might not look great, I have added at the bottom an Appendix with all (hopefully, I might have missed some) acronyms and abbreviations. Thanks for the suggestion!
The cloud is just someone else’s computer
With a lot of stuff on top!
I'm sorry, but this started like a recipe article and I lost all interest. I don't care about your life story, I clicked the link to read your opinions, and you spent the first several paragraphs avoiding them.
Nothing to be sorry for. I didn't write for you nor for any particular individual, and it's fair if you are not interested in it. I also added a table of content at the beginning, so you can jump directly to the relevant section (Technical Side) skipping the (in my opinion needed) introduction completely, if you wish. Cheers
Two brief paragraphs of light nonsense on a blog post, then a quick summary of what the article will cover?
Tell me you don't read often without telling me you don't read often:
I hate websites with low contrast text.
How do you get this? Anything that tries to force a light mode?
This is how the site is supposed to look like (there is no light/dark theme selection):
I was reading the site on Android, and it looked dark, but after seeing this comment, I tried disabling Android system wide dark mode, and sure enough it became white like in the screenshot! For the record, I tried with both Firefox and a Chromium-based browser.
Thanks! I went and tried on my phone and indeed setting Firefox to light mode indeed causes that horrendous and unreadable result. I will need to figure out way, eventually, and provide an alternative light scheme.
I get the same white background on Windows, Chromium and Firefox. Checking settings, I see FF is set to "Automatic" light/dark mode. When I manually select Dark mode, I see the dark background.
there are too many points of failure for me to ever be comfortable using the cloud as a primary storage option.
i've always maintained this opinion when "the cloud" started being touted as being the future. and yet more corporations (including mine) are reliant on it. i mean sure, i can log in on my home computer and have some access to stuff as though i were physically at the office but that convenience ain't worth the headache if the main storage site crashes.
there are too many points of failure for me to ever be comfortable using the cloud as a primary storage option.
If everything that you run is local as in the same physical location and there is no requirement for external or internet access then sure. Not everyone has that luxury. Otherwise, There are the same number of points of failure in a non-cloud configuration. You just feel more comfortable with those because you have direct hands on control.
Having done everything from building my own servers 30 years ago to managing hundreds of servers in data centers to now managing hundreds of instances and other services in AWS, I’ll gladly stick with AWS. The hardware management alone makes it well worth the overhead.
25 or so years ago I had to troubleshoot a hardware issue in a SCSI-based server with 6 hard drives in it. A drive appeared to be failing so I replaced it and immediately another drive failed, then another, and so on. After almost a full day of troubleshooting later and we realized the power supply was actually the culprit and could no longer provide sufficient power to the full set of hard drives.
20 years ago while managing 700+ servers in a datacenter we had to manage a recall of about 400 of them thanks to the Capacitor plague that caused a handful of our servers to literally burst into flames.
Hardware failures like the above and dozens of others were mitigated in most cases thanks to redundancies in the software we wrote. But dealing with hardware failures and the resulting software recovery was a real PITA.
With AWS I may occasionally have a Linux instance lock up due to a hardware failure but it’s usually fairly easy to reboot the instance and have it migrate to new hardware. It’s also trivial to migrate a server to run on more (or less) number of CPU’s, RAM, etc. with only a couple of minutes of downtime.
The more advanced services AWS offers like object storage, queues, databases, etc. are even more resilient. We occasionally get notified that a replica for one of these services had failed or was determined to be on hardware that was failing, and it was automatically replaced with a new replica.
I’d much rather work this way than the way I did 20+ years ago.
If the storage "crashes" it doesn't matter if it's in the cloud or on-prem.
With the cloud you get two substantial advantages:
- the storage is built so it doesn't break so easily. I trust AWS engineers more than Mike, no matter how cool Mike is to hang out with. Additionally, if the storage breaks while Mike is on vacation we're screwed, with the cloud you get a whole team 24/7 on it.
- you can prevent data loss with backups or multi-region setups with a few clicks/terraform lines. Try telling the PO that you need to rent datacenter space in Helsinki and Singapore for redundancy...
Of course all this costs big bucks, but technically it's superior, easier and less risky.
trust AWS engineers more than Mike, no matter how cool Mike is
-
AWS engineers' first responsibility is to shareholders
-
Mike's responsibility is to your same boss.
They are not the same.
Bonus: you can see Mike's certs are real.
the storage is built so it doesn’t break so easily. I trust AWS engineers more than Mike, no matter how cool Mike is to hang out with. Additionally, if the storage breaks while Mike is on vacation we’re screwed, with the cloud you get a whole team 24/7 on it.
That's easily mitigated just following established standards. Redundancy is cheaper than anything else in the aftermath and documentation can be done easy with automation.
you can prevent data loss with backups or multi-region setups with a few clicks/terraform lines. Try telling the PO that you need to rent datacenter space in Helsinki and Singapore for redundancy…
You don't, you rent rack space in a location far enough away but close enough to get the data in a few hours.
It's neither superior, easier or less risky, it's just a shift in responsibility. And in most cases, it's so expensive that a second or third on site engineer is payed for.
Yes, I hate cloud too. Now tell this to my company, which received about 100k dollar credits from Azure and Google Cloud :)
Very good read. I totally agree with your sentiment that more and more, "engineering" is becoming just gluing together and managing cloud services and features.
My job as a sys admin has become the same. It's not about actually understanding the technology at a deep level and troubleshooting problems, it's about learning specific applets and features to click on and running down daily and weekly checklists.
I feel you very much. Security work is also somewhat similar.
I think this takes a way basically the component that made it interesting, understanding what you are doing to the point that you can build stuff.
it's about learning specific applets and features to click on and running down daily and weekly checklists.
Well said.
I'm immensely disappointed!
Not kidding: when I first saw the post title, I was fully convinced that I'll read the post of a crazy person, rambling about (rain) clouds.
I am sorry! As an amateur landscape photographer I actually like very much those clouds. There are a few r-word posts about people hating those clouds though, but I checked and they are nowhere near as long as you would expect a proper rant to be
Anything that requires a fancy buzzword is usually stupid but a good way to make money for someone. The "cloud" has always existed as offsite hosting. Off-site shared servers, VPSs, whatever. It's no different than running CPanel on an LAMP VPS in 2003.
But calling it "the cloud" gave all the business majors a hard on and then the accounts department realized they could manipulate share pricing by reducing the amount of assets a company holds. It's the same stupid reason many companies don't own their corporate headquarters or remote centers. They lease the, even if from themselves through another holding. It looks better on paper so the share price goes up. It's all mind boggling stupid.
The cloud today significantly different than the 2003 cpanel LAMP server. It's a whole new landscape. Complex, highly-available architectures that cannot be replicated in an on-prem environment are easily built from code in minutes on AWS.
Those capabilities come with a steep learning curve on how to operate them in a secure and effective manor, but that's always going to be the case in this industry. The people that can grow and learn will.
I'm fully aware of the few buzzword and marketing pitches that cloud hosting uses. I'm forced to use both GCP and AWS for different contracts and I'm good at it.
The real truth is that most websites and internet services do not need scale. They do not need all this crap. A Pentium 3 could host all the data for most of these businesses and services. You don't need serverless lambda functions to handle an api when an actual endpoint does the same thing to pull some info. The few companies that need such distributed computing and power, will need a big on-site or off-site implementation. It makes sense for that sometimes. But most times, it doesn't even then. You're just outsourcing your engineering and paying a premium.
I have seen so many startups spin up cloud accounts costing thousands of dollars a month when they're in "private beta stealth". Literally a $500 laptop could host all of their services just as quickly with no monthly fee. But as long as the VCs are paying, just flush that cash down.
Thanks for sharing. Great read and points.
Technology
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed