Last night, between 23.00 on 23 June and 1.00 AM on 24, the Poliverse instance "exploded" a thousand messages around the fediverse!
** We have to apologize to a lot of people: ** • we apologize to anyone who was ** woken up by our notifications **; • we apologize to those who have cisted their ** federated timeline invaded by messages ** coming from Poliverso; • We apologize to the ** administrators and instance moderators who know us ** who probably wondered who hacked our server; • We apologize to the ** administrators who didn't know us ** and who noticed us in the worst way • We apologize most of all ** to some of our users ** who, through our fault, have ** UNKnowingly ** behaved as a spammer ...
Unfortunately, we could not foresee this collapse and we did everything possible to limit the damage: unfortunately we are sure that our instance or some of our users have ended up in some blacklist.
** If so, we ask you to read this message, in which we explained what happened. **
PS: if any Friendica expert should realize that some considerations are wrong or should be better defined, please let us have your own considerations in the comments!
What happened to the Poliverso server?
The update
The new Friendica release, the 2022.06 called “Giant Rhubarb” is a seemingly useless release, because nothing changes for users! Yet it has greatly improved compatibility with other software in the fediverse: there has in fact been immense work by the developers and there have been many changes; in such cases it is possible that disservices arise and, although some instances have already installed an exploratory version, it is clear that some problems can only emerge during the production phase.
Precisely for this reason we decided to proceed with the installation only several days after the release, to understand what the problems would be for users and notify them in time; and yesterday 23 June at 22.00, the Poliverso server was updated.
The problem
At first it didn't seem like there were any problems and, in fact, after half an hour we stopped monitoring. But at 11.00 something happened ...
Admin 1 22.56 (the developers of friendica) have changed something: some users get a lot of notifications about old things .. then everything is fixed 22.56 (I'm seeing a lot of queue in processes)
Admin 2 23.00 So far I am not having this type of problem. Actually, I am not having any problems 23.01 To be more precise, I still can't see anything different than before ... 😅 23.04 Right now, after a few tens of minutes of pause, he's posting stuff on Twitter 23.07 Holy shit! my accounts are flooding the universe with messages
Admin 1 23.14 in what sense? : D
Admin 2 23.15 in what sense? The system is shitting out all the messages posted by the RSS Feed. It is a disaster. I'm emptying the ocean with a spoon 🤣 23.15 This time we are banned from all the galaxies of the fediverse
Admin 1 23.16 I noticed a lot of RSS stuff, but I said to myself "boh"
Admin 2 23.16 They are the queues of those messages
In short, yesterday there was a real disaster for which the Poliverse instance will pay the consequences for a long time as in reputational terms. However, the management of the problem was absolutely timely both from the point of view of resolution (at 1:00 AM almost all the messages had been deleted) and from the point of view of communication.
https://poliverso.org/display/0477a01e-4062-b4e8-2a73-375308279474
The reasons for the problem
However, the problem was not systemic or due to problems of administration of the instance; the problem was due to a series of factors that multiplied its impact:
- Friendica upgrade bugs exist: the "system" upgrade all together, some integration problems with other activitypub instances; but all this causes a "re-reading" of the notifications from the RSS FEED; in this way, users who also repost some posts from the feed get** a new repost (😱!)** of the feed.
- this system (republishing from feed) is not very common among Friendica users; so the problem was not noticed too much on the forums
- this system, however, is very powerful and we Poliverse administrators have often sponsored it with users: therefore some of our users use it; and some "service" and information users that we manage directly on a personal level or as administrators also use it: [@ macfranc @ poliverso.org [(https://poliverso.org/profile/macfranc), [@ informapirata @ poliverso .org [(https://poliverso.org/profile/informapirata), [@ news @ poliverso.org [(https://poliverso.org/profile/notizie), [@ cybersecurity @ poliverso.org [(https: //poliverso.org/profile/cybersecurity), [@ privacypost @ poliverso.org [(https://poliverso.org/profile/privacypost), [@ piratepost @ poliverso.org [(https://poliverso.org/ profile / piratepost), [@ instruction @ poliverso.org] (https://poliverso.org/profile/ Formazione)
- inside Friendica there are no valid administration or moderation tools (an ancestral and well-known Friendica problem: https://github.com/friendica/friendica/issues/8724), which would allow us to delete messages in bulk or to immediately stop the republishing service from feed or to temporarily suspend the "unwittingly spammy" users. So we either manually deleted and deactivated everything or had users manually delete their messages - this was an incredible waste of time!
The reputational damage
All of this prevented us from predicting the problem and then amplified it. However, we can assure you that the management of the application was professional and within a couple of hours from the alert, we have erased all traces of the problem!
The fact remains that some users who followed the accounts in question probably wondered "what's going on?"; others have probably defollowed, silenced or banned the accounts of innocent and unaware users ...
The same thing will have happened for administrators: some of them, to protect their users from spam, will have silenced or banned users; or they will have silenced or banned the instance!
Fortunately, those in the Central European Time Zone were probably asleep at the time and were not online on Friendica, Mastodon, Misskey, Pleroma or Hubzilla
Final thoughts
Of course during those hours we felt [a furious rage] (https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782) towards the Friendica developers who caused us such a problem...
https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782
...but we clearly don't get mad at them. Moreover, it is thanks to them that Friendica exists; it is thanks to them that there is what we believe is the software of the fediverse that allows the most complete social experience today.
Next time, however, we recommend that you report the problem well and above all, express at least a little regret to the Friendica instance administrators and other administrators who had to manage what seemed to all intents and purposes a massive spam activity!