How does Mastodon react to delivery of a post from streams?

Wed, 11 Sep 2024 00:05:27 -0400 from Bunny of Doom

Bill Statler wrote the following post Wed, 11 Sep 2024 00:05:25 -0400

How does Mastodon react to delivery of a post from streams?
I'm doing a little digging through my server logs. This is related to the issues Giac has been seeing for 2 months. Question: When a Streams instance sends out a post via ActivityPub, what does a Mastodon server send back within the first 1-2 minutes?

It looks like Mastodon may request to search for a hashtag mentioned in your post. (I'm not sure if this always happens.) Here's an example, searching for #⚠️ :

195.201.25.153 - - [05/Sep/2024:19:22:10 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.10; +https://masto.ai/) Bot"
57.128.95.181 - - [05/Sep/2024:19:22:13 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.12; +https://goblin.masto.host/) Bot"
5.75.161.98 - - [05/Sep/2024:19:22:21 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.10; +https://mstdn.io/) Bot"
46.22.99.207 - - [05/Sep/2024:19:22:35 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.12; +https://social.vivaldi.net/) Bot"
18.169.170.229 - - [05/Sep/2024:19:22:36 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.2.0 (Mastodon/4.2.10; +https://backend.newsmast.org/) Bot"
54.91.240.230 - - [05/Sep/2024:19:22:37 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.12; +https://flipboard.social/) Bot"
94.130.105.194 - - [05/Sep/2024:19:22:41 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.green/) Bot"
141.95.205.35 - - [05/Sep/2024:19:22:43 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.12; +https://fosstodon.org/) Bot"
78.46.218.100 - - [05/Sep/2024:19:22:46 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.12; +https://social.touha.me/) Bot"
135.181.87.175 - - [05/Sep/2024:19:22:47 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.1.4; +https://jako.social/) Bot"
51.77.122.82 - - [05/Sep/2024:19:23:00 -0400] "GET /search?tag=%E2%9A%A0%EF%B8%8F HTTP/1.1" 403 27227 "-" "http.rb/5.1.1 (Mastodon/4.2.10; +https://universeodon.com/) Bot"

My channel has Grant search access to your channel stream and posts set to "Restricted - Connections Only", so my instance responds with 403 Forbidden. If you have this set to Public or Semi-Public, a whole lot of identical searches will be running on your instance at the same time. This could maybe cause a freeze-up.

(More research to follow.)

@Bill Statler

Giac El Vecio

Wed, 11 Sep 2024 07:04:39 -0400 from Unio Diversi Spiritus

Interesting, but I also use 'restricted-connections only'.

Bill Statler

Wed, 11 Sep 2024 15:11:52 -0400 from Bunny of Doom

That is what I assumed. So this isn't the cause of your problem (probably!)

I'm trying to make a complete list of the most-demanding things that Mastodon is doing to us.

Giac El Vecio

Wed, 11 Sep 2024 08:19:06 -0400 from Unio Diversi Spiritus

Test comment

Giac El Vecio

Wed, 11 Sep 2024 08:27:06 -0400 from Unio Diversi Spiritus

Having some problem, little little coma here today, but I or we need more tests, because it could be a fórum/group problem and not the "same" problem
I remember days ago the coma in the @Jupiter Rowland's (streams) outlet site after a comment in a group channel...

Bill Statler

Thu, 12 Sep 2024 22:25:44 -0400 from Bunny of Doom

Continuing the list of "What does Mastodon do to us?"...

We comment on somebody's public post, and our comment gets delivered to the poster. And then all of his followers send an HTTP "GET" request to our instance (not to the original poster's instance!) for a copy of our comment. That can be many GETs in just a few seconds.

If these requests arrive via ActivityPub, they use "apgateway":
https://codeberg.org/streams/streams/src/branch/dev/src/Module/Apgateway.php

From looking at Giac's server logs, it appears that apgateway is pretty slow, and the requests may pile up until our server surrenders and starts replying with "504 Gateway Timeout" to every request. I don't know why it is slow (and I don't know for certain if this is where the slowdown is).

Bill Statler

Fri, 13 Sep 2024 23:13:49 -0400 from Bunny of Doom

One more for the list: FediFetcher, an optional third-party addon used mostly by very small Mastodon instances:

FediFetcher is a simple Python script that can pull missing posts into Mastodon using the Mastodon API. It can be run as either a GitHub Action, as a scheduled cron job, or a pre-packaged container.

Here is what FediFetcher can do:

It can pull missing remote replies to posts that are already on your server into your server. FediFetcher can be configured to
fetch missing replies to posts in your home timeline,
fetch missing replies to your bookmarks,
fetch missing replies to your favourites,
fetch missing replies to posts in your lists.

It can also backfill profiles on your instance. FediFetcher can be configured to
fetch missing posts from users that appear in your notifications,
fetch missing posts from users that you have recently followed,
fetch missing posts form users that have recently followed you,
fetch missing posts form users that have recently sent you a follow request,
fetch missing posts from users that have recently been added to your lists.

FediFetcher is almost certainly most suitable for users of single user and other very small mastodon instances. Whilst it could be used elsewhere, the need on larger instances will be far less, as these naturally pull in far more posts from remote instances.

This could possibly put a large demand on your server. There are a few ways to block it:

You can block FediFetcher using either the robots.txt or your firewall based on the User Agent.

FediFetcher uses a User Agent of the form FediFetcher/{VERSION}; +{SERVER} (https://go.thms.uk/ff), where {VERSION} is the currently running version of FediFetcher, and {SERVER} is the instance of the user that's making use of FediFetcher.

So, you can put a robots.txt file in the root directory of your instance, containing the following:

# Block FediFetcher
User-agent: FediFetcher
Disallow: /

That should work.

Mike pointed out that we also have a mechanism for blocking evil bots that ignore robots.txt. You can take the existing fckmeta addon, and modify it to add FediFetcher or other things you need to block. For example:

function fckmeta_useragents()
{
    return ['fb_iab', 'instagram', 'facebookexternalua', 'fedifetcher'];
}

Christopher

Sun, 15 Sep 2024 10:53:56 -0400 from forte

I am throwing a guess and will honestly say I have no idea, except could it be related to Authorized Fetch?
Maybe you've already thought of that. 🤷‍♂️
https://docs.joinmastodon.org/admin/config/#authorized_fetch

Bill Statler

Sun, 15 Sep 2024 15:30:55 -0400 from Bunny of Doom

Maybe -- Unfortunately, I'm not smart enough to understand how Mastdon's Authorized Fetch would interact with us.

Giac El Vecio

Sun, 15 Sep 2024 14:15:13 -0400 from Unio Diversi Spiritus

https://github.com/mastodon/mastodon/issues/31921

Bill Statler

Mon, 16 Sep 2024 12:17:30 -0400 from Bunny of Doom

"Replies" collection: Some Mastodon instances request this. (Giac's logs show only one instance out of 30+ that tried.) Quoting Mike from 2024-08-17:

This is a long shot, but one change that was made around that time frame was the addition of a Mastodon compatible 'replies' collection. Normally this wouldn't be an issue, because we only fetch this endpoint when we can't find the original parent of the thread, so it's kind of rare. But it occurred to me that there's an off-chance that the elephant people might be fetching this endpoint every time they receive a message. This would be incredibly wasteful, but such an action could have a debilitating impact on some servers due to a subsequent explosion of processes. They're already hitting your server with requests for the actor record on every single message delivery.

To find out if this is the case, comment out src/Lib/Activity.php::L1139
$activity['replies'] = z_root() . '/replies/' . basename($item['mid']);
and see if performance improves.

Bill Statler

Sun, 22 Sep 2024 14:59:10 -0400 from Bunny of Doom

Mike has added (to 'dev') a way to kill the "replies" collection. Edit .htconfig.php to add this line:

App::$config['system']['show_replies_collection'] = '0';