RobinLive search engine for Fediverse and Bluesky

Bill Statler

US, Sun, 09 Mar 2025 21:16:29 -0400 from Bunny of Doom

Well, this should be controversial within a day or two. Introduced today by Charles Iliya Krempeaux (aka 'reiver' on several social networks) at the "Fediverse House" meetup at SXSW:

RobinLive

Decentralized social media search engine integrated with AI, AT Protocol, BlueSky, SpaceHost.

Looks like it ignores Mastodon privacy settings:

The Nexus of Privacy wrote the following post Sun, 09 Mar 2025 19:53:47 -0400

@@@reiver ⊼ (Charles Iliya Krempeaux) does robin.live respect existing opt-in/opt-out settings? Because my post at https://infosec.exchange/@thenexusofprivacy/114134881967629586 showed up in a search their for #FediverseHouse even though my "privacy and reach" settings have currently turned off "Include public posts in search results" and "Feature profile and posts in discovery algorithms".

Waitman Gobble

Sun, 09 Mar 2025 21:46:01 -0400 from Get Ready To Rumbly

Hey you show up #1

Mike Macgirvin 🖥️

Sun, 09 Mar 2025 21:56:38 -0400 from Y

I have to laugh at folks that continue to use robots.txt as a model for decentralised permissions in 2025.

"You aren't allowed to do this, so please stop doing it. Pretty please? How about pretty please with ice cream on top?"

Waitman Gobble

Sun, 09 Mar 2025 22:00:32 -0400 from Get Ready To Rumbly

Looks like it took 20s for my test search string to show up.

Bill Statler

Sun, 09 Mar 2025 22:01:50 -0400 from Bunny of Doom

If it's open-source, I haven't found the source. I don't know how it works.

Waitman Gobble

Sun, 09 Mar 2025 22:05:37 -0400 from Get Ready To Rumbly

Oh. Good point. I just generated 2,048 random characters in a post and put them in the "live search" on the robbin' site

Scott M. Stolz

Mon, 10 Mar 2025 04:55:00 -0400 from hubzilla

One concern is that private posts could wind up in their search, depending on how they source their posts.

For example, what if you sent a post only intended for certain recipients via Hubzilla or (streams), but one of the recipients was on Mastodon, which does not understand privacy. Will that private post now leak to this public search?

Bill Statler

Mon, 10 Mar 2025 13:44:38 -0400 from Bunny of Doom

I think we send out that sort of message as a direct (private) message to each recipient, don't we? Mastodon ought to understand that.

But it really depends on how RobinLive actually works. My guess is they're just scraping every new post that is created or delivered on a list of large instances. That ought to be safe... I think.

Scott M. Stolz

Mon, 10 Mar 2025 16:08:27 -0400 from hubzilla

I love the idea of a fediverse search. But people should be able to opt out. We have that setting in Hubzilla, and based on what I heard, Mastodon has something similar. Those user preferences should be respected.

*_jayrope

Mon, 10 Mar 2025 16:32:55 -0400 from hubzilla

@Scott M. Stolz I'd say people should be able to opt-in, not opt-out. @Mike Macgirvin 🖥️

Scott M. Stolz

Mon, 10 Mar 2025 17:01:23 -0400 last edited: Mon, 10 Mar 2025 17:06:22 -0400 from hubzilla

@*_jayrope In an ideal world. Unfortunately, if you publish anything to the public, it's public. Anyone can see it. They can copy it, quote it, screenshot it, index it, feed it to a bot or AI. Public is public. The most you can do is ask them to not use your material, and hit them with a copyright lawsuit or DMCA takedown notice for using your material without permission, and that may not even work in some use cases due to fair use rules. For example, quoting and indexing publicly posted information are considered fair use in most countries.

This is the specific reason why Friendica, Hubzilla, (streams), and Forte all have post permissions and access control. If you make your post public, you can't trust people to respect your wishes. And in most countries, the law is not on your side.

*_jayrope

Mon, 10 Mar 2025 17:28:28 -0400 last edited: Mon, 10 Mar 2025 17:30:46 -0400 from hubzilla

@Scott M. Stolz In a normal (not ideal, but rather reasonable) world people opt-in to sthg they conciously want. In the most common demoninator of the conditions of now ... people are forced to opt-out of sthg. only when they learn about its existence and yet mostly haven't fully understood, what that means at all or what it is about in detail. So opt-out almost never makes any sense.

Scott M. Stolz

Mon, 10 Mar 2025 17:41:37 -0400 from hubzilla

@*_jayrope I would like to see platforms broadcast a user's preference for content search and being included in directories. And I do think that respectable platforms and search engines will follow that.

But I also think that we need to educate people that public is public, and if you don't want certain people to see it, it should not be public. The main reason being that we cannot force nefarious actors into complying with our wishes. And we already know that governments are slurping up public information. We should not give people a false sense of security. If you insulted someone in Germany, the police may come knocking, regardless of your search preferences. It's against the law to insult someone in Germany and much of Europe.

It ain't right, but it is what we have to deal with.

Mike Macgirvin 🖥️

Mon, 10 Mar 2025 16:18:30 -0400 from Y

The issue is enforcing only things you can enforce. You cannot stop somebody searching for something on their own computer. Can't be done. You can stop somebody from searching for something on your computer. streams searches respect the latter. I would assume this is also what Mario implemented, though I might be wrong. Mastodon search permissions are just "pretty please don't search my stuff", but there's no way possible to enforce it. I have the same issue with Mastodon quote posts permissions. You cannot stop/prevent somebody from talking about you behind your back. Can't be done. If you're offering this as a "feature", prepare for the bug reports, because they will continue until you remove the feature. Prepare also for losing lots of "customers" because you lied to them by suggesting that the setting actually works in the first place. In a federated world, the only permissions you can enforce are those on your own server. Pretending you can control everybody and every platform in the network is hilarious -- and wrong.

Scott M. Stolz

Mon, 10 Mar 2025 16:37:08 -0400 from hubzilla

@Mike Macgirvin 🖥️ Good points. The setting is "Opt-out of search engine indexing" and it probably should include the disclaimer that it discourages search engines from indexing the content, but doesn't stop them. I am not sure how it works, probably robots.txt, but, as you said, it can't actually stop search engines from indexing it. I am not sure if this is communicated via Web Finger or anywhere else though. But even if it was, there is no guarantee that anyone would respect it.

The most that can be done is put pressure on projects that we are aware of. FediDB was pressured to support robots.txt, for example, and made that change. Perhaps Robins.Live could be persuaded to do something similar.

Mike Macgirvin 🖥️

Mon, 10 Mar 2025 16:56:03 -0400 from Y

Mastodon could start by making the setting available in an FEP instead of it being being a proprietary platform specific setting. But the thing about FEPs is they are optional extensions people are free to ignore. If you want this permission to be honored by everybody, it needs to be part of Activitystreams or ActivityPub specifications. Otherwise it's frankly dishonest.

The streams canSearch setting is advisory only, because it is already enforced by the server. This does not require any specifications to support - if you don't have permission to search, we're going to 403 the search endpoint or filter the content that you are not permitted to access. The canSearch setting just lets you know we're going to do this. We aren't asking people on other servers to support this because we do not need to do so. We're enforcing our permissions on our servers and the setting works as described. Mastodon's setting only works on Mastodon servers. And since it is open source, it doesn't even work on all Mastodon servers. So it's a joke.

Wishful thinking is not a permission. And robots.txt is also a joke. Always has been, always will be.

Mike Macgirvin 🖥️

Mon, 10 Mar 2025 17:04:38 -0400 last edited: Mon, 10 Mar 2025 17:09:21 -0400 from Y

And "putting pressure on other projects" is Trump behaviour. "You must conform to our wishes because we're Mastodon and we're big and popular." It's just like the Mastodon alt-text nazis - bully everybody into submission.

Don't even get me started.

Now let's say for a moment I'm a fediverse admin. Somebody uploaded a video that is problematic. We don't know which one, so we need to search our database for recent videos. It's our own database. But first I need to find out who published the video and find out if I'm allowed to look for their video in my database. Bugger off. I'm going to search for it whether you like it or not, because it is causing problems on my server.

Scott M. Stolz

Mon, 10 Mar 2025 17:16:36 -0400 last edited: Mon, 10 Mar 2025 17:30:21 -0400 from hubzilla

@Mike Macgirvin 🖥️ I didn't say it was right. And there is a difference between asking and demanding.

Personally, I don't mind when people ask for stuff. If it aligns with what I am doing and I have time and resources to do it, I will seriously consider doing it. I like making people happy. But, on the other hand, if someone demands or tries to force me to do something, screw them. I'll say no just because of how they asked for it. Treat me and other people with respect, or it is an automatic no. There are penalties for being an asshole.

To give you an example of what happened with FediDB. A single person decided that they would publish node info, but then put in their robots.txt to not index node info. The logical thing would be to just disable node info, but no, this guy had to make a point. He even put false data in node info to corrupt the stats! So FediDB caved and updated their site to look at robots.txt. I can understand asking for FediDB to support robots.txt. Daniel seems to be a nice guy and probably would have supported it if asked nicely. But to me, trying to force him was just a jerk move and I disapprove of that.