Board errors

[bradrn]

Fair. I am susceptible to heavy mood swings because of my autism spectrum condition. I have to work against that. Sorry for the embarrassment.

It’s fine. I think we all understand…

[WeepingElf]

Let me utter a thought here I had also laid out in a private message to zompist: I wonder how long the Web can survive the onslaught of the AI scrapers. When more and more web sites break down, the big players may revert to the model of the centralized online service, like AOL or CompuServe in the early 90s. Such a move would also be to the liking of the far right, as such services would be far easier to monitor and censor than the rhizome-like Internet. And that's another thing the far right probably like about AIs, on top of the fact that AIs have no opinions of their own and just parrot what they are trained on.

[bradrn]

Let me utter a thought here I had also laid out in a private message to zompist: I wonder how long the Web can survive the onslaught of the AI scrapers.

We’ve already discussed the existence of tools which can limit the scrapers, most notably Anubis. They definitely make things more difficult but it’s hardly impossible.

[WeepingElf]

Let me utter a thought here I had also laid out in a private message to zompist: I wonder how long the Web can survive the onslaught of the AI scrapers.

We’ve already discussed the existence of tools which can limit the scrapers, most notably Anubis. They definitely make things more difficult but it’s hardly impossible.

Of course, what I uttered was just a thought what I think could happen, and not a prediction of what likely will happen.

[Raphael]

In any case, good to keep you!

[WeepingElf]

Thank you all for the encouraging words you had for me!

[jcb]

The board was giving me a 429 "Too Many Requests" error last night.

[Man in Space]

It’s been giving me a few headaches as well.

[zompist]

The board was giving me a 429 "Too Many Requests" error last night.

Me too, I couldn't get on all day. I assume it's all AI scrapers, a problem for many websites.

[Raphael]

Pedant report: I just found a bad link on this old Bob page: https://zompist.com/bob9.html It's the link that should go to Bone in the entry for The King of Persia. (OK, technically a zompist.com error rather than a Board error.)

[malloc]

The board was giving me a 429 "Too Many Requests" error last night.

Me too, I couldn't get on all day. I assume it's all AI scrapers, a problem for many websites.

Any progress on finding some way to protect the site from them? What are other sites doing to avoid this problem?

[Raphael]

What are other sites doing to avoid this problem?

Things that coast more money than zompist can afford?

[Ketsuban]

I know X has talked about how they handle LLM scrapers on TCRF (wikis, like forums, are particularly vulnerable to being DDoSed by LLM scrapers because they're rude and indiscriminately access pages that aren't cached and so need to thrash the database) by simply blocking the entirety of the cloud providers which are hosting them. (They have some measures for dealing with malicious DDoSes as well, which TCRF also has to deal with because some people have extremely normal reactions to the fact it's openly queer.)

The other option is a challenge screen. CloudFlare will of course provide one as part of their protection racket services, or you can use something like Anubis. This is lower-effort, but it sucks because it means presenting a challenge to everyone, including legitimate visitors (and possibly giving money to CloudFlare to protect you from a problem they created) so X isn't a fan.

[WeepingElf]

I don't know what to do about it, but obviously something must be done soon, or I'm out.

[bradrn]

The other option is a challenge screen. CloudFlare will of course provide one as part of their protection racket services, or you can use something like Anubis. This is lower-effort, but it sucks because it means presenting a challenge to everyone, including legitimate visitors (and possibly giving money to CloudFlare to protect you from a problem they created) so X isn't a fan.

As it happens, the creator of Anubis (Xe Iaso) recently joined a Discord server I’m on and has been helpful in answering questions. If zompist would like me to inquire I’d be happy to do so.

(It doesn’t necessarily mean ‘presenting a challenge to everyone’, mind you. The way it works is that it presents a challenge on first request, then caches your identity in a cookie. If you have the cookie you don’t need to pass the challenge again, but you get rate-limited based on your identity. If a scraping server has cookies disabled then it sees the challenge each time, which slows it down anyway.)

[malloc]

I don't know what to do about it, but obviously something must be done soon, or I'm out.

Quite. I have been a loyal member of this site for over two decades at this point and I really hate to see it collapse. Despite my growing frustration, I am doing my best to take people's word for it that this problem has no simple or affordable solution. Given the growing consensus that AI scrapers are causing the problem, I think this vindicates my prescient hostility to AI.

[alice]

I couldn't get in yesterday because I'd apparently made too many requests (two or three). I'm still recovering.

I don't know what to do about it, but obviously something must be done soon, or I'm out.

That's a little bit drastic, isn't it?

Quite. I have been a loyal member of this site for over two decades at this point and I really hate to see it collapse. Despite my growing frustration, I am doing my best to take people's word for it that this problem has no simple or affordable solution. Given the growing consensus that AI scrapers are causing the problem, I think this vindicates my prescient hostility to AI.

You are not alone in your hostility to AI, as I hope you're aware. Nor do I want the ZBB to collapse.

[zompist]

I don't know what to do about it, but obviously something must be done soon, or I'm out.

Quite. I have been a loyal member of this site for over two decades at this point and I really hate to see it collapse. Despite my growing frustration, I am doing my best to take people's word for it that this problem has no simple or affordable solution. Given the growing consensus that AI scrapers are causing the problem, I think this vindicates my prescient hostility to AI.

If these sort of comments are intended to motivate me to put more effort and money into the board, they're doing the opposite.

Much bigger and better-resourced sites than mine are struggling with declining visitors and increasing scraper attacks. In a better world website hosts would handle the second problem for us, but they want to charge more instead. (Bluehost offers something but it looks like it's virus scanning.)

I went to bed last night thinking it was time to wind down the board. Today I'm not so sure. I did look at how much the site costs... thanks to auto-renewal, I hadn't realized the total. I have three hosting companies and I'm going to have to figure out how to get rid of one (certainly incatena.org; I'll have to move Almeopedia too).

I looked at the Anubis site and can't make head nor tails of it. Brad has done admin work before for the board, I don't know if he'd be willing to figure it out.

[bradrn]

I looked at the Anubis site and can't make head nor tails of it. Brad has done admin work before for the board, I don't know if he'd be willing to figure it out.

Sure, I’d be very happy to help! (And not just because I’m selfishly affected by the errors too…)

As for ‘not making heads nor tails of it’: the basic idea is that it’s a layer which sits in between the reverse proxy (which connects incoming HTTP(S) requests to the right places) and the web app (which actually handles said requests; in our case phpBB). It filters out requests which look like they’re coming from bots and rejects them before they can interfere with the app.

(Or, of course, you could just use Cloudflare for the same purpose. But I know lots of people dislike that.)

EDIT: just remind me again which hosting company this board is with?

[zompist]

Sure, I’d be very happy to help! (And not just because I’m selfishly affected by the errors too…)

Thank you!

As for ‘not making heads nor tails of it’: the basic idea is that it’s a layer which sits in between the reverse proxy (which connects incoming HTTP(S) requests to the right places) and the web app (which actually handles said requests, in our case phpBB). It filters out requests which look like they’re coming from bots and rejects them before they can interfere with the app.

Yeah, I know what it is, I don't know how you actually install it, much less configure it.

EDIT: just remind me again which hosting company this board is with?

Bluehost