Wednesday, November 05, 2025

Web Crawler Meta

For the last 10 days I've been observing somewhat odd web traffic at trvth.org.  First I noticed that traffic was significantly higher than usual, with a few days of 300 to 500 page views, instead of the usual 70 to 100.  The next couple of days had more than 1000 page views each, and I started to look into it.

The culprit appears to be a bot running on Tencent Cloud Computing, connecting out of Singapore.  I suspect it is consuming trvth.org as training material for an AI.

The connections come every 1 to 2 minutes and last less than 5 seconds.  It started by reading the pages by year and is following every link, though not in any recognizable order, and the queries come from a range of IP addresses, not a single address.

I'm not sure how I should feel about all that.

No comments: