Yeah, that's a problem, and I see that @live627 has already submitted a pair of PRs with a simple change to address it.

However, there is a reason why the ?msg=<id_msg> URLs exist. Since topics can be split or merged, the value for topic in ?topic=<id_topic>.msg<id_msg>#msg<id_msg> can change, causing previous links to that individual post to break. The ?msg=<id_msg> form ensures that a link to a specific post always resolves to the correct canonical URL.

In light of that, I think we need a more nuanced solution to this problem.

My immediate thought is to use some conditional logic to serve the long form to robots and the short form to humans, but perhaps someone can think of a more clever idea...

In that case, SMF should ignore the topic & just use the msg. I.e., continue to use the existing msg oriented logic.

No need to get complex.

Well, the ?topic=... URL is indeed the canonical form, since single posts are always shown within a topic. That's why we redirect from ?msg=... to ?topic=... in the first place.

Honestly, what should happen is that Google should respect nofollow. We're doing the correct thing, whereas they are not. But since we can't control what they do, we need to do something clever ourselves.

Hm. I wonder whether it would help if we sent a 301 or 308 response code instead of a 302.

When we see topic=<id_topic>.msg<id_msg>#msg<id_msg>, we can just use the msg & ignore the topic. Kinda simple, I think.

Not really. As I think about it, I don't think that using ?topic=<id_topic>.msg<id_msg>#msg<id_msg> would actually help with the original problem anyway. Search engines will ignore the fragment, but since the message ID is in the query string, each post will still have a unique URL that ultimately leads back to the same page. So the search engines will still follow all 25 links. It doesn't really matter what form the URLs are presented in. What matters is that the URLs contain the posts' ID values, yet all resolve to the same page in the end.

The only way to stop the search engines from making 25 pointless extra queries is to not include the post ID in the URL query params. We could use something like ?topic=<id_topic>.<page_start>#msg<id_msg>, since that would confine the post ID only to the fragment. But then we are back to the problem that any existing links to an individual post will break if its topic changes.

Is it really such a problem if the search engine is making these unnecessary queries to the server? Yes, it's inefficient, but is it an inefficiency that makes a practical difference? Do we have evidence that this causes notable performance degradations or other issues?

YES!!!

https://www.simplemachines.org/community/index.php?msg=4181475

Then we should use conditional logic to simply not show robots any links for individual posts, while still showing them to humans. Problem solved.

Yep that would work. Only (minor) hole in that is not all bots identify with useragents. Google Ireland keeps crawling me with a plain browser useragent... (The GoogleOther useragent is a true PITA - I've blocked it outright...)

On high MySQL CPU days, I may see thousands of pairs of these entries in my access logs from Google:

Using ugc instead of nofollow is also a consideration... Note their definition of what 'nofollow' means keeps changing:
https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify

Note that Google does honor disallows in robots.txt (though it ignores crawl-delay). I am experimenting with a disallow for PHPSESSID and msg= and early indications are that it's working great...

And oh yeah, by the way, Mele Kalikimaka!

Shall we ship a basic robots.txt in the final built package with a few starter rules?

When we see topic=<id_topic>.msg<id_msg>#msg<id_msg>, we can just use the msg & ignore the topic. Kinda simple, I think.

The reason we do "topic=X.msgY" is that it tells SMF to start on whichever page of the topic the specified post would appear on according to forum and/or user settings, since admins and users can choose how many posts to display per page. We need the topic ID in order to determine which posts to display, and there's no point in looking it up if we already have it in the URL - one less subquery to deal with.

Shall we ship a basic robots.txt in the final built package with a few starter rules?

We wouldn't want to risk overwriting any existing robots.txt file.

However, we could add some code that would try to append some rules to an existing robots.txt, possibly creating the file if necessary. This could only be a best effort attempt, though, because the file might be outside of the forum directory and we might not have file permissions, etc.

We could ship one in the install package at least. For the upgrade package we could just name it something else like "smf-robots.txt" and tell users to rename it if they want to use it.

#8382 provides SMF with the ability to add rules to robots.txt. Currently, it adds rules to tell all spiders that they should ignore URLs that match the pattern /path/to/index.php?msg= (where /path/to/index.php will be set to the appropriate value for the individual forum instance), as well as URLs containing PHPSESSID or ;topicseen.