News: September 21st, 2019, The Tinyportal team is pleased to announce the release of ~~ TinyPortal 1.6.4! GET IT NOW!!!

Login  |  Register
HTML5 Icon HTML5 Icon HTML5 Icon
TP on Social Media
Welcome, Guest. Please login or register.
Did you miss your activation email?

December 10, 2019, 03:14:59 PM

Login with username, password and session length

Recent

Members
Stats
  • Total Posts: 189279
  • Total Topics: 20775
  • Online Today: 386
  • Online Ever: 629
  • (November 08, 2018, 01:36:54 PM)
Users Online
Users: 0
Guests: 48
Total: 48

Author Topic: Robots meta tag and noindex  (Read 6805 times)

0 Members and 1 Guest are viewing this topic.

Offline Putoguiri

  • Jr. Member
  • **
  • Posts: 11
Robots meta tag and noindex
« on: October 29, 2012, 11:49:10 AM »
As you may or may not know, SMF nowadays adds a robots meta tag with the value "noindex" for pages it considers "duplicates" such a mobile/wap versions, etc.

This is a well made feature and really useful in order not to spam the indexers and for our bandwidth.

But, for us TP users this is a problem, since for smf index.php (main site) and index.php?action=forum (forum view) is the same, the forum view gets a noindex tag! This means for forums running TP, the main forum view NEVER gets indexed by crawlers.


I don't know if this is known or if there is a fix for it but I think this is definitely not the behavior I would expect it to have.

Offline IchBin™

  • Developer
  • *
  • Posts: 16228
    • My Website
Re: Robots meta tag and noindex
« Reply #1 on: October 29, 2012, 01:22:14 PM »
I have no clue when it comes to robot stuff. But I do wonder how you think that index.php is the same as index.php?action=forum Those are different pages right? If you want me to understand the problem though, you'll have to get into some good details so I know what you are talking about.

Offline Putoguiri

  • Jr. Member
  • **
  • Posts: 11
Re: Robots meta tag and noindex
« Reply #2 on: October 29, 2012, 02:30:27 PM »
Yes, for us who use TP index.php != index.php?action=forum but for a normal SMF user those are the same.

And exactly that's the problem.

From the index.template.php

Code: [Select]
   
// Please don't index these Mr Robot.
   if (!empty($context['robot_no_index']))
      echo '
   <meta name="robots" content="noindex" />';
 

Since for the forum (plain smf) both are the same site, the meta tag will be shown on index.php?action=forum but not for index.php. Or atleast those are my conclusions after observing this behaviour.


As for the robot stuff, it's a simple tag in order to prevent crawlers from indexing that site. SMF does this in order to prevent the same site being indexed multiple times (I checked on their forums).

A quick fix would be excluding index.php?action=forum from the "noindex" list, wherever this is located. But I think TP should do this by default on installation.

Right now, the forum index page won't be indexed/crawled by bots with this configuration.

Edit: I've checked the source code for tinyportal.net, the same thing happens, the main forum view has the noindex tag active.


Offline IchBin™

  • Developer
  • *
  • Posts: 16228
    • My Website
Re: Robots meta tag and noindex
« Reply #3 on: October 29, 2012, 06:08:56 PM »
Ok, so to make sure I understand you. I understand what content="noindex" is for. But you are saying that TP's frontpage does not get indexed. And this is most likely because TP does not set anything for these types of things. It does not modify the code for SMF for any crawler options.

You can possibly add to the TPortal.php file in the doTPfrontpage() function a piece of code to prevent it if you want to test it. It probably depends on when things are loaded, and at the moment I do not know without looking deep into the code.

Just add this line in the function right after the first return; you see.
$context['robot_no_index'] = '';

Offline Putoguiri

  • Jr. Member
  • **
  • Posts: 11
Re: Robots meta tag and noindex
« Reply #4 on: October 30, 2012, 03:47:43 AM »
No, the TP frontpage gets indexed just fine, the one that doesn't get indexed is the main forum view. (Which, without TP was previously the output of calling index.php without any parameters)

Offline IchBin™

  • Developer
  • *
  • Posts: 16228
    • My Website
Re: Robots meta tag and noindex
« Reply #5 on: October 30, 2012, 01:59:47 PM »
SMF checks for any $_GET parameters in the URL on board index. If there is any, it will set robot_no_index to true. I do not want to modify any SMF code, as I'm trying to remove as many edits from the SMF files as possible. If you have any ideas I am all ears.

Offline Putoguiri

  • Jr. Member
  • **
  • Posts: 11
Re: Robots meta tag and noindex
« Reply #6 on: October 31, 2012, 04:42:51 AM »
By no means am I telling you what to do, what to code or how to code it.

I just posted here what I think is a bug or an improper behaviour. I told you what happens, what is expected to happen and even why I think this error/bug happens. For you it is to decide if it really is and if it needs fixing.

So please don't take anything I posted as an offense because it wasn't meant that way.


Edit:

In my forums I added something like:

Code: [Select]
if (isset($_GET['action'])) {
if($_GET['action']=='forum'){
$context['robot_no_index'] = '';
}
}

to the index.template.php as a quick fix.

Offline IchBin™

  • Developer
  • *
  • Posts: 16228
    • My Website
Re: Robots meta tag and noindex
« Reply #7 on: October 31, 2012, 08:16:14 AM »
I know you weren't telling me how to code or anything. I was sincerely asking if you had any ideas on how to fix it, in particular without editing SMF code. Of course you can modify it like you did, and that works fine for a work around. Just putting it out there to see if anyone can think of something I haven't is all.

Offline Putoguiri

  • Jr. Member
  • **
  • Posts: 11
Re: Robots meta tag and noindex
« Reply #8 on: October 31, 2012, 09:22:28 AM »
The only way to fix it would be editing smf files, I guess. Either editing the template as a quick fix or searching for the "blacklist", the list of duplicate sites smf has to determine which have to be noindexed or not. Both involve editing smf files.

But, from what I read on smf's forums, it's a matter they don't want the user to touch.

Offline IchBin™

  • Developer
  • *
  • Posts: 16228
    • My Website
Re: Robots meta tag and noindex
« Reply #9 on: October 31, 2012, 09:34:03 AM »
It will probably have to wait until SMF 2.1 as I think there are hooks in there that load much later, and could possibly use one of the hooks to set the robots index parameter at that point. In the mean time, people can just use your work around if they are concerned with it. Thanks for bringing attention to this.