[costyn1]

I have a problem using robots.txt.
I want to hide /referrer/ , it should not be in the search engines.
So I wrote this robots.txt file:

User-agent: *
Disallow: /referrer/
Allow: /

The problem is that Google reads the following lines:


User-agent: *
Disallow: /referrer/
Allow: /


<!-- www.000webhost.com Analytics Code -->
<script type="text/javascript" src="http://analytics.hosting24.com/count.php"></script>
<noscript><a href="http://www.hosting24.com/"><img src="http://analytics.hosting24.com/count.php" alt="web hosting" /></a></noscript>
<!-- End Of Analytics Code -->

So Google "thinks" the file is not valid.
I removed the "Analytics Code" using the CPANEL, infact the code doesn't appear in any of my html pages.
Why does it appears when Google reads the robots.txt files?
This is a really really big problem.
Sites with a bad robots.txt file could be banned from search engines.
Could you help me?

[lark]

go to http://members.000webhost.com/analytics.php to turn off analytic code.

[costyn1]

I tried but the problem is still present.

[lark]

refresh the page to make sure you are looking at a non-cached version.

Also I'm not sure whether the CP has an option for analytic code, but just search through there in case.

gl.

[sizzlefire]

What is your website?

[Araneda]

The code is not disabled if Google crawls robots.txt.
Perhaps the only solution is to redirect "robots.txt" to a script.

[lark]

I had a similar problem, does your robots.txt file display like the following:

User-agent: * Disallow: /referrer/ Allow: /

instead of:

User-agent: *
Disallow: /referrer/
Allow: /

If so check the robots.txt content type. (if you are using mozilla firefox right click and choose 'View page info' and the type of the file should be next to 'Type:'

My file was being read as text/html which doesn't work. It should be read as text/plain

Here's what I did to fix it..

If you do not already have a .htaccess file, make one in the public_html folder. (The same folder that your main index file goes into.)
And the file is named ".htaccess" there is nothing before the dot.

Inside that file add these lines..

.htaccess file:

<FilesMatch "robots.txt">
 SetHandler application/x-httpd-php
</FilesMatch>

This will make your robots.txt file be read by the php parser.

Now inside your robots.txt file add the following..

PHP Code:

<?php
header("Content-type: text/plain");
?>
(robots.txt file goes underneath this. eg)
User-agent: *
Disallow: /referrer/
Allow: /

Hope this helped.


edit: Just did this for a new site I set up. The above is necessary for it to be read as text and interpreted as a robots.txt file but not sure if robots.txt will still work if the analytic code is still below it.
All I can say is that I went to members.000webhost.com/analytics.php and followed the instructions.

Make sure you type in your site's address without http:// or www.
For example: mysite.site11.net <-- that is all you should put in, obviously replacing it for your sites address..

Make sure the drop down option is on 'Disable analytics code', NOT enable.

And obviously make sure that your password is correct.

Maybe post your site's address here so others can have a look.
And if nothing here works, submit a support ticket.

[Araneda]

The problem is the content type. The same applies to the XML file.
Add the following line in the .htaccess file:

AddType text/plain .txt

[KnK]

once you have disabled analytic code add thi to your .htaccess file

php_value auto_append_file none
php_value auto_prepend_file none

[Araneda]

In the case of the robots.txt file simply add the correct MIME type:

AddType text/plain .txt

Anyway, KnK is right. Running phpinfo() script you'll see:

auto_append_file /usr/lib/php/foot.php
auto_globals_jit On
auto_prepend_file /usr/lib/php/head.php

So, to disable the code injection in scripts you need to add the php_value directive