[MusoFreak200]

G'day,

I am having difficulty with Search engine crawlers which are currently retrieving results which display not only legitimate pages, but also directories such as this:
http://cutmoreproperty.com/error_pages/

I am seeing this as an index of pages and files. this is not what i want but i have tried using robots.txt and i have been told that i can use my htaccess file will do the correct job.

can anyone please show me how to do this / post links for assistance

cheers.
any and all help would be very much appreciated

[d3iti]

Mate user, with a little lateral thinking, wrote how use robots.txt in 000webhost (.txt file are not working) in this thread.

I hope that it helps you.

[Waleed]

Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots.

In general, we prefer that our webpages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Like the personal images folder, website administration folder, customer’s test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don’t want them to be indexed.

Try this for creating a robots.txt file ,
http://www.yellowpipe.com/yis/tools/robots.txt/

[MusoFreak200]

What i meant is how it shows the actual list of files

[Mate]

The .htaccess directive to prevent indexing is

Options -Indexes