Made In China Talk  

Go Back   Made In China Talk > International Business Talk > International Business Talk
Directory Register FAQ Members List Calendar Mark Forums Read 中文

How to Use robot text?

Thread Tools Display Modes
Old 05-16-2017, 10:30 PM
maxsmith maxsmith is offline
Join Date: Apr 2016
Posts: 56
Feedback: (0)
Default How to Use robot text?

Hello friends tell me
How to Use robot text?
Reply With Quote

Old 05-16-2018, 04:24 PM
damponting44 damponting44 is offline
Senior Member
Join Date: Jan 2017
Posts: 866
Feedback: (0)

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Following way to use Robot.txt file:

1.Allow access to all but a single crawler
User-agent: Unnecessarybot
Disallow: /
User-agent: *
Allow: /

2.Disallow crawling of the entire website. Keep in mind that in some situations URLs from the website may still be indexed, even if they haven't been crawled. Note: this does not match the various AdsBot crawlers, which must be named explicitly.
User-agent: *
Disallow: /

3.Block a specific image from Google Images:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

| Business Forum | Toronto Chinese

All times are GMT +9. The time now is 01:53 AM.

All rights reserved. Copyright 2007 Made In China Talk