Google's Prashanth Koppula wrote a ready-to-bookmark post over at the official Webmaster Tools blog, showing tons of different robots-exclusion protocol (REP) directives that can be implemented in various ways. Following is a listing of directives discussed and the methods of implementation:
Home » All posts
The Difference Between Crawling and Indexing
The terms crawling and indexing (and indexing's cousin, caching) are frequently used together, but you should not consider them synonyms.
Exact definitions probably differ from person to person, but following is how I explain the processes:
Crawling is the process of an engine requesting -- and successfully downloading -- a unique URL. Obstacles to crawling include no links to a URL, server downtime, robots exclusion, or using links (such as some JavaScript links) from which bots cannot find a valid URL.
Exact definitions probably differ from person to person, but following is how I explain the processes:
Crawling is the process of an engine requesting -- and successfully downloading -- a unique URL. Obstacles to crawling include no links to a URL, server downtime, robots exclusion, or using links (such as some JavaScript links) from which bots cannot find a valid URL.
Link Development Vocabulary
There are a lot of misconceptions out there about link development. A lot. So, it's probably a good idea to start with some basics and clarify some of the lingo we use in the industry.
How to Evaluate a Link Development Firm
Welcome to my world! You understand it's critical to your optimization efforts and you're evaluating a firm that offers link development services. For many in the industry, link development is the thorn in the side. Very few people actually like doing it (present company excluded) and I've seen that some companies approach to the process is indicative of how they feel about it.
