If you go to google and search for CNN (http://www.google.com/search?hl=en&q=cnn&btnG=Search)
How do you get some of your site's categories listed like World, US News, etc like CNN has?
Thanks all!
Full Version: How to get your site listed like this....
Google has CNN indexed and crawled very specifically. CNN.com has been set up to be listed in search engines in that way. Plus CNN is also a damn popular site so google is going to have a bit more detail on them then other sites.
I suggest the use of a robots.txt file and giving the crawler some good content to crawl through.
I suggest the use of a robots.txt file and giving the crawler some good content to crawl through.
QUOTE(raidsilver @ Mar 31 2007, 12:51 PM) 
Google has CNN indexed and crawled very specifically. CNN.com has been set up to be listed in search engines in that way. Plus CNN is also a damn popular site so google is going to have a bit more detail on them then other sites.
I suggest the use of a robots.txt file and giving the crawler some good content to crawl through.
I suggest the use of a robots.txt file and giving the crawler some good content to crawl through.
Appreciate it the reply and suggestion. Been waiting for a few days on this one. Appreciate it
Question: How would you implement the "robots.txt" thing exactly ?
Would you just place that as a link in your coding?
Would you just place that as a link in your coding?
Having links like domainname.com/world_news/ helps.
Page titles that reinforce the structure helps too/
Page titles that reinforce the structure helps too/
Hmm, if you search "ehko.net" (my domain), you'll see that the beta page is listed like that. I don't have a robots.txt file, so I don't think it's required.
How can you tell google to crawl something from robots. I mean.. i do know how to tell it NOT to but didn't know you could actually tell it which URLs to give preference to.
that type of listing in googlespeak is called a "SiteLink".
This is what Google says about sitelinks:
This is what Google says about sitelinks:
QUOTE
Sitelinks are additional links Google sometimes generates from site contents in order to help users navigate your site. Google generates these sitelinks periodically from your site's contents.
Because we generate sitelinks dynamically, this list can change from time to time.
Sitelinks are completely automated, and we show them only if we think they'll be useful to the user. If your site's structure doesn't allow our algorithms to find good sitelinks, or we don't think that the sitelinks are relevant to the user's query, we won't show them. However, we are always working to improve how we find and display sitelinks.
Because we generate sitelinks dynamically, this list can change from time to time.
Sitelinks are completely automated, and we show them only if we think they'll be useful to the user. If your site's structure doesn't allow our algorithms to find good sitelinks, or we don't think that the sitelinks are relevant to the user's query, we won't show them. However, we are always working to improve how we find and display sitelinks.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.