Search Engine Optimization- SEO Class -5 Topics Covered: Advance bolgging Robots Txt Learning what a Sitemap is Generating & submitting a sitemap in for Blogger blogs using Google Webmasters Site Verification
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Search Engine Optimization- SEO Class -2
Search Engine Optimization- SEOClass -5
Topics Covered: Advance bolgging Robots Txt Learning what a Sitemap is Generating & submitting a sitemap in for Blogger blogs using Google Webmasters Site Verification
1
What is Robots.txt?
Robots.txt is a text file which contains few lines of simple code. It is saved on the website or blogs server which instruct the web crawlers how to index and crawl your blog in the search results. That means you can restrict any web page on your blog from web crawlers so that it cant get indexed in search engines, like your blog labels page, your demo page or any other pages that are not as important to get indexed. Always remember that search crawlers scan the robots.txt file before crawling any web page.
In Blogger/Blogspot Robots.txt is known as Custom Robots.txt that means now you can customize this file according to your choice.
Robots.Txtrobots.txt , crawl crawl robots.txt .
robots.txt crawl robots.txt
robots.txt , (Robots ExclusionProtocol) (Robots Exclusion Standard) .
Robots.txt Protocol - Standard Syntax & Semantics
/
1. User-agent:
2. *
3. disallow:
4. #
()
2. Wildcard. User-agent: *
disallow: / URL path path path disallow allow
4.
Each blog hosted on blogger have its default robots.txt file which is something that looks like this:User-agent: Mediapartners-GoogleDisallow: User-agent: * Disallow: /search Allow: / Sitemap: http://allhotnewz.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
User-agent: *This is for all robots marked with asterisk (*). In default settings our blogs labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.
Disallow: /search
That means the links having keyword searchjust after the domain name will be ignored.
See below example which is a link of label page named HoT Gossip
http://allhotnewz.blogspot.com/search?q=hot+gossip
And if we remove Disallow: /searchfrom the above code then crawlers will access our entire blog to index and crawl all of its content and web pages.
Here Allow: /refers to the Homepage that means web crawlers can crawl and index our blogs homepage.
User-agent: Mediapartners-GoogleThis code is for Google Adsense robots which help them to serve better ads on your blog. Either you are using Google Adsense on your blog or not simply leave it as it is.
Disallow Particular Page
If we need to disallow a particular page then we can use the same method as above. Simply copy the page URL and remove blog address from it which will something look like this:
Disallow: /p/page-url.html
For example- http://getseoanswer.blogspot.com/p/privacy-policy.html
Disallow: /p/ privacy-policy.html
Setting UP Robot Txt For Blogger/Blogspot Blog
1. Log in to your blogger account.2. Now navigate to Setting >> Search Preferences
Click on edit link under the Custom Robots Header Tags section as shown in screenshot.
Once you click on the edit link you will see many options. Simply tick on the options as shown in the image.
Now click on Save changes button.
You are done!
You must learn about tags & when should you use them.
all: There are no restrictions for indexing or serving. This is default for all pages
noindex: Do not show this page in search results and do not show a "Cached" link in search results.
nofollow: Do not follow the links on this page
none:Equivalent to noindex, nofollow
noarchive: Do not show a "Cached" link in search results.
nosnippet:Do not show a snippet in the search results for this page
noodp: Do not use metadata from the Open Directory project (DMOZ) for titles or snippets shown for this page.
notranslate: Do not offer translation of this page in other languages in search results.
noimageindex: Do not index images on this page.
unavailable_after: [RFC-850 date/time]:Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format. Example: 17 May 2012 15:00:00 PST
SUMMARY:
Adding Custom Robots.Txt to Blogger
Now the main part is how to add custom robots.txt in blogger. So below are steps to add it.
Go to your blogger blog.
Navigate to Settings >>Search Preferences Crawlers and indexing Custom robots.txt Edit Yes
Now paste your robots.txt file code in the box.
Click on Save Changes button.
You are done!
How to Check Your Robots.txt File?
You can check this file on your blog by adding /robots.txtat last to your blog URL in the browser. Take a look at the below example for demo.
http://towfiqularafat.blogspot.com/robots.txt
Robots.txt File Generator
http://tools.seobook.com/robots-txt/generator/
*** Create in notepad robots.txt (File Name) and upload it to public_html in CPANEL
Sitemap
Asitemapis a file where you can list the web pages of your site to tell Google and other search engines about the organization of your site content. Search engine web crawlers like Googlebot read this file to more intelligently crawl your site.
Free XML Sitemap generator
Google search Console
Generate Sitemap using google search consoleImage-1
Image-2
Site Verification
Project Co-ordinatorM. Towfiqul Arafatwww.towfiqularafat.com
Related Documents
