Robots and Sitemap By Naji El Kotob. Information Technology Consultant _______________________________ naji [@] DotNETHeroes.com Review 1.0.1 - 13 . Business Intelligence Group BIG SARL
May 13, 2015
Robots and Sitemap
By Naji El Kotob.
Information Technology Consultant
_______________________________
naji [@] DotNETHeroes.com
Review 1.0.1 - 13
.
Business Intelligence Group BIG SARL
by Naji El Kotob
Outlines
SEO
Robots.txt
Sitemap.xml
Q&A
by Naji El Kotob
SEO
by Naji El Kotob
What is SEO
SEO is short for search engine optimization or search engine optimizer.
Search engine optimization is a methodology of strategies, techniques and tactics used to increase the amount of visitors to a website by obtaining a high-ranking placement in the search results page of a search engine (SERP)
-- including Google, Bing, Yahoo and other search engines.
Source | http://www.webopedia.com/TERM/S/SEO.html
by Naji El Kotob
Robots.txt
by Naji El Kotob
Robots.txt
Robots.txt is the file that Search Engines use to see what they should index.
This file and site maps help make your site easier to navigate by the bots and let them know what is legal and what you would rather not have the published in their engines.
by Naji El Kotob
Robots.txt
by Naji El Kotob
Robots.txt Syntax
User-agent: *
Disallow:
Sitemap: http://dotnetheroes.com/sitemap.xml
by Naji El Kotob
Robots.txt Generator
http://tools.seobook.com/robots-txt/generator/
by Naji El Kotob
Interactive Demo
by Naji El Kotob
Demo
Design and Create a Dynamic RobotsSource | http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/
Adding a Dynamic Robots.txt file to an ASP.NET MVC siteRobots.txt is required to allow search engines to properly index your site, and more importantly not index it.
If you have a public-facing staging or preliminary site that you don’t want to show up in Google results, you need to make sure that it returns the correct robots.txt with the
Disallow: /
line to prevent indexing. However, manually adding robots.txt files to staging and production environments as a manual process can be improved with the process below – the same code can serve up a locked down robots.txt in staging or internal URLs, and allow indexing in production.
by Naji El Kotob
Demo (cont'd)
You’ll also need to make sure that runAllManagedModulesForAllRequests is true in web.config as normally text files bypass the ASP.NET pipeline:
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"></modules>
...
</system.webServer>
by Naji El Kotob
Demo (cont'd)
routes.MapRoute("Robots.txt","robots.txt",
new{
controller = "Robots",action = "RobotsText"
});
by Naji El Kotob
Demo (cont'd)
The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will go here:
public class RobotsController : Controller
{
public FileContentResult RobotsText()
{
var contentBuilder = new StringBuilder();
contentBuilder.AppendLine("User-agent: *");
// change this to however you want to detect a production URL
var isProductionUrl = Request.Url != null && !Request.Url.ToString().ToLowerInvariant().Contains("elasticbeanstalk");
if (isProductionUrl)
{
contentBuilder.AppendLine("Disallow: /elmah.axd");
contentBuilder.AppendLine("Disallow: /admin");
contentBuilder.AppendLine("Disallow: /Admin");
contentBuilder.AppendLine("Sitemap: http://www.mysite.com/sitemap.xml");
}
else
{
contentBuilder.AppendLine("Disallow: /");
}
return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");
}
}
by Naji El Kotob
Sitemap
by Naji El Kotob
Sitemap
Sitemap (XML) is a list of pages on your Website. It is easiest way to give Google, and other search engines, information about your site and may speed up Google’s crawlers finding you.
There are many advantages of submitting a Sitemap to Google, especially if your site is new and doesn’t have many backlinks.
Source | http://diywebjem.com/webdefinitions.html
by Naji El Kotob
Sitemap Format
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
Source | http://www.sitemaps.org/protocol.html
by Naji El Kotob
Sitemap Generators
There are hundreds of online tools that can be used to create an XML sitemap that can be submitted to Google, Bing, Yahoo and other◦ E.g.
http://www.xml-sitemaps.com/
http://www.sitemapdoc.com/
And more…
by Naji El Kotob
References
http://geekswithblogs.net/jjulian/archive/2012/04/25/adding-robots.txt-to-your-asp.net-mvc-3-applications.aspx
http://tools.seobook.com/robots-txt/
http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/
by Naji El Kotob
QnA?
by Naji El Kotob
Thank You
Please send your feedback and suggestions to:◦ naji [@] DotNETHeroes.com
Join us at www.facebook.com/DotNETHeroes