SEO AJAX Crawlability in a Responsive Publisher World

AJAX in a responsive publishing world CRAWLABILITY

google.me/+EricWu

@eywu

linkedin.com/in/ericywu

github.com/eywu

AJAX is like violence if it doesn't solve

your problems, you’re not using

enough of it.

16ms

10kb

100ms SPEED,

PERFORMANCE,

HUMAN PERCEPTION

PAGES CRAWLED PER DAY

80% INCREASE

20,000

80,000

May 2013 – Aug 2013

80% INCREASE

ORGANIC SEARCH SESSIONS

600,000

1,200,000

May 2013 – Aug 2013

Mar 2004: “Googlebot/Test” External JS

Mar 2006: Googlebot Uses Onsite Live Chat

June 2010: Caffeine (Full Rollout)

Nov 2010: Instant Preview

May 2014: GWT Fetch & Render

May 2012: Matt PSA. Don’t Block JS & CSS

Oct 2009: AJAX Crawlability _escaped_fragment_

Nov 2007: Spider’s View on Web 2.0

May 2013: Matt Video. Googlebot & AJAX

SCROLL

http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html

THINK BASIC

PAGINATION

NAVIGATE / PAGINATE

NAVIGATE / PAGINATE

<a href="/page/2/">Load More</a>

LOAD MORE LINK

NAVIGATE / PAGINATE

<a href="/page/2/">Load More</a>

LOAD MORE LINK

NAVIGATE / PAGINATE

REL=NEXT / PREV

<link href="/page/10/" rel="prev" /> <link href="/page/11/" rel="canonical" /> <link href="/page/12/" rel="next" />

POINTER URLS*

example.com/?last-post=42

example.com/?time=1401424133

CONTINUOUS CONTENT

PUSHSTATE OR SHEBANG #! http://caniuse.com/#search=history

history.pushState( {}, titleOfPage, newURL

); history.replaceState();

AJAX GALLERIES

CONTINUOUS CONTENT

1.  PUSHSTATE

2.  VIEW-SOURCE

NEXT SLIDE LINKS

3.  REL=NEXT/ PREV

DEFERED IMAGE LOADING

I WISH CRAWLERS SUPPORTED

REL=CANONICAL HTTP HEADERS FOR IMAGES

80% DECREASE

IMAGE SEARCH SESSIONS

400,000

800,000

Nov 2012 – Mar 2013

LAZY LOADING •  1x1 “BLANKS”

•  SKELETON SCREENS

•  LOW RESOLUTION Z Z

Z

RESPONSIVE IMAGES •  <PICTURE>

•  SRCSET

•  POLYFILL

•  UA DETECTION

NOSCRIPT

<img data-src='<480:small.jpg, <960:medium.jpg, >960:large.jpg' />

<noscript> <img src='full.jpg' /> </noscript>

CRAWLABLE PARTIALS

CRAWLABLE PARTIALS LYRIC SEARCH LINKS TO http://rock.rapgenius.com/1884632/Pharrell-williams-happy/ It-might-seem-crazy-what-im-bout-to-say-sunshine-shes-here-you-can-take-a-break

CRAWLABLE PARTIALS JS ENABLED REDIRECTS http://rock.rapgenius.com/Pharrell-williams-happy-lyrics#note-1884632

CRAWLABLE PARTIALS JS DISABLED RENDERS http://rock.rapgenius.com/1884632/Pharrell-williams-happy/It-might-seem-crazy-what-im-bout-to-say-sunshine-shes-here-you-can-take-a-break

4X PAGES CRAWLED PER DAY

QUADRUPALED

75,000

300,000

May 2013 – Aug 2013

+1 MILLION

ORGANIC SEARCH SESSIONS

2,000,000

4,000,000

May 2013 – Aug 2013

Technical SEO Community

HelpDeskHangouts.com

SearchSignals.com

bit.ly/ajax-crawlability

APPENDIX

ARCHITECTURE PRERENDER SERVERSIDE VS

PRERENDER

PRERENDER

PRERENDER

SEO.js

CONS PROS • SINGLE VIEW TEMPLATE

• SINGLE ROUTING

• REVERSE PROXY CACHING

• _ESCAPED_FRAGMENT_

• INTERPOLATED SOURCE

SERVER SIDE

SERVER SIDE

SERVER SIDE

SERVER SIDE

SERVER SIDE

SERVER SIDE

SERVER SIDE RENDER

CONS PROS • CLEAN URLS*

• ALL REQUESTS ARE EQUAL

• TRUE DYNAMIC RESPONSE

• 2 ROUTERS

• 2 VIEW TEMPLATES*

Quote conflated from my favorite ruby XML parser » http://nokogiri.org/ Speed, Performance, and Human Perception » https://www.youtube.com/watch?v=7ubJzEi3HuA SERoundtable Timeline Links » http://www.seroundtable.com/google-javascript-webmaster-tools-18602.html Googlebot/Test External JS » http://www.seroundtable.com/archives/000236.html Googlebot Uses Onsite Live Chat » http://www.seroundtable.com/archives/003492.html Spider’s View on Web 2.0 » http://googlewebmastercentral.blogspot.com/2007/11/spiders-view-of-web-20.html AJAX Crawlability Proposal » http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html Caffine Rollout » http://googlewebmastercentral.blogspot.com/2010/06/our-new-search-index-caffeine.html Instant Previews »

http://googleblog.blogspot.com/2010/11/beyond-instant-results-instant-previews.html http://googlewebmastercentral.blogspot.com/2010/11/instant-previews.html

http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews Matt Cutts PSA: Don’t Block JS & CSS » http://www.seroundtable.com/googlebot-javascript-css-14930.html Matt Video: How Does Googlebot handle content loaded via AJAX? » https://www.youtube.com/watch?v=_6mtiwQ3nvw

REFERENCES

GWT Fetch & Render » http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html Google Blog: Infinite Scroll Recommendations & Example » http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html LA Times Reimagined by Code and Theory » http://www.codeandtheory.com/things-we-make/the-los-angeles-times-reimagined Google Blog: Specify your canonical » http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html Google Blog: Pagination with rel=“next” and rel=“prev” » http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html Google Blog: Video about Pagination » http://googlewebmastercentral.blogspot.com/2012/03/video-about-pagination-with-relnext-and.html One Page Wonder: Coverage on QZ » http://www.foliomag.com/2013/one-page-wonder-infinite-scroll The Next Web Redesign Coverage » http://www.niemanlab.org/2012/10/the-next-web-redesigns-to-be-more-app-like/ The Next Web Press Release » http://thenextweb.pr.co/010a893a11df2bb61d981b2b0607c1b6784a5ab07b5ab100790b2bb3168a35f8

REFERENCES

USA Today Redesign » http://blog.f-i.com/usatoday-com-redesigning-one-of-americas-most-popular-news-site/ http://designenvy.aiga.org/usa-today-website-redesign-fantasy-interactive/ http://www.businessinsider.com/usa-todays-homepage-redesigns-2012-9

Gawker 1 Year Later Success » http://thenextweb.com/insider/2012/02/02/remember-that-gawker-redesign-a-years-worth-of-data-says-it-worked/ http://www.businessinsider.com/nick-denton-loses-bet-that-the-gawker-redesign-wouldnt-hurt-traffic-2011-10 http://www.businessinsider.com/gawker-media-traffic-numbers-2011-4 Gawker Failed Coverage » http://www.catchmyfame.com/2013/05/02/how-gawker-sabotaged-their-own-network-with-a-horrible-new-layout/ http://www.theatlantic.com/technology/archive/2011/04/gawkers-traffic-numbers-are-worse-than-anyone-anticipated/237594/ http://www.webmonkey.com/2011/02/gawker-learns-the-hard-way-why-hash-bang-urls-are-evil/ Paul Irish to Matt Cutts Video » https://www.youtube.com/watch?v=yiAF9VdvRPw Google Developer Documentation on AJAX Crawlability » https://developers.google.com/webmasters/ajax-crawling/ Browser Compatibility Chart » http://caniuse.com/#search=history Breaking The Web With Hash Bangs » http://isolani.co.uk/blog/javascript/BreakingTheWebWithHashBangs

REFERENCES

Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog: A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun-and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s

REFERENCES

Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog: A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun-and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design Serious Angular SEO » http://www.ng-newsletter.com/posts/serious-angular-seo.html AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s

REFERENCES

Josh Kadis Quartz on VIP Wordpress Video » http://vip.wordpress.com/2013/09/26/josh-kadis-qz-wordpress/ https://docs.google.com/file/d/0B2Z4K6ynFLg5TVdvWVV1aTRmYUU/edit?pli=1

AirBNB: Our First Node.js App » http://nerds.airbnb.com/weve-launched-our-first-nodejs-app-to-product/ AirBNB: Rendr (Backbone in the Browser and Node) » http://nerds.airbnb.com/weve-open-sourced-rendr-run-your-backbonejs-a/ StackOverflow: PushState, Backbone, and Node » http://stackoverflow.com/questions/7098130/reusing-backbone-views-routes-on-the-server-when-using-backbone-js-pushstate-for Google: How do I create an HTML Snapshot (HIJAX) » https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot

REFERENCES