What To Do When Google Can't Understand Your JavaScript By Jody O'Donnell

#SMX #21A2 @gimpslice

What To Do When Google Can't

Understand Your JavaScript

Althoughthisisatemplate,theseslideshavebeencreatedasexamplestoshowyouwhat’spossiblewithinthetemplate.PLEASEDELETEALLEXAMPLESLIDESANDNOTESBEFORECREATINGORIMPORTINGYOUROWNDECK. YOU MUST USE THE SMX FOOTER ON YOUR TITLE SLIDE! Thetemplateworksbestifyouuseittocreateapresentationfromscratch.However,thisblanklayoutoffersyouflexibilitytoinsertyourownslidesabovetheSMXfooter.Youmustusethisfooteratonalmostallofyourslides.Ifascreenshotoverlapsthefooteronoccasion,that’sfine–butdoyourbesttocropappropriately!Beforeyoudoanythingelse,replace“#XXa”inthefooterwithyoursession’sspecifichashtag.InPowerPoint,SelectVIEW>SLIDEMASTERtoeditandincludeitonallslidesofyourpresentation.Alsoaddyour@twitterhandle(or@companyhandleifyouprefer).Askyoursessioncoordinatorforthesession’sspecifichashtag!Thistemplateishigh-resolution16:9.DONOTchangeitto4:3.Alsonotethatifyouimporta4:3presentationinto16:9youmayencounterdisplayissues.ThetemplateusesCenturyGothic&Corbelasdefaultfonts.Arialisasecondaryfontthatmayappear.NotethatolderversionsofPowerPointmaynothavetheseasanoption;changetoArialorotherstandardsans-serif-font.Wheninsertingtext,you*MUST*useonlystandardfonts.Wecannotguaranteeeventlaptopswillhavespecialtyfontsinstalled,norcanwedosoonsite.Powerpointswithanimationsmaynottranslatewelltopdf/slideshareformat–youmayalsosubmita2ndversion/scrubbeddeckmoreappropriateforpostingonlinebySMXproductionstaff.

Example Title Slide:

Althoughthisisatemplate,theseslideshavebeencreatedasexamplestoshowyouwhat’spossiblewithinthetemplate.PLEASEDELETEALLEXAMPLESLIDESANDNOTESBEFORECREATINGORIMPORTINGYOUROWNDECK. YOU MUST USE THE SMX FOOTER ON YOUR TITLE SLIDE! Thetemplateworksbestifyouuseittocreateapresentationfromscratch.However,thisblanklayoutoffersyouflexibilitytoinsertyourownslidesabovetheSMXfooter.Youmustusethisfooteratonalmostallofyourslides.Ifascreenshotoverlapsthefooteronoccasion,that’sfine–butdoyourbesttocropappropriately!Beforeyoudoanythingelse,here’showtoreplace“#XXa”inthefooterwithyoursession’sspecifichashtag.InPowerPoint,SelectVIEW>SLIDEMASTERtoeditSLIDE#1toincludeitonallslidesofyourpresentation.Alsoaddyour@twitterhandle(or@companyhandleifyouprefer).Askyoursessioncoordinatorforthesession’sspecifichashtag!Thistemplateishigh-resolution16:9.DONOTchangeitto4:3.Alsonotethatifyouimporta4:3presentationinto16:9youmayencounterdisplayissues.ThetemplateusesCenturyGothic&Corbelasdefaultfonts.Arialisasecondaryfontthatmayappear.NotethatolderversionsofPowerPointmaynothavetheseasanoption;changetoArialorotherstandardsans-serif-font.Wheninsertingtext,you*MUST*useonlystandardfonts.Wecannotguaranteeeventlaptopswillhavespecialtyfontsinstalled,norcanwedosoonsite.Powerpointswithanimationsmaynottranslatewelltopdf/slideshareformat–youmayalsosubmita2ndversion/scrubbeddeckmoreappropriateforpostingonlinebySMXproductionstaff.


JodyJ.O’DonnellSEOTechnicalDirectorMerkle,Inc


WebsitesneedtobeDiscoverableandCrawlableWebpagesneedtobeunderstoodthroughSEOsignalsWebpagesneedtheirContenttobeScoredcorrectlytobeIndexedWebsitesneedtobeIndexedproperlytoRank

Let’s Start with the Basics


WearenotgoingtotalkaboutBingexceptinthetraditionalsenseofreadingtheHTTPHeadersandtheHTMLSourceBingandtheothersearchenginesstillrequireHTMLsnapshots:§ HTMLSnapshotsandtherenderedDOMbyanormalbrowserengine(readbyGoogle)shouldbeidentical

Oh…Bing.


FirstthingyoudidwasturnJSoffinnewbrowsersJavaScriptwasn’ttheproblem,butittooktheblame.Lookin’atyouMSIE3-6

The Browser Wars


Itwasn’tthatJavaScriptgotbetter§ Microsoftdecidedtoplayball§ InternetExplorergotbetterToday,JavaScriptisoneofthemostpopularprogramminglanguagesintheworld

So…What Happened?


jQueryandMootoolscameoutinlessthanayearofeachotherOutoftheboxsolutionsforfree!Client-SideApplicationsthatusetraditionHTMLandCSSstatements

JavaScript Libraries


Developmentismuchfasterwhenyoureferenceafreelibraryratherthancreateiteachtime§ Browsercompatibilitywasabiggerdeal,too

ItworkedwithSEO!

What did this do for Us?


SEOAin’tAllThat§ Userexperienceismoreimportantthanever

§ Attentionspansareshort§ Wecan'tholdtechnologyhostagebecauseofSEO

At the End of the Day


LibrariesEvolvedtoentireframeworksFrameworkswereoutoftheboxsolutionsforcreatingapps§ Appscanbeonclientorserver§ TheframeworkscreatedoutoftheboxMVC’s

Programmersarenowfreetoconcentrateonfunctionality

JavaScript Frameworks


FromanSEOstandpoint,welookattheresult§ Ourexaminationisinthecode,notthebackendtoproduceit

Wearenotgoingtodiscussthedifferencesbetweentheframeworks§ Northeclient-sidevs.server-siderendering§ SPA’sandframeworksshouldbeabletoproduce1URLtoonepieceofuniquecontent

SPA’s and JavaScript Frameworks



TheybuiltahistoryAPIcallbetweentheCodeandthebrowserSpecifically,afunctioncalledpushState()Twoofthosevariables§  TitleTag§ URL

History API - pushState()



SEO’sneedtounderstandthedifferencebetween:

§ HTTPHeaders§ HTMLSource

§ HTMLSnapshot

§ RenderedDOM

Different Places to Look


Allsearchengineshavetwoorders§ HTTPHeaders–  x-robots-tag,link:canonical,link:hreflang

§ HTMLSource– metarobotstag,<link>canonical,<link>hreflang

SEO Signals by Order of Precedence


DOM – The Third Order


JavaScriptredirectsaretreatedinasimilarmanneras301redirects

What were the Results?


DynamicallyinsertedcontentistreatedinanequivalentwhetherintheHTMLsourceorintheDOM.§ ThisgoeswithSEOSignalsaswell

What were the Results?


Oneoftheteststhatfailed§ Therel=“nofollow”tagwascompletelyignoredintheDOM

WethinkthisisanOrderofPrecedenceproblem§ CrawlsignalsarepickedupstartingintheHTMLsource,lookingfortherel=”nofollow”signal

§ Essentially,adedupingmechanismmayberesponsible

One Discrepancy


SEOSignal:§ wewillpickitupanduseitthefirsttimeweseeit

Content:§ Moreofamixedbag,theycouldchoosewhichcontenttheyscoreandindex

SEO Signal vs. Content


Asasecondaryeffect,ourarticlegotpickedupinahackernewsthreadThesinglepiecethatcamefromthiswholediscussionwasasmallpartwithaself-purportedEx-GoogleEmployee§ Discussionaboutifgooglewouldwaitfor120secondsbeforetakingthesnapshotincaseofinjectedcontent

§ Googleengineerdidagreethattheydidanditwasafixedtime

Wasittrue?

Hackernews


Google Fetch & Render PageSpeed Insights

www.maxxeight.com/js-timer/


MostEcomsiteshavecategorypageswithhundredsofproductstolist

The Category Page Conundrum


Youhadthreechoices:§ View-AllPages§ PaginatedPages§ LazyLoading(whichdidn’thaveanSEOoption)

The Category Page Conundrum


InfinitescrollchangedwithHTML5’sHistoryAPINowwecantieaJSListeningHandlerwithapushState()call§ Youcanpush{{URL+?page=2}}intotheURLbar–  REMEMBER:Youcanupdatetitletaghere,too

Infinite Scroll + HTML5


SearchenginescanreferenceanyoftheindividualpagesandrendertheHTMLequivalentofthatsinglepage

§  THISISWHATYOUDOWITHPAGINATIONALREADY!!!



TheuserscrollsasfarastheywantFantasticUserExperience



ThereisaproblemWestillneedlinks!

Partial Solution



TherenderedHTMLversionoftheDOMshouldbeaworkingHTMLcopyofthepage§ Whenyou“InspectElement”youareseeingthevisualrepresentationoftheDOM

The Rendered DOM


HTMLSnapshotsshouldbeasclosetoanexactHTMLinstanceoftheDOMaspossibleGooglebotdoesnotneedtheseatthistime

Let’s Talk Snapshots


DonothaveconflictingsignalsbetweentheOrdersofPrecedence

§ BeconsistentbetweentheHTTPHeaders,HTMLSourceandrenderedDOM

Conflicting SEO Signals


InorderfortheDOMtofullyrendercorrectly,thebrowserneedsaccesstoalltheassetsbeingrequestedforthepageThisistrueforgooglebottorendertheDOM,too§ Ifgooglebotcan’taccessassetslikeJavaScript,pagerenderscanbeincomplete

Crawl Signals – Robots.txt


Crawlsignalscanbepickedupanywhereinthe:§ HTTPHeaders(canonicals)§ HTMLSource(links)§ DOM(links)

CrawlsignalsshouldbeconsistentbetweentheOrdersofPrecedence§ Conflicting,orsignalsthatonlyappearonlyintheDOM(suchastherel=“nofollow”)mightnotbeseenorinterpretedcorrectly

Crawl Signals


ThewebisstillbasedonlinksAJSfunctionisnotahyperlinkelement§ WehaveseenGoogleincorrectlycreateURL“strings”andgenerateURLsthatdon’texist

Onclick + window.location != <a href=”#”>


NavigationstillneedstobelinksthatgotothecorrectpageSamewithFacetedNavigation§ Eachfacetlinkneedstocorrespondtoanactualpageintheclick+reloadmethod

§ TheactualloadingcanbeAJAXcallsfortheuser

Crawl Signals - Navigation


GooglebotisalameuserGooglebotdoesn’tclickonbuttonsanddoesn’tscrolldownthepage,etc§ ThereforethecontentneedstobeloadedintheDOMautomatically,notbasedonuserinteractions

§ Wehaven'tseenAJAXsequencesbeingindexedandinterpretedbyGooglebot.

Content Considerations


1URLperpieceofcontentand1pieceofcontentperURL–§ ItisessentialtohaveeverypieceofcontentaccessibleviaitsownURL

§ SinglePageApplications(SPAs)shouldactuallynotbeusinga“singlepage”orsingleURLwhendeliveringthecontent

Changing Content Best Practices


TabbedcontentshouldallbeintheDOM§ SamewaywewouldwantitifweweretalkingaboutputtingitintheHTMLsourcetobeginwith

Changing Content


BecauseoftheDOMSnapshot(5seconds)§ Contentinjectedautomaticallyafter5secondswon’tbescoredorindexed

§ SEOsignalsafter5secondswon’tbeincludedinthescoring

§ HTMLSnapshotsneedtoalignherewiththecontentwithinthat5secondcutoff

Content - DOM Timeouts


Reviewsonpagesaregenerallylikeoldlazyloadpages§ ThefirstgroupingisloadedintheDOM

§ TherestareAJAXcalls§ ReviewsbehindtheAJAXcallswillmostlikelystillnotbeindexed

Content - Reviews


Indexationsignaldirectivesshouldbealignedaswell§ X-Robots-TagandMetaRobots“noindex”§ JavaScriptRedirects

Indexationsignalhints§ Relnext/prevconsolidationsignals§ LinkCanonicaltags§ LinkAlternatetags

Index Signals



SPA’s and Status Codes


SPA’s and Status Codes


IfURLdoesn’texist->§ JSredirecttoapagethatactually404§ DonotuseMeta-EquivRefreshtoredirect

Ifyouneedtoredirect:–  302–ItneedstobeserversidebeforebeforetheJSapploads(rewriterule)

–  301-sameas302orJSredirect(consideredas301byGoogle)

Status Code Challenges



HTTPHeaders,HTMLSource,HTMLSnapshotsandtheDOMallcontainSEOsignals§ GoogleislookingatallthreeOrdersofPrecedenceforsignals§ Bing/Restofworldlookatitthroughthetraditionaltwoorders§ BeconsistentinyourcontentandSEOsignals

A Signal is a Signal is a Signal


GoogleisdumpingtheDOM§ LineupcontentandSEOsignalatalllevelsofprecedence§ TheDOMshouldbeanHTMLrepresentationoftheworkingpage§ LookatyourpagesinFetchandRendertoseehowGoogleisabletorenderthepage

Googlebot only as Good as the DOM


Googlebotcan’trenderdynamiccontentdrivenbyuserinteractionssuchasclickandmouseover§ Itisn’tauser,itisn’tgoingtointeractwiththepagebeyondalinkorapost§ Itwon’t“scr0ll”tothebottomofthepage§ YouneedtoassignauniqueURLtoeachpieceofuniquecontent

Googlebot is a Lame User


WeTestedJavaScript!§ http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157

HacknewsThread§ https://news.ycombinator.com/item?id=9529782

AngularAirHangout§ https://www.youtube.com/watch?v=lxulee01zyY

Links to articles referenced


LEARN MORE: UPCOMING @SMX EVENTS

THANK YOU! SEE YOU AT THE NEXT #SMX

What To Do When Google Can't Understand Your JavaScript By Jody O'Donnell

Marketing