The Switchabalizer - our journey from spell checker to homophone corrrecter

Post on 23-Aug-2014

326 Views

Category:

Internet

3 Downloads

Preview:

Click to see full reader

DESCRIPTION

Presentation given at Open Data Bay Area by Oskar Singer on using Common Crawl and NLP techniques to improve grammar and spelling correction, specifically homophones.

Transcript

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The SwitchabalizerOur journey from spell checker to homophone correcter

Oskar Singer

July 23, 2014

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

How I got here

I am a rising senior in the UMass Amherst CS program specializingin machine learning and natural language processing.

Last summer, I interned at an Amherst/Boston-based textanalytics company called Lexalytics

I worked with Lexalytics’ head of software engineering on thisproject

Lexalytics often uses CommonCrawl, and it was a great option fora training data set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

How I got here

I am a rising senior in the UMass Amherst CS program specializingin machine learning and natural language processing.

Last summer, I interned at an Amherst/Boston-based textanalytics company called Lexalytics

I worked with Lexalytics’ head of software engineering on thisproject

Lexalytics often uses CommonCrawl, and it was a great option fora training data set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

How I got here

I am a rising senior in the UMass Amherst CS program specializingin machine learning and natural language processing.

Last summer, I interned at an Amherst/Boston-based textanalytics company called Lexalytics

I worked with Lexalytics’ head of software engineering on thisproject

Lexalytics often uses CommonCrawl, and it was a great option fora training data set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

How I got here

I am a rising senior in the UMass Amherst CS program specializingin machine learning and natural language processing.

Last summer, I interned at an Amherst/Boston-based textanalytics company called Lexalytics

I worked with Lexalytics’ head of software engineering on thisproject

Lexalytics often uses CommonCrawl, and it was a great option fora training data set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Motivation

Lexalytics provides sentiment analysis software

Sentiment analysis relies heavily in sentence parsing andpart-of-speech tagging

Misspellings and misusage can do serious damage to accuracy forthose two tasks

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Motivation

Lexalytics provides sentiment analysis software

Sentiment analysis relies heavily in sentence parsing andpart-of-speech tagging

Misspellings and misusage can do serious damage to accuracy forthose two tasks

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Motivation

Lexalytics provides sentiment analysis software

Sentiment analysis relies heavily in sentence parsing andpart-of-speech tagging

Misspellings and misusage can do serious damage to accuracy forthose two tasks

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Failure

Hunspell had an error rate of

216%

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

Failure

Hunspell had an error rate of

216%

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

How is this possible? Two reasons:

Hunspell missed all the mistakes

Hunspell made false corrections

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

How is this possible? Two reasons:

Hunspell missed all the mistakes

Hunspell made false corrections

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

How is this possible? Two reasons:

Hunspell missed all the mistakes

Hunspell made false corrections

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Hunspell was a poor choice for a couple reasons:

Hunspell’s vocabulary is not appropriate or flexible enough forTwitter domain

Hunspell can’t detect correctly spelled words that are out ofcontext

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Hunspell was a poor choice for a couple reasons:

Hunspell’s vocabulary is not appropriate or flexible enough forTwitter domain

Hunspell can’t detect correctly spelled words that are out ofcontext

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Hunspell was a poor choice for a couple reasons:

Hunspell’s vocabulary is not appropriate or flexible enough forTwitter domain

Hunspell can’t detect correctly spelled words that are out ofcontext

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Twitter’s vocabulary of abbreviations and acronyms is constantlygrowing

Hunspell’s internal dictionary is not prepared for this

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Twitter’s vocabulary of abbreviations and acronyms is constantlygrowing

Hunspell’s internal dictionary is not prepared for this

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Example: ur

What was Hunspell’s correction?

Ur (the ancient Sumerian city-state)

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Example: ur

What was Hunspell’s correction?

Ur (the ancient Sumerian city-state)

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

Example: ur

What was Hunspell’s correction?

Ur (the ancient Sumerian city-state)

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

When the issue is misuse rather than misspelling, Hunspellcompletely ignores the problem

Specifically, commonly misused homophones were a huge problemin our data

Examples: two/too/2/to; their/there/they’re; your/you’re

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

When the issue is misuse rather than misspelling, Hunspellcompletely ignores the problem

Specifically, commonly misused homophones were a huge problemin our data

Examples: two/too/2/to; their/there/they’re; your/you’re

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The ApproachThe Weaknesses

What Happened?

When the issue is misuse rather than misspelling, Hunspellcompletely ignores the problem

Specifically, commonly misused homophones were a huge problemin our data

Examples: two/too/2/to; their/there/they’re; your/you’re

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Bayes network

Conditioned on the preceding and succeeding words

Assumes these two words are independent

Does not use bag-of-words approach (considers position)

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Conditional Probability of Preceding or Succeeding Words

P(pre(wi )|wj) =#(wiwj)

#(wj),

where pre(w) is the event that w is the preceding word and #(∗)is the number of occurences of a sequence of words

P(suc(wi )|wj) =#(wjwi )

#(wj),

where suc(w) is the event that w is the succeeding word

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Conditional Probability of Preceding or Succeeding Words

Conditional Probability of Preceding or Succeeding Words

P(pre(wi )|wj) =#(wiwj)

#(wj),

where pre(w) is the event that w is the preceding word and #(∗)is the number of occurences of a sequence of words

P(suc(wi )|wj) =#(wjwi )

#(wj),

where suc(w) is the event that w is the succeeding word

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Conditional Probability of Both Words

P(pre(wi ), suc(wk)|wj) = P(pre(wi )|wj)× P(suc(wk)|wj)

log(P(pre(wi ), suc(wk)|wj)) = log(P(pre(wi )|wj))

+ log(P(suc(wk)|wj))

The first equation holds because of our assumption ofindependence between the preceding and succeeding words

There is a missing term in the scoring function that I will addressin the Future Work section

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Conditional Probability of Both Words

Conditional Probability of Both Words

P(pre(wi ), suc(wk)|wj) = P(pre(wi )|wj)× P(suc(wk)|wj)

log(P(pre(wi ), suc(wk)|wj)) = log(P(pre(wi )|wj))

+ log(P(suc(wk)|wj))

The first equation holds because of our assumption ofindependence between the preceding and succeeding words

There is a missing term in the scoring function that I will addressin the Future Work section

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Probability Model

Conditional Probability of Both Words

Conditional Probability of Both Words

P(pre(wi ), suc(wk)|wj) = P(pre(wi )|wj)× P(suc(wk)|wj)

log(P(pre(wi ), suc(wk)|wj)) = log(P(pre(wi )|wj))

+ log(P(suc(wk)|wj))

The first equation holds because of our assumption ofindependence between the preceding and succeeding words

There is a missing term in the scoring function that I will addressin the Future Work section

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Switchable Sets

Only certain groups should be compared, e.g. ”too” should not bescored against ”their”

Comparable switchables are groups in switchable sets

Each switchable is mapped to its switchable set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Switchable Sets

Only certain groups should be compared, e.g. ”too” should not bescored against ”their”

Comparable switchables are groups in switchable sets

Each switchable is mapped to its switchable set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Switchable Sets

Only certain groups should be compared, e.g. ”too” should not bescored against ”their”

Comparable switchables are groups in switchable sets

Each switchable is mapped to its switchable set

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Picking the Word

The Final Equation

S(wi ,wj ,wk) = log(P(pre(wi ), suc(wk)|wj))

v∗ = argmaxv∈VwjS(wi , v ,wk)

where S(wi ,wj ,wk) is the score for the sequence of words wiwjwk

and Vwj is the switchable set corresponding to wj and v∗ is theideal switchable

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

What about common misspellings that intersect with switchables?

Example: ”ur”

Should we put them in the switchable sets?

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

What about common misspellings that intersect with switchables?

Example: ”ur”

Should we put them in the switchable sets?

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

What about common misspellings that intersect with switchables?

Example: ”ur”

Should we put them in the switchable sets?

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

My opinion: no!

Realistically, its probably okay. I opted for a more elegant solution

Replace all common mispellings with something from theappropriate switchable set

The model’s results are agnositc to the switchable that activates it

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

My opinion: no!

Realistically, its probably okay. I opted for a more elegant solution

Replace all common mispellings with something from theappropriate switchable set

The model’s results are agnositc to the switchable that activates it

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

My opinion: no!

Realistically, its probably okay. I opted for a more elegant solution

Replace all common mispellings with something from theappropriate switchable set

The model’s results are agnositc to the switchable that activates it

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

First Pass

My opinion: no!

Realistically, its probably okay. I opted for a more elegant solution

Replace all common mispellings with something from theappropriate switchable set

The model’s results are agnositc to the switchable that activates it

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Testing

Assume Wikipedia has correct usage of all switchables

Replace target words in Wikipedia articles with words from theirswitchable set

Run the Switchabilizer on corrupted articles

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Testing

Assume Wikipedia has correct usage of all switchables

Replace target words in Wikipedia articles with words from theirswitchable set

Run the Switchabilizer on corrupted articles

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Testing

Assume Wikipedia has correct usage of all switchables

Replace target words in Wikipedia articles with words from theirswitchable set

Run the Switchabilizer on corrupted articles

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Results

How did we do?

20% error

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

BrainstormThe ApproachTesting and Results

Results

How did we do?

20% error

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Ideal Scoring Function

S(wiwjwk) = log(P(wj , pre(wi ), suc(wk))

= log(P(wj)P(wi |wj)P(wk |wj))

Forgot the P(wj) term in the factorization of the joint distribution,which resulted in a slightly unfitting conditional distribution.Remember this for reimplementation!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Ideal Scoring Function

Ideal Scoring Function

S(wiwjwk) = log(P(wj , pre(wi ), suc(wk))

= log(P(wj)P(wi |wj)P(wk |wj))

Forgot the P(wj) term in the factorization of the joint distribution,which resulted in a slightly unfitting conditional distribution.Remember this for reimplementation!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Testing conditions were not ideal because:

Test data is not target data

Mistakes are contrived

Somebody make a labeled test set, then tune the algorithm to it!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Testing conditions were not ideal because:

Test data is not target data

Mistakes are contrived

Somebody make a labeled test set, then tune the algorithm to it!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Testing conditions were not ideal because:

Test data is not target data

Mistakes are contrived

Somebody make a labeled test set, then tune the algorithm to it!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Testing conditions were not ideal because:

Test data is not target data

Mistakes are contrived

Somebody make a labeled test set, then tune the algorithm to it!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Here are some ideas I had for future experiments:

Use a discriminative model like maximum entropy

Consider higher order neighbor words

Implement for other languages

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Here are some ideas I had for future experiments:

Use a discriminative model like maximum entropy

Consider higher order neighbor words

Implement for other languages

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Here are some ideas I had for future experiments:

Use a discriminative model like maximum entropy

Consider higher order neighbor words

Implement for other languages

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Future Work

Here are some ideas I had for future experiments:

Use a discriminative model like maximum entropy

Consider higher order neighbor words

Implement for other languages

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Oskar Singer The Switchabalizer

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

Future WorkCall to Action

Thank You, CommonCrawl!

Thanks so much to Lisa, Stephen, Grace and the rest of the teamfor providing such a fantastic resource and bringing me down toSan Francisco to present!

Oskar Singer The Switchabalizer

top related