Justin Isaf - How To Reduce Your Moderation Costs

Moderating for Fractions of a Penny per Comment

@justinisafThe Communal Group

SPRINT San Francisco 2014

Ohai!

Principal at The Communal Group

Ohai!

I used to manage the HuffPost CommunityWe pre-moderated 450,000 comments a day

with 28 moderators

*This is rough initial data, scraped from the huffingtonpost.comPlease do not share

Disclaimer

I do not claim to have thought of everything, for every situation, nor do I intend to. These things worked in at least one situation and are

not guaranteed to work everywhere.

Individual results may vary.

There is no substitute for experience and timely, intelligent decision making. This discussion is designed to explore several aspects of

community building which, when used collectively, could produce positive results.

:)

Why

It’s a super unsexy cost centerBut it’s essential

So let’s make it cheap and best

Why

Industry standard is about $0.25 per comment for low-scale and $0.04 at scale

I’m going to show you how to do it for $0.04 at low-scale and $0.005 at scale

Terms

Active vs Reactive moderation - are you actively looking for good/bad content or are you waiting for “flags” before you react

Pre- and Post- moderation - are you looking at content before or after it’s posted

Moderation Types

Housekeeping - Maintaining baseline standards, designed to avoid offensive content, legal headaches and rapid decline to “no, you’re a d00die head” by deleting “bad” comments.

Curating - Attempting to improve the level of discussion, dialog and debate by promoting “good” comments.

Mediating - Actively engaging with members to work through disagreements and fights without necessarily deleting or promoting content.(Tip: Watch the first sentence, last sentence and first sentence of the last paragraph)

Baselines - Performance

Professional moderators working at @ 85-90% accuracy:

Housekeeping - 400 - 600 comments per hourCurating - 250 - 350 comments per hourMediating - 20-30ish comments per hour

(Tip: Moderators get faster and better with time, but don’t hire below 250)

Baselines - Salaries

Remote moderators start at (plus benefits):

Housekeeping - $12 an hour/$28K per year ($0.03 cpc)Curating - $15 an hour/$30-32K per year ($0.06 cpc)Mediating - Priceless

(Tip: Partner with organizations that place wounded vets or people with disabilities)

Baselines - Outsourcing

$25-$35 per hour or

$0.17-$0.30 cents per comment0-10% Account maintenance fees50-400 comments per hour

(Tip: Outsourcing and Offshoring are VERY different)

Embrace the Machines

People are very good at nuance, but bad at volume

Machines are very good at volume, but bad at nuance


Machine assisted Human moderation lets machines take care of “sure” tasks, and passes “unsure” tasks to

humans

Human assisted Machine moderation puts humans adjusting and auditing machines, without actively

moderating themselves


Machine assisted Human moderation can give you approximately a 4:1 leverage on your humans

(getting us to about $0.0075 cpc)

More importantly, it frees up time and resources for more Curation which is more valuable for the community and more interesting for

moderators

Design, for Moderation

Highlighting trigger words can give moderators a 50% increase in speed

Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist

pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks

about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just

generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is

not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial

to want to avoid means you undoubtedly treat women exactly this way and see no problem with it.

You aren't confused, you're an asshole.


Highlighting trigger words can give moderators a 50% increase in speed

Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist

pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks

about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just

generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is

not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial

to want to avoid means you undoubtedly treat women exactly this way and see no problem with it.

You aren't confused, you're an asshole.


Hotkeys for publish and delete (~5% increase in speed)

Size 14 Times New Roman (or Arial) (~7.5% increase over other fonts)

40 characters per line (~3% increase over 80 characters per line)

User/story context at the top left of the comment (increases accuracy at speed)

Pagination, not infinite scroll (gives a psychological “end” to an endless job)

“Special” comments at random times (breaks monotony)

Automatic audits and reporting (increases accuracy and consistency)

Special shout out to Mountain Dew and 6 hour shifts


All in, a well designed moderation tool will give you another ~200 comments an hour

(bringing us down to $0.005 for machine assisted human mods)

Utilization

We’ve assumed 100% utilization so far(that’s 3.5million comments per year)

If you have that “problem”, congrats

So let’s look at this in a real example

Case Study - Salon.com

Media site dealing with politics~75,000 comments per monthVolume peaks during daytime hours


Started with 2 part time mods, reactive housekeeping moderationTouched ~3-4% of the incoming volumeUp to 24 hour SLACost per comment reviewed was ~$1


Now 1 full time moderator during peak hours for real-time, active, post-moderation and reviews overnight content during lulls

Plus outsourced, reactive, post-mod with a 30 minute SLA on per-comment pricing


Now they touch ~80% of incoming contentWithin 5 minutes during the day and 30 minutes during nights and weekendsAt a cost per comment reviewed of ~$0.04


~2500% increase in coverage, for 96% reduction in cost per comment

to get better coverage

Next Steps - Nascent CommunityFewer than 5K comments per month

Hands on by the owner/CM

Consider volunteer “leaders” to help you Mediate, set the tone and identify promotable content

Next Steps - Small Community5-50K comments per month

Outsource 24/7 flagged comment review with a 30 minute SLA (assuming ~10% flag rate)

Continue hands on Mediation and step up Curation

Next Steps - Medium Community50-200K comments per month

Outsource flagged comments

Hire in-house moderator(s) for peak hour housekeeping (6 hour shifts 6 days a week)

CM/Owner kept for strategic interventions

Next Steps - Large Community200K+ comments per month

Hire 3 full time/2 part time moderators in-house

Turn on the machines

Begin managing through design

Cat

Cat

Done

Thanks!Questions?

@justinisaf / [email protected] / 415-789-ohai

Most common type of moderation.

Often necessary. Often not sufficient.

Never ending, escalating battle.

Moderation Types - Housekeeping

Do:

First sentence, last sentence and the first sentence of the last paragraphAim for a 15 minute SLA (no more than 60 minutes)Leverage mutli-faceted filtersEnsure you have robust flaggingLean on the side of do-removalBe consistent

Dont:

Leave large time gaps in coverage (like weekends)Blanket delete termsAssume your users are unbiased when they flag

Most likely to increase the quality of discussion.

Often overlooked because the Risk Of Ignoring is low.

Automation tends to be harder.

Moderation Types - Curating

Do:

Aim for a 1 hour SLA (no more than 24 hours)Enlist usersAdd a “good” flagLean on the side of don’t-promote

Dont:

Set an expectation of promotionFavor certain users because it’s easyAssume your users are unbiased when they flagCreate a “grind”

Most likely to increase the quantity of discussion.

Often seen in early stage communities.

Hard to measure, but high long term return.

Moderation Types - Mediation

Do:

Be humanBe humanRemember that it’s just the internet - it’s not that badBe human

Dont:

Take sidesQuote the guidelinesLet an argument go for more than 3 repliesLet problem users become entrenched

Justin Isaf - How To Reduce Your Moderation Costs

Business

bad comments

good comments

ish comments

human moderation

machine moderation

offensive content

goodbad content

unsure tasks