Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Network

Crowdsourcing Citizen Science Data Quality with a

Human-Computer Learning Network

Wiggins, Gerbracht, Lagoze, Yu, Wong, & Kelling

7 December, 2012 ~ Lake Tahoe, NVWorkshop on Human Computation for Science and Computational Sustainability

Crowdsourcing Scientific Work

eBird

• Online checklist program for bird abundance & distribution

• Data (mostly) from recreational birders; used widely

• Over 100 million records & growing eBird observations per month

Data Quality

Dogbird Catbird

Data Quality

Dogbird Catbird

X

The eBird HCLN

S Kelling, C Lagoze, W-K Wong, J Yu, T Damoulas, J Gerbracht, D Fink, C Gomes. 2012. eBird: A Human/Computer Learning Network to Improve Biodiversity Conservation and Research. Artificial Intelligence.

Emergent Filters

Kelling, S., J. Yu, J. Gerbracht, and W. K. Wong. 2011. Emergent Filters: Automated Data Verification in a Large-scale Citizen Science Project. Proceedings of the IEEE eScience Conference.

Modeling Expertise

Yit$Zi$

i=1,…,N$

Xi$ Wit$

t=1,…,Ti$

Environmental Covariates Detection

Occupancy (Latent)

Detection Covariates

oi dit

Yit Zi

i=1,…,N

Xi

t=1,…,Ti

Ej Uj j=1,…,M

Wit$

Expertise Covariates

Expertise vj

oi dit, fit

Occupancy-Detection-Expertise

Average Detection Probabilities

Yu, J., W. K. Wong, and R. A. Hutchinson. 2010. Modeling Experts and Novices in Citizen Science Data for Species Distribution Modeling. IEEE 10th International Conference on Data Mining (ICDM),

Hard-to-detect birds Common birds

-0.05

0.00

0.05

0.10

0.15

0.20

Blue Ja

y

Whit

e-brea

sted N

uthatc

h

Northe

rn Card

inal

Great B

lue H

eron

Brown T

hrashe

r

Blue-he

aded

Vireo

Northe

rn Rou

gh-w

inged

Swallow

Wood T

hrush

Emergent Filters + Expertise

Spizella passerina

Emergent Filters + Expertise

Spizella passerina

0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"

Emergent Filters + Expertise

0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

8)Jan" 8)Feb"8)Mar" 8)Apr"8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov"8)Dec"

Spizella passerina

0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"

Emergent Filters + Expertise

0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

8)Jan" 8)Feb"8)Mar" 8)Apr"8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov"8)Dec"0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

1(Jan" 1(Feb"1(Mar" 1(Apr"1(May" 1(Jun" 1(Jul" 1(Aug" 1(Sep" 1(Oct" 1(Nov"1(Dec"

Spizella passerina

0"

5"

10"

15"

20"

25"

30"

35"

40"

45"

50"

8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"

Improving Spatial Coverage

Locations in NY with eBird submissions in 2009

Improving Spatial Coverage

Areas with enough data for emergent filters

Future Work

• Preliminary studies integrated into eBird for better data quality on multiple levels

• Resulting human-computer learning network will use eBird data in new ways

• Evaluation of motivation, learning, and skills related to expertise ranking & birding routes

Thanks!

www.ebird.org

@AndreaWiggins

andrea.wiggins@cornell.edu

www.andreawiggins.com

Acknowledgements

• Leon Levy Foundation

• Wolf Creek Foundation

• National Science Foundation Grants OCI-0830944, CCF-0832782, ITR-0427914, DBI-1049363, DBI-0542868, DUE-0734857, IIS-0748626, IIS-0844546, IIS-0612031, IIS-1050422, IIS-0905385, IIS-0746500, IIS-1209589, AGS-0835821, CNS-0751152, CNS-0855167.