Top Banner
http://astrometry.net [email protected] Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang & Keir Mierle University of Toronto David Hogg & Michael Blanton New York University
48

Http://[email protected] Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

Jan 01, 2016

Download

Documents

Joleen Turner
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data

Sam Roweis, Dustin Lang & Keir MierleUniversity of Toronto

David Hogg & Michael BlantonNew York University

Page 2: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Organize All Astronomical Data• Vision: Take every astronomical

image ever taken in the history of the world and unify them into a single accurately annotated and easily searchable database.

• We want to include all modern professional telescope surveys plus all amateur photos, satellite images, historical plate archives…

• How could this ever be possible?

Page 3: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

• You show me a picture of the night sky.

• I tell you where on the sky it came from.

Core Technology: Starfield Search

Page 4: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Example (a million times easier)

Find this “field” on this “sky”.

Page 5: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Example (a million times easier)

Find this “field” on this “sky”.

Page 6: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

(Inverted) Index of Features• To solve this problem, we employ the

classic idea of an “inverted index”.• We define a set of “features” for any

particular view of the sky (image).• Then we make an (inverted) index, telling

us which views on the sky exhibit certain (combinations of) feature values.

• When we see a new test image,we compute which features arepresent, and use our inverted indexto look up which possible viewsfrom the catalogue also have those feature values.

• Index is built from a huge all-sky catalogue.

Page 7: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Catalogues: USNO-B 1.0 + TYCHO-2

• USNO-B is an all-sky catalogue compiled from scans of old Schmidt plates.Contains about 109 objects, both stars and galaxies.

• TYCHO-2 is a tiny subset of 2.5Mbrightest stars.

Page 8: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

“Solving” is Easy as 1-2-3

1) Get your image.

2) Upload it to us.

3) Your exact location + lots of other data!

Page 9: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Astronomy Picture of the Day

?

Page 10: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 11: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 12: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 13: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

0.5 degrees

Page 14: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 15: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 16: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 17: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

7 arcminutes

Page 18: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 19: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 20: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 21: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

1.5

degr

ees

Page 22: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 23: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 24: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 25: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

7.5

degr

ees

Page 26: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 27: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 28: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 29: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

68 degrees

Page 30: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

What have we done already• Built a prototype at astrometry.net.• Resolved all (340K) Sloan Digital

Sky Survey images blind.• Solved historical plates (~1810).• Tested a live service (professional

& amateur astronomers as users.)• Created a “picture of the day”

layer for upcoming Google Sky.• This was all done by two grad

students and minimal resources.

Page 31: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

What we need to do next• Prototype is extremely

successful. Algorithmsare highly sophisticated.Now we need to scale up.

• Next steps:– Get funding & resources.– Work with researchers,

amateur & professional astronomers and others to understand/develop needs.

– Go live and change science!

+

You?

Page 32: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

The Core TeamDavid Hogg

Michael BlantonKeir Mierle Dustin Lang

Sam Roweis

The real talent!

Page 33: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Preliminary Results: SDSS• The Sloan Digital Sky

Survey (SDSS) is an all-sky, multi-band survey which includes targeted spectroscopy of interesting objects.

• The telescope is located at Apache Point Observatory.

• Fields are 14x9arcmin corresponding to 2048x1361 pixels.

Page 34: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Preliminary Results: SDSS• 336,554 fields

science grade+

• 0 false positives

• 99.84% solved 530 unsolved

• 99.27% solve w/ 60 brightest objs

Assume known pixel scale(for speedup of solving only.)

Magnitudes used only to decide search order.

Page 35: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Preliminary Results: GALEX• GALEX is a space-based

telescope, seeing only in the ultraviolet.

• It was launched in April 2003 by Caltech&NASA and is just about finished collecting data now.

• It takes huge (80 arcmin) circular fields with 5arcsec resolution and spectraof all objects.

Page 36: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Preliminary Results: GALEX

• GALEX NUV fields can be solved easily using an index built from bright blue USNO stars.

Page 37: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Speed/Memory/Disk• Indexing takes ~12 hours,

uses ~ 2 GB of memory and ~100 GB of disk.

• Solving a test image almost always takes <<1sec (not includingobject detection).

• Solving many fields is done by coarse parallelization on about 100 shared CPUs.

Reduces computation time from ~ 4months to overnight.

All the work is in the hardest 10% of fields

SDSS

Page 38: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

Caching Computation• The idea of an inverted index is that is

pushes the computation from search time back to index construction time.

• We actually do perform an exhaustive search of sorts, but it happens during the building of the inverted index and not at search time, so queries can still be fast.

• There are millions of patches of the scale of a test image on the sky (plus rotation), so we need to extract about 30 bits.

Page 39: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &
Page 40: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

1.4 degrees

Page 41: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

Query image(after object detection).

An all-sky catalogue.

Page 42: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

Query image(after object detection).

Zoomed in by a factor of ~ 1 million.

Page 43: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

Query image(after object detection).

The objects in our index.

Page 44: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

All the quads in our index whichare present in the query image.

Page 45: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

A single quad which we happened to try.

Page 46: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

The query image scaled, translated & rotated as specified by the quad.

Page 47: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

The proposed match, on which we run verification.

Page 48: Http://astrometry.netroweis@cs.toronto.edu Making the Sky Searchable: Automatically Organizing the World’s Astronomical Data Sam Roweis, Dustin Lang &

http://astrometry.net [email protected]

A Real Example from SDSS

The verified answer, overlaid on the original catalogue.

The proposed match, on which we run verification.