Top Banner
VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director
38

VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Mar 27, 2015

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

VO Standards and ProtocolsXML

VOTableUCD

ConeSearch

Roy Williams

California Institute of Technology

NVO co-director

Page 2: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML: Structured Information

<From>Antonio Stadivarius</From><To>Domenico Scarlatti</To><Date>

<Day>13</Day><Month>4</Month><Year>1723</Year>

</Date><Body>Io bisogno una appartamento acoglienti a Cremona …</Body>

4/13/23

April 13, 1723

17.iv.1723

Separation of structure from presentation

The computer can read the document and answer queries like this:“Find all memos from April 1723”

Page 3: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML

•Documents and data•Human readable, editable, mailable•Schema constrains structure

-- can encode data models

•Can be transformed (XSLT)-- other xml-- html/pdf/excel etc

•ToolsParsers in Java, C, C++, Perl, Python, ...Browsers and editorsXML databasesBinding to make API

•For serialization, mediation, brokers

Page 4: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML for science

XML is a comfortable vehicle for our metadata and data models

But the real challenge is:

To define NVO-specific data objectsAnd how they are used

We need consensus more than either software or hardware

VOTableVOResourceservices -- WSDL

Page 5: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML example(no schema)

<?xml version="1.0"?><BookCatalogue>

<Book><Title>The Cambridge Star Atlas</Title><Author>Wil Tirion</Author><ISBN>0-52156-098-5</ISBN><Publisher>Cambridge UP</Publisher>

</Book><Book>

<Title> Parallel Computing Works!</Title><Author>Geoffrey C. Fox</Author><Author>Roy D. Williams</Author><Author>Paul C. Messina</Author><ISBN>1-55860-253-4</ISBN><Publisher>Morgan Kaufmann</Publisher>

</Book></BookCatalogue>

Page 6: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML Parsing

SAX: Event-BasedHandlers functions for StartElement, Text, EndElement, etc.

Found element BookCatalogueFound element BookFound Element TitleFound Text The Cambridge Star AtlasFound End Element Title….

Page 7: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Parsing

DOM: Document Object ModelReturns a tree-like Document object with data attached

BookCatalogue

BookBook

Title

ISBN Parallel Computing Works!

Cambridge Star Atlas

Title

Author

Wil Tirion

Page 8: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

<?xml version="1.0"?><schema xmlns="http://www.w3.org/2000/10/XMLSchema"

xmlns:cat="uri://BookCatalogue">

<element name="BookCatalogue"> <complexType> <sequence>

<element ref="cat:Book" minOccurs="0" maxOccurs="unbounded"/> </sequence> </complexType> </element> <element name="Book"> <complexType> <sequence> <element ref="cat:Title" minOccurs="1" maxOccurs="1"/> <element ref="cat:Author" minOccurs="1"/> <element ref="cat:Date" minOccurs=”0" maxOccurs="1"/> <element ref="cat:ISBN" minOccurs="1" maxOccurs="1"/> <element ref="cat:Publisher" minOccurs="1" maxOccurs="1"/> </sequence> </complexType> </element> <element name="Title" type="string"/> <element name="Author" type="string"/> <element name="Date" type="string"/> <element name="ISBN" type="string"/> <element name="Publisher" type="string"/></schema>

Book.xsd = Xml-Schema Definition

XML Schema

Page 9: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

<?xml version="1.0"?><schema xmlns="http://www.w3.org/2000/10/XMLSchema"

xmlns:cat="uri://BookCatalogue">

<element name="BookCatalogue"> <complexType> <sequence>

<element ref="cat:Book" minOccurs="0" maxOccurs="unbounded"/> </sequence> </complexType> </element> <element name="Book"> <complexType> <sequence> <element ref="cat:Title" minOccurs="1" maxOccurs="1"/> <element ref="cat:Author" minOccurs="1"/> <element ref="cat:Date" minOccurs=”0" maxOccurs="1"/> <element ref="cat:ISBN" minOccurs="1" maxOccurs="1"/> <element ref="cat:Publisher" minOccurs="1" maxOccurs="1"/> </sequence> </complexType> </element> <element name="Title" type="string"/> <element name="Author" type="string"/> <element name="Date" type="string"/> <element name="ISBN" type="string"/> <element name="Publisher" type="string"/></schema>

Book.xsd = Xml-Schema Definition

XSchema

All XML schemas have “schema”as the root element

Page 10: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

<?xml version="1.0"?><schema xmlns="http://www.w3.org/2000/10/XMLSchema"

xmlns:cat="uri://BookCatalogue">

<element name="BookCatalogue"> <complexType> <sequence>

<element ref="cat:Book" minOccurs="0" maxOccurs="unbounded"/> </sequence> <annotation>Catalog is a sequence of books</Annotation> </complexType> </element> <element name="Book"> <complexType> <sequence> <element ref="cat:Title" minOccurs="1" maxOccurs="1"/> <element ref="cat:Author" minOccurs="1"/> <element ref="cat:Date" minOccurs=”0" maxOccurs="1"/> <element ref="cat:ISBN" minOccurs="1" maxOccurs="1"/> <element ref="cat:Publisher" minOccurs="1" maxOccurs="1"/> </sequence> </complexType> </element> <element name="Title" type="string"/> <element name="Author" type="string"/> <element name="Date" type="string"/> <element name="ISBN" type="string"/> <element name="Publisher" type="string"/></schema>

XSchema

Default Namespace declaration:all these come from this standard namespace

Page 11: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

<?xml version="1.0"?><schema xmlns="http://www.w3.org/2000/10/XMLSchema"

xmlns:cat="uri://BookCatalogue">

<element name="BookCatalogue"> <complexType> <sequence>

<element ref="cat:Book" minOccurs="0" maxOccurs="unbounded"/> </sequence> </complexType> </element> <element name="Book"> <complexType> <sequence> <element ref="cat:Title" minOccurs="1" maxOccurs="1"/> <element ref="cat:Author" minOccurs="1"/> <element ref="cat:Date" minOccurs=”0" maxOccurs="1"/> <element ref="cat:ISBN" minOccurs="1" maxOccurs="1"/> <element ref="cat:Publisher" minOccurs="1" maxOccurs="1"/> </sequence> </complexType> </element> <element name="Title" type="string"/> <element name="Author" type="string"/> <element name="Date" type="string"/> <element name="ISBN" type="string"/> <element name="Publisher" type="string"/></schema>

Book.xsd = Xml-Schema Definition

XSchema

This namespace is defined here& abbreviated as "cat"

This element comes from thenamespace called “cat”

Book elementdefined here

Page 12: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Namespace Content

uri://BookCataloguecan be abbreviated as "cat"

Here:

The “cat” namespace contains:BookCatalogueBookTitleAuthorISBNDatePublisher

Page 13: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XML example(with schema)

<?xml version="1.0"?><BookCatalogue xmlns= "uri://BookCatalogue"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation= "uri://BookCatalogue http://www.mydomain.com/schemas/bookcatalog.xsd">>

<Book><Title>The Cambridge Star Atlas</Title><Author>Wil Tirion</Author><ISBN>0-52156-098-5</ISBN><Publisher>Cambridge UP</Publisher>

</Book><Book>

<Title> Parallel Computing Works!</Title><Author>Geoffrey C. Fox</Author><Author>Roy D. Williams</Author><Author>Paul C. Messina</Author><ISBN>1-55860-253-4</ISBN><Publisher>Morgan Kaufmann</Publisher>

</Book></BookCatalogue>

Here is the namespace that we are using in this document

Here is the URL of its schema

Document is instance of a w3c schema

Page 14: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

VOTable

• Full metadata representation• Hierarchy of RESOURCEs• containing PARAMs and TABLEs• UCD (unified content descriptor)

– a has unit meter

– a has UCD ORBIT_SIZE_SMAJ (Semi-major axis of the orbit )

• Can reference remote and/or binary streams• Table can be

– Pure XML

– "Simple Binary"

– FITS Binary Table

Page 15: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Sample VOTable

<?xml version="1.0"?><!DOCTYPE VOTABLE SYSTEM "http://us-vo.org/xml/VOTable.dtd"><VOTABLE version="1.0"> <DEFINITIONS> <COOSYS ID="myJ2000" equinox="2000." epoch="2000." system="eq_FK5"/> </DEFINITIONS> <RESOURCE> <PARAM name="Observer" datatype="char" arraysize="*" value="William Herschel"> <DESCRIPTION>This parameter is designed to store the observer's name </DESCRIPTION> </PARAM> <TABLE name="Stars"> <DESCRIPTION>Some bright stars</DESCRIPTION> <FIELD name="Star-Name" ucd="ID_MAIN" datatype="char" arraysize="10"/> <FIELD name="RA" ucd="POS_EQ_RA" ref="myJ2000" unit="deg" datatype="float" precision="F3" width="7"/> <FIELD name="Dec" ucd="POS_EQ_DEC" ref="myJ2000" unit="deg" datatype="float" precision="F3" width="7"/> <FIELD name="Counts" ucd="NUMBER" datatype="int" arraysize="2x3x*"/> <DATA> <TABLEDATA> <TR> <TD>Procyon</TD><TD>114.827</TD><TD>5.227</TD> <TD>4 5 3 4 3 2 1 2 3 3 5 6</TD> </TR> <TR> <TD>Vega</TD><TD>279.234</TD> <TD>38.782</TD><TD>8 7 8 6 8 6</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE></VOTABLE>

<DATA> <FITS> <STREAM href="ftp://server.com/mydata.fits" expires="2002-02-22" actuate="onRequest"/> </FITS></DATA>

Page 16: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Table Cell

scalar

arrays

variable length arrays

etc

etc

boolean

bit

unsignedByte

short

int

long

char

unicodeChar

float

double

floatComplex

doubleComplex

Primitives

follows FITS binary tabledoes NOT follow XML schema

Page 17: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

VOTable is Flexy

• eg Table of images• UCD="meta.code.mime; image.jpeg"

datatype="unsignedByte" arraysize="*"

• eg Table of URL links• UCD=“meta.ref.url"

datatype="char" arraysize="*"

Page 18: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

VOTable Schema (xsd)

Page 19: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Table Data Model

• Metadata• Class definition for Row• FIELD

– data type

– semantic type

• Data• Each Row is a list of Cells• Each Cell is an array of Primitives

– may be variable length

Page 20: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Table Data Layout

• All metadata first – small, complex, XML

• Class definition for table record• + params, description, etc etc

• Then data– (may be) large, remote– XML | binary | FITS

• Instantiations of table record• All records MUST have same format• binary data allows streaming, parallelism

Page 21: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Param Data Model

• Param is “Table with one cell”• Like a FIELD value• But with a “value” attribute

Page 22: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Primitives

• All have fixed binary length

• Same as FITS primitives

• Except Unicode

datatype Meaning FIT

S Bytes

"boolean" Logical "L" 1

"bit" Bit "X" *

"unsignedByte" Byte (0 to 255) "B" 1

"short" Short Integer "I" 2

"int" Integer "J" 4

"long" Long integer "K" 8

"char" ASCII Character "A" 1

"unicodeChar" Unicode Character   2

"float" Floating point "E" 4

"double" Double "D" 8

"floatComplex" Float Complex "C" 8

"doubleComplex" Double Complex "M" 16

Page 23: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Multidimensional Array Cell

• A table cell can have lots of Primitives• Example: WCS parameters are arrays

– <FIELD name=“CRVAL”datatype=“double”arraysize=“2”/>

• Example: up to 10 images, each 64x64– <FIELD name="thumbs"

datatype="unsignedByte" arraysize="64x64x10*"/>

Page 24: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Hierarchy

• A VOTable contains RESOURCES– RESOURCE can contain:

• TABLE• RESOURCE• etc etc

• Usage example• Many observations in the file,

– each is a RESOURCE

• Each observation is– Parameters

– Calibration table

– Raw data table

Page 25: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Hierarchy

• New feature: GROUP

<TABLE name=“Nutation and Aberration”> <GROUP name=“Nutation”> <FIELD name=“Longitude”/> <FIELD name=“Obliquity”/> </GROUP> <GROUP name=“Aberration”> <GROUP name=“Equinox 1950.0”> <FIELD name=“C”/> <FIELD name=“D”/> </GROUP> <GROUP name=“Equinox 1955.0”> <FIELD name=“C”/> <FIELD name=“D”/> </GROUP> </GROUP></TABLE>

Page 26: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Astronomical Data

• Image– Standard file format: FITS

• Standardized c.1980• Keyword-value dictionary + binary block

• Catalog– Derived from image

• Connected set of bright pixels

– “Table of stars”– Standard format: VOTable

• Standardized 2002• XML with remote binary

• Spectrum

Page 27: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XSLT Example <VOTABLE version="1.0"> <DESCRIPTION>Output from the messier catalog at VirtualSky.org</DESCRIPTION> <RESOURCE type="results"> <PARAM ID="RA" datatype="E" value="200.0" /> <PARAM ID="DE" datatype="E" value="40.0" /> <PARAM ID="SR" datatype="E" value="30.0" /> <PARAM ID="PositionalError" datatype="E" value="0.1" /> <PARAM ID="Credit" datatype="A" arraysize="*" value="Charles Messier, Richard Gelderman" /> <TABLE> <DESCRIPTION>Output from messier Catalog Server</DESCRIPTION> <FIELD ID="I" name="Messier Number" datatype="char" arraysize="*" ucd="ID_MAIN"> <DESCRIPTION>Messier Number</DESCRIPTION> </FIELD> <FIELD ID="RA" name="Right Ascension" datatype="float" unit="degrees" ucd="POS_EQ_RA_MAIN"> <DESCRIPTION>Right Ascension J2000</DESCRIPTION> </FIELD>.... <DATA> <TABLEDATA> <TR> <TD>3</TD> <TD>205.5</TD> <TD>28.402</TD> <TD /> <TD>16.2'</TD> <TD>6.4004</TD> <TD>Globular Cluster</TD> <TD>Canes Venatici</TD> <TD>M3 is one of more heavily studied globular clusters due to its position in the galaxy, putting it far from interstellar absorbtion. More than 200 variable stars have been observed out of a total of near 50,000. Being one of the brightest clusters, M3 is</TD> </TR>

Page 28: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XSLT Result

this table is the result of a conesearch

Page 29: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

XSLT Program

<h2>Data</h2> <table border="1"><xsl:for-each select="FIELD">

<td><b><xsl:value-of select="@name" /> </b></td> </xsl:for-each> <xsl:for-each select="DATA">

<xsl:for-each select="TABLEDATA"><xsl:for-each select="TR">

<tr><xsl:for-each select="TD">

<td width="100"><xsl:value-of select="." /></td> </xsl:for-each> </tr> </xsl:for-each> </xsl:for-each> </xsl:for-each> </table>

Page 30: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Binding to make a Parser

for(int i=0; i<table.getFieldCount(); i++){ Field field = (Field)table.getFieldAt(i); String u = field.getUcd(); if(u != null && u.equals("POS_EQ_RA_MAIN")) System.out.println("Field " + i + " is for RA"); }

From the Schema an API and library is generatedJAXBBreezeCastor

This is JAVOT (Caltech)

Page 31: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Unified Content Descriptor

• UCD is a “semantic type”• phot.mag;em.opt.B Integrated total blue magnitude • src.orbital.eccentricity Orbital eccentricity • stat.median Statistics Median Value

• Base + Specifiers• eg error in default right ascension

– stat.error; pos.eq.ra; meta.main

• First word is "type"• "what kind of thing is this?"• How do we add a stat.error to another?

Page 32: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Unified Content Descriptor

• UCD has services• Natural Language Description• Find best UCD

– Search in NLD

• Matching functions– if I want pos.eq.ra, is stat.error;pos.eq.ra correct?

• What about Ontology???

Page 33: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Some UCD

S stat Statistical parametersQ stat.Fourier Fourier coefficientQ stat.Fourier.amplitude Amplitude Fourier coefficientP stat.covariance Covariance between two parametersP stat.error Statistical errorP stat.error.sys Systematic errorQ stat.fit FitQ stat.fit.chi2 Chi2Q stat.fit.dof Degrees of freedomQ stat.fit.goodness Goodness or significance of fitQ stat.fit.omc Observed minus computedQ stat.fit.param Parameter of fitQ stat.fit.residual Residual fitQ stat.likelihood LikelihoodS stat.max Maximum or upper limitS stat.mean Mean, average valueS stat.median Median valueS stat.min Minimum or lowest limit

Page 34: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Some UCD

S phot PhotometryQ phot.calib Photometric calibrationQ phot.color Color index or magnitude differenceQ phot.color.Cous Color index in Cousins systemQ phot.color.Gen Color index in Geneva systemQ phot.color.Gunn Color index in Gunn systemQ phot.color.JHN Color index in Johnson 65+ system

S meta MetadataP meta.bib Bibliographic referenceP meta.bib.author Author nameP meta.bib.bibcode BibcodeP meta.bib.ivo IVOA identifier ivo://P meta.bib.fig Figure in a paperP meta.bib.journal Journal nameP meta.bib.page Page numberP meta.bib.volume Volume numberP meta.code Code or flagP meta.code.class Classification code

Page 35: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Cone Search

• First VO standard service• Input: RA, DEC, SR must be present

– decimal degrees J2000

• Output: VOTable of sky-located data records– must have columns with UCDs:

POS_EQ_RA_MAIN, POS_EQ_DEC_MAIN, ID_MAIN

RA=300DEC=25SR=0.1

ID RA DEC x y z

RequestResponse

Page 36: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Cone Searches in a VO Registry

Page 37: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Result of Cone Search

RADecID

Page 38: VO Standards and Protocols XML VOTable UCD ConeSearch Roy Williams California Institute of Technology NVO co-director.

Cone Search + Density Probe

Cone

Search

Density

Probe

baseURL

Spacing

Search radius

interoperating NVO-compliant services!

Federation of Multiple Services