2004-09-15 NVO Summer School, Aspen Center for Physics 1 Publishing and Resource Discovery with Registries Ray Plante Gretchen Greene
Mar 27, 2015
2004-09-15NVO Summer School, Aspen Center for Physics 1
Publishing and Resource Discovery with Registries
Ray PlanteGretchen Greene
2004-09-15NVO Summer School, Aspen Center for Physics 2
All about Registries
• Overview of the Registry Framework
• Publishing to the NVO
• Visits to Registries (publishing)
• VOResource: Resource Metadata in XML
• Visits to Registries (searching)
• IVOA Standard Registry Interface
• Exercise: query registry in an application
2004-09-15NVO Summer School, Aspen Center for Physics 3
The role of Resource Registries
• Used to discover and locate resources—data and services—that can be used in a VO application
• Resource: anything that is describable and identifiable.– Besides data and services: organizations, projects,
software, …– Presently concerned with simple set of resource types
• Registry: a list of resource descriptions– Expressed as structured metadata
to enable automated processing and searching
2004-09-15NVO Summer School, Aspen Center for Physics 4
An Overview of Data Discovery
• You can search the main NVO registry to find resources based on descriptive criteria
• NVO Registries are “coarse-grained”– You can find organizations, archives, catalogs– Won’t find images, celestial objects, table
records
• Registry framework contains multiple registries:– searchable registries– publishing registries
2004-09-15NVO Summer School, Aspen Center for Physics 5
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
Registry Framework
2004-09-15NVO Summer School, Aspen Center for Physics 6
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
Registry Framework
harvest(pull)
2004-09-15NVO Summer School, Aspen Center for Physics 7
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
Registry Framework
harvest(pull)
Cross-harvest
2004-09-15NVO Summer School, Aspen Center for Physics 8
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
Registry Framework
harvest(pull)
selectiveharvesting
Cross-harvest
2004-09-15NVO Summer School, Aspen Center for Physics 9
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
ClientApplications
searchqueries
Registry Framework
2004-09-15NVO Summer School, Aspen Center for Physics 10
Local PublishingRegistry Local
SearchableRegistry
FullSearchableRegistry
Local PublishingRegistry
FullSearchableRegistry
DataCenters
VOProjects
SpecializedPortals & Services
ClientApplications
searchqueries
Registry Framework
2004-09-15NVO Summer School, Aspen Center for Physics 11
NVO Public Registries
Registry URL Searchable?
Publishing?
STScI/JHUNVO Registry
http://nvo.stsci.edu/voregistry/ Yes Yes
Caltech Carnivore http://nvo.caltech.edu:8080/carnivore/ Yes Yes
NCSARegistrationPortal
http://nvo.ncsa.uiuc.edu/nvoregistration.html
No Yes
Private Publishing Registries• HEASARC• CDS
Only support harvesting protocol
2004-09-15NVO Summer School, Aspen Center for Physics 12
Overview of Publishing
• Resources are published if one can use NVO facilities to find them.
• How to Publish to the NVO http://us-vo.org/pub/files/PublishHowto.html
– Multiple layers of publishing• Starts with registry description of resource• Data Access Services
Incremental exposure for incremental effort
– Who are you? How you publish depends on what you want to publish.
• An individual with a small data collection• An archive center• Someone with a cool service
2004-09-15NVO Summer School, Aspen Center for Physics 13
Small collections:VO-ready Repositories
• Repositories that allow users to deposit data to share with community– Guarantee long-term storage, availability
• Automatic support for VO publishing mechamisms– Entries into NVO Registry– Support for standard services:
Cone Search, SIA, SSA, SkyNode
• Currently available Repositories– Images: NCSA Astronomy Digital Image Library
http://adil.ncsa.uiuc.edu/– Spectra: Spectrum Service for the VO
http://voservices.net/spectrum/
• More public repositories are expected to emergeCheck NVO website (http://us-vo.org/) for latest
2004-09-15NVO Summer School, Aspen Center for Physics 14
Persistent Archives:Tools for Federation
• Registering your resources with a VO publishing registry– Enter description into registration form at one of the
available NVO registries:• STScI/JHU Registry: http://nvo.stsci.edu/voregistry/• NCSA Registration Portal:
http://nvo.ncsa.uiuc.edu/nvoregistration.html• Caltech Carnivore:
http://nvo.caltech.edu:8080/carnivore/
– If you have a large number of resources to register, you can run your own registry on your own site
• Caltech Carnivore: http://nvo.caltech.edu:8080/carnivore/
• NCSA VORegistry-in-a-Box http://nvo.ncsa.uiuc.edu/VO/software/
2004-09-15NVO Summer School, Aspen Center for Physics 15
• Which registry should I register with?– Behavior is slightly different– Different strategies for making it “easier” to
register multiple resources– More uniform support is underway
• Support for standard services is strongest– Cone Search, SIA, SkyNode– SkyNode: go to STScI or Carnivore– SSA: go to NCSA
• Your feedback is valuable!
Caution: Construction Ahead
2004-09-15NVO Summer School, Aspen Center for Physics 16
• What can/should you register?– Should: your Organization
• Declares yourself as a publisher with an ID– Should: your Data Collection
• Users at least know how to access it via a Browser– Can: your existing services
• Browser-based services: e.g. search page• Traditional CGI services• Web Services
The next level…• Implement and register one or more standard services
– Cone Search– Simple Image Access– SkyNode– Simple Spectral Access*
*standard still in development
Persistent Archives:Tools for Federation
2004-09-15NVO Summer School, Aspen Center for Physics 17
Cool Services:Integrating with the VO
1. Register your service at a registry• Currently as a generic resource• Improved support for non-standard services coming• (immediate future: contact NVO project)
2. Integrate support for standard VO formats, schemas• FITS and VOTable• Standard Data Model schemas (emerging)
• VOResource, Space-time Coordinates, Spectra
3. Implement Standard Support Interface• a standard in development for:
Self-description, tracking health and usage
2004-09-15NVO Summer School, Aspen Center for Physics 18
A word about Identifiers…
• IVOA Identifier: a globally-unique URI identifying a resource
Ex: ivo://adil.ncsa/targeted/SIA
• Required as part of a registered resource description
• As publisher, you control what it looks like• Two components:
– Authority ID: e.g. adil.ncsaDefines a namespace for identifiersOwned by a single publishing organization
– Resource Key: e.g. targeted/SIAName for the resource unique within the namespaceEncourage re-use of local identifiers
2004-09-15NVO Summer School, Aspen Center for Physics 19
Visits to the Publishing Registries
• Publishing Registries– STScI/JHU Registry:
http://nvo.stsci.edu/voregistry/
– NCSA Registration Portal: http://nvo.ncsa.uiuc.edu/nvoregistration.html
– Caltech Carnivore: http://mercury.cacr.caltech.edu:8080/carnivore/
• Recommend you stick with one registry– Authority IDs are currently “stuck” to a single registry
• Can’t do something? Contact us!
2004-09-15NVO Summer School, Aspen Center for Physics 20
Resource Metadata: XML Schema
• Classes of Resources– Generic Resource– Extensions: e.g.
DataCollection, Service, ConeSearch, SimpleImageAccess, SkyNode
• Organized into separate schemas:– Core resource metadata: VOResource
– Various extensions schemas containing specific types
• Capable of describing…– Data centers, research organizations, missions, observatories– Data collections, archives – VO standard services: Cone Search, Simple Image Access,
Simple Spectral Access, SkyNode– Existing Browser/CGI-based services– Web Services
2004-09-15NVO Summer School, Aspen Center for Physics 21
Searching the Registry
Available Interfaces:– Keyword Search– Intermediate Interfaces
• Form-based• Under-development• e.g. http://nvo.caltech.edu:8080/carnivore/search
– Advanced SQL/ADQL/XQuery interfaces
Registry URL Searchable?
Publishing?
STScI/JHUNVO Registry
http://nvo.stsci.edu/voregistry/ Yes Yes
Caltech Carnivore http://nvo.caltech.edu:8080/carnivore/ Yes Yes
NCSARegistrationPortal
http://nvo.ncsa.uiuc.edu/nvoregistration.html No Yes
2004-09-15NVO Summer School, Aspen Center for Physics 22
IVOA Standard Registry Interface
• IVOA Working Draft: 2 parts– Harvesting:
sending descriptions from publishers to searchable registry– Searching
• Searching– methods
• keywordSearch(string words, boolean combineByOr)• search(ADQLWhere constaints)
– Returns a list of VOResource descriptions– Advanced searching with ADQL:
• Just the “where” part—i.e. search constraints--of ADQL• In place of column names, use XPath to VOResource
element– Curation/Publisher like ‘%NASA%’– Query extensible to any VOResource XML extension– Maps readily to registry implementations based on RDBMS
or XML-DB.
2004-09-15NVO Summer School, Aspen Center for Physics 23
Why might a client use the standard search interface?
• Uniform interface to all VO Registries
– Not dependant on a single registry
• Direct relationship between information you are querying and information you get back.
• Extensible to any type of resource description
• Re-use of ADQL
Custom interfaces to Registries:
• Registries can provide extended functionality
– More advanced capabilities
– Simpler interfaces for specialized purposes
• Client toolkits can provide simplifying interfaces
– Support for ADQL/s
– XPath aliases
2004-09-15NVO Summer School, Aspen Center for Physics 24
What you might use this week
• STScI or Carnivore Registry portals to search for resources
• Any of the registration portals to register resources
• Software for querying registry from an application– Java: see java/dev/nvoregistry– IDL: see idl/VOlib_0.1 (call_registry.pro)– Python: see python/PythonVOClientLibrary.html
(VORegistry.py)– PHP: see in php/web: kwdSearch.php, phpVOlib